Max Desiatov
Max DesiatovI'm Max Desiatov, a Ukrainian software engineer.

How do closures and callbacks work? It's turtles all the way down

26 June, 2018

This article is a part of my series about concurrency and asynchronous programming in Swift. The articles are independent, but after reading this one you might want to check out the rest of the series:

The saying holds that the world is supported by a chain of increasingly large turtles. Beneath each turtle is yet another: it is “turtles all the way down”.

Wikipedia

Turtles. All. The. Way. Down.

By Pelf at en.wikipedia, public domain, via Wikimedia Commons

In a few episodes of great podcast Analog(ue) hosts Casey and Myke discuss Crash Course Computer Science series and how it presents abstractions. In software engineering and computer science, we constantly deal with layers of abstractions, but we don’t cross too many of them on daily basis. We work with software libraries that are built on top of a few modules and functions, which are written in a programming language, which is executed as binary code, which runs on CPUs and GPUs, which use logic gates built with transistors, to understand which you might need to understand chemistry and physics. Quite possibly someday particle physicists will discover a few more turtles further down the stack. But as long as you use that software library, you don’t need to care about logic gates, transistors and quantum tunnelling.

So here’s an abstraction that developers work with a lot: asynchronous computations. These days it’s hard to write code that doesn’t touch something asynchronous: any kind of I/O, most frequently networking, or maybe long-running tasks that are scheduled on different threads/processes. For something more concrete, consider a database query, which could be local disk I/O or something that sits somewhere on the network.

I want to write better code and build more abstractions where needed and where it makes the most sense. Because of that, I’ve recently spent a lot of time researching and trying to understand different ways to deal with asynchronous code. In a lot of programming languages callbacks is one of the ways to write it, but it’s also a relatively low level to deal with this abstraction. Nevertheless, it’s important to understand how it works on this level and what can be built on top of it. I hope my explanation could be useful to you as well.

Using closures

In Swift using callbacks is the main way to deal with asynchronicity. While Swift is taken as an example, I don’t use or discusss any advanced features here and I hope it’s going to be interesting even if you don’t use Swift on daily basis.

To pass a callback you could use a “closure”. The Swift Programming Language Guide introduces it this way:

Closures are self-contained blocks of functionality that can be passed around and used in your code. Closures in Swift are similar to blocks in C and Objective-C and to lambdas in other programming languages.

Closures can capture and store references to any constants and variables from the context in which they are defined. This is known as closing over those constants and variables.

[…]

Global and nested functions, as introduced in Functions, are actually special cases of closures. Closures take one of three forms:

  • Global functions are closures that have a name and do not capture any values.
  • Nested functions are closures that have a name and can capture values from their enclosing function.
  • Closure expressions are unnamed closures written in a lightweight syntax that can capture values from their surrounding context.

Copyright © 2018 Apple Inc. Excerpt provided for personal use.

Here’s a simple example of a closure that returns a sum of captured variable and its own argument.

func example() {
  var a = 10

  func f(_ x: Int) -> Int {
    return x + a
  }
  f(5)
  // x = 5, a = 10, returns 15

  a = 15
  f(5)
  // x = 5
  // a in captured environment is 15
  // f returns 20

  let g = f
  // assigned closure f to a new constant g

  g(5)
  // x = 5
  // g now references same environment,
  // where a is set to 15
  // g returns 20

  a = 20
  f(5)
  // a in captured environment is 20
  // f returns 25

  g(5)
  // g shares environment with f,
  // where a is set to 20
  // g returns 25
}

At this point, we can say that a closure is just function code with some captured environment. “Environment” means variables and constants defined outside of closure body. “Capturing” means you have access to a variable value in the environment. If that variable was subsequently modified before closure execution, code within the closure would get the latest up-to-date value. Obviously, constant values are captured as well, but by definition, they can’t be changed.

The example code is wrapped in a function to demonstrate that environment can be local to a current scope as opposed to global environment. If variable a was declared globally, there wouldn’t be anything interesting in using its value, but capturing access to it in local scope requires special treatment in the language implementation.

As seen in the example above, closures have reference semantics, not value semantics. Assigning a closure to a variable doesn’t create a copy of that closure and its environment, but only creates a reference to that. If you were to pass a closure as an argument, it would be passed by reference, not by value.

A more interesting example of closure and callback use:

let backend = Backend()

func example() {
  let indicator = UIActivityIndicatorView()
  let label = UILabel()
  // non-blocking code
  backend.login(email, password,
    onCompletion: { success, failure in
    // code called when `backend.login` completes.

    // indicator variable captured here
    indicator.stopAnimating()

    // UILabel instance is captured here
    label.text = "\(success ?? failure)"
  })

  // ...
  // some other code here that
  // won't be blocked by `backend.login`
  // e.g. UI updates
  indicator.startAnimating()
}

How would you implement closures on a lower level?

Because explaining closures in terms of closures doesn’t make much sense, let’s imagine a programming language that doesn’t have them. To tell a story about the closure turtle, we need to place it on top of other turtles.

What follows is an example of how this could work. It doesn’t require any knowledge of how compilers work, it’s just a lower level representation of closures for our studies.

Consider a subset of Swift, where functions (including anonymous functions) can’t capture the outside environment. That is a language where this:

func example() {
  var a = 5

  func f(_ x: Int) -> Int {
    return x + a
  }

  let g = { x in return x + a }
}

would cause two compiler errors about a being undefined in the body of functions f and g.

To implement closures in this imaginary language, we’d need to use classes to get the reference semantic mentioned above. It’s also useful to have a protocol defining a generic closure shape. It could look like this:

protocol Closure: class {
  associatedtype Environment
  associatedtype Input
  associatedtype Output
  var env: Environment { get set }

  func willRun(_: Input) -> Output
}

And here’s a class conforming to this protocol and implementing our closure with example usage code:

class F: Closure {
  class Environment {
    var a: Int
    init(_ a: Int) {
      self.a = a
    }
  }

  var env: Environment

  init(_ e: Environment) {
    self.e = e
  }

  func willRun(_ x: Int) -> Int {
    return x + e.a
  }
}

func example() {
  var e = F.Environment(10)
  let f = F(e)
  f.willRun(5)
  // returns 15

  e.a = 15
  f.willRun(5)
  // returns 20

  let g = f
  g.willRun(5)
  // returns 20

  e.a = 20
  f.willRun(5)
  // returns 25

  g.willRun(5)
  // returns 25
}

Well, this is verbose. But it’s a pattern you might consider when writing in C, C++ before C++11 and its support for lambdas, or Objective-C before blocks were introduced. And here’s a point that I often don’t see mentioned when closures are explained: you can think of closures as syntactic sugar for anonymous delegates. If you have experience with Objective-C or Cocoa, you might have noticed the analogy already, I deliberately named the protocol function willRun similarly to Cocoa delegate function naming pattern.

Under the hood, a closure-capable compiler will generate a separate function body with a corresponding environment when it sees there are actual variable captures going on. In some implementations, like C++11, you have to specify a list of captured variables explicitly. In Objective-C, that’s what the __block modifier is for, while in Swift it’s automatic by default with a possibility to override it. The latter is quite handy when you want some variables to be captured with weak references to avoid reference cycles.

Callbacks and asynchronous code

In a lot of cases using the right abstractions saves us from verbosity in our code. Are callbacks a good abstraction for all asynchronous scenarios then?

Consider a more complex version of the code we’ve seen before:

let label = UILabel()
// non-blocking code
backend.login(email, password,
  onCompletion: { success, failure in
  // called when `backend.login` completes.
  // UILabel instance is captured here
  label.text = success

  guard !failure else {
    // ...
    // error handling here
    return
  }

  // notice `backend` is captured here as well
  backend.fetchUserInfo { userInfo, failure in
    guard !failure else {
      // ...
      // error handling here again
      return
    }

    if userInfo.isAdmin {
      backend.doAdminStuff { success, failure in
        guard !failure {
          // tired of repeated manual
          // error handling by this point...
          return
        }

        // ...
        // more code here to
        // handle admin stuff results
      }
    }
  }
})

// ...
// some other code here that
// won't be blocked by `backend.login`

To handle a sequence of async actions we need to use nested callbacks. Turns out it’s quite easy to start building these pyramids of doom. Error handling is quite tedious and overall this type of code becomes quite error-prone with shadowed variables all around: notice shadowed failure callback argument within all completion handlers. With plain callbacks, this isn’t an exaggerated scenario, you can often stumble upon this when building real apps that involve any kind of long-running non-blocking operation, e.g. networking.

Promises of future turtles

There is a way out, consider a concept called “futures”, also know as “promises”. Using BrightFutures library as an example, the pyramid of doom can be converted to this flat chain of futures:

let label = UILabel()
// non-blocking code
backend.login(email, password)
.flatMap { success in
  // called when `backend.login` succeeds.
  // UILabel instance is captured here
  label.text = success

  return backend.fetchUserInfo()
}
.onSuccess { userInfo in
  if userInfo.isAdmin {
    return backend.doAdminStuff()
    .onSuccess {
      // ...
      // more code here to handle
      // admin stuff results
    }
  }
}
.onFailure {
  guard !failure else {
    // ...
    // unified error handling here,
    // that can be triggered by any
    // future in the chain
    return
  }
}

// ...
// some other code here that
// won't be blocked by `backend.login`

That’s much better, we have only one error handler that will be triggered regardless of where an error might occur. There is only one nested callback after backend.fetchUserInfo() and only because we have other async code executed on a condition that our user is an admin. More readability and much easier to maintain and refactor if you need to re-order the async operations.

This code assumes you’ve converted all callback-based functions like login, fetchUserInfo and doAdminStuff to return futures. What’s a future? Think of it as a container that can only be opened at some later time (in the future) and it might contain either an actual result or an error. You can attach callbacks to a future “container” to be triggered when the computation is ready, and you can chain these containers together to run a sequence (or sometimes a parallel batch) of asynchronous computations. A simplified declaration could look like this:

class Future<Result> {
  func onSuccess(callback: (Result) -> ())
    -> Future<Result>
  func onFailure(callback: (Error) -> ())
    -> Future<Result>
  func map<NewResult>(transformer: (Result) -> NewResult)
    -> Future<NewResult>
  func flatMap<NewResult>(transformer: (Result) -> Future<NewResult>)
    -> Future<NewResult>
}

There’s a peculiar flatMap function here. In a few other Swift libraries implementing futures, it’s called then, which is a more straightforward name if you don’t focus on the container aspect of futures. In the JavaScript standard library the Promise class uses the name then as well.

So why name it flatMap? Turns out there is already a flatMap function in Swift standard library defined on the Sequence protocol, which is similar to map you’ve probably used if you’re into functional programming. Standard map transforms a container of elements of type A to a container of elements of type B with a closure of type (A) -> B. flatMap does the same for closures of type (A) -> [B], if we assume this container is an array. With simple map the end result would be of type [[B]], which is a container of containers. This isn’t something you’d always want, so to “flatten” these containers to only one level [B] use flatMap. It works the same way for all types of containers that implement the Sequence protocol.

To chain Future<A> with Future<B> that depends on result A, our result transformer closure would need to look like A -> Future<B>. Taking this analogy further, if you applied this closure to simple map, you’d get the end result of type Future<Future<B>>, which is bananas and doesn’t make any sense. Hence we use flatMap to get a “flat” result type.

More turtles coming…

We’ve just reviewed closures and had a quick look at how they’re used for callbacks. We tried to write asynchronous code with callbacks and found that it’s not very maintanable. In the end we tried using futures, which leveraged closures as well, but allowed us to structure the example code in a better way.

Admittedly, this futures stuff is an ok abstraction, but it doesn’t feel as smooth writing plain old synchronous code. If only we could write non-blocking asynchronous code that almost looked like usual synchronous code… Turns out, plenty of programming languages provide a few abstractions to work with asynchronous code and concurrency, like coroutines and async/await syntax sugar in JavaScript, Python, C# to name a few. Some take a different approach, e.g. Erlang/Elixir and Pony are built on actor model, while Go introduces goroutines and channels.

Acknowledgements

Special thanks to Neil Kimmett for feedback on draft versions of this article. Thanks to Thomas Visser for BrightFutures library, which is still the best helper for writing async code in Swift in my opinion.

If you enjoyed this article, please consider becoming a sponsor. My goal is to produce content like this and to work on open-source projects full time, every single contribution brings me closer to it! This also benefits you as a reader and as a user of my open-source projects, ensuring that my blog and projects are constantly maintained and improved.