First class functions and higher-order functions

We’ll start by looking at the things that are probably most familiar to you if you’re a JavaScript developer (even if you haven’t necessarily heard the names): first-class functions and higher-order functions.

When we talk about first class functions, we mean that functions are just data—they’re first-class items in the language just like any other type. As such, a function is just another thing you can hand around as an argument to other functions. There’s no distinction between a function and a number or a string or some complex data structure. This is essential because, when you combine it with higher-order functions, it allows for incredible simplicity and incredible reusability.

Higher-order functions, in turn, are functions which take other functions as parameters or return them as their values. We’ll see this in detail in a worked example in a few, but for right now, let’s just use a really simple example that will be familiar to anyone who’s done much JavaScript: using map.

If we have a collection like an array and we want to transform every piece of data in it, we could of course do it with a for loop, and with iterable types we could use for ... of. But with map, we can just leave the implementation details of how the items in the array are iterated through, and instead worry about what we want to change. We can do that because map takes functions as arguments.

const initialValues = [1, 2, 3];
const doubledValues = initialValues.map(value => value * 2);

We did it there with a function explicitly, but we could just as easily extract the function like this:

const double = value => value * 2;
const initialValues = [1, 2, 3];
const doubledValues = initialValues.map(double);

This is possible because functions are just data—they’re first-class members of the language—and therefore functions can be arguments or return values—the language supports higher-order functions.

Pure functions

What about pure functions? Pure functions are functions with no effects. The input directly translates to the output, every time. The examples we looked at just a moment ago with map are all pure functions (and it’s a really weird antipattern to use effectful functions with map! Don’t do that! Use forEach if you must have an effect). Here are a few more super simple examples:

const add = (a, b) => a + b;
const toString = (number) => `The value is ${number}`;
const toLength = (list) => list.length;

Here are some examples of straightforward functions which are not pure:

const logDataFromEndpoint = (endpoint) => {
  fetch(endpoint).then(response => {
    console.log(response);
  });
};

let foo = 42;
const setFoo = (newValue) => {
  foo = newValue;
};

const getFoo = () => foo;

So a pure function is one whose output is solely determined by its input That means no talking to a database, no making API calls, no reading from or writing to disk.

And of course, you can’t do anything meaningful with just pure functions. We need user input, and we need to put the results of our computation somewhere. So the goal isn’t to write only pure functions. It’s to write mostly pure functions and to isolate all impure functions.

What this gets us is two things:

1. A much smaller list of things to worry about when we’re looking at a given function.
2. The ability to compose functions together more easily.

We have fewer things to keep in our heads when we look at any given pure function, because we don’t have to worry at all about whether something it touches has been changed by another function or not. We have inputs. We transform them into outputs. That’s it. Compare these two things in practice.

Here’s a traditional OOP approach:

class Order {
  constructor() {
    this.subTotal = 0.0;
    this.taxRate = 0.01;
  }

  getTotal() {
    return this.subTotal * (1 + this.taxRate);
  }
}

const order = new Order();
order.subTotal = 42.00;

const total = order.getTotal();

Note that the total is always dependent on what has happened in the object. If we write order.subTotal = 43, order.total will change. So if we want to test how total behaves, or if there’s a bug in it, we need to make sure we’ve made all the appropriate transformations to the object ahead of time. That’s no big deal here; the total getter is incredibly simple (and in fact, we’d normally just write it with a property getter). But still, we have to construct an order and make sure all the relevant properties are set to get the right value out of getTotal(). Things outside the method call itself affect what we get back. We have no way to test getTotal() by itself, and no way to debug it if there’s a bug without first doing some object setup.

Now, here’s a functional approach.

const order = {
  subTotal: 42.0,
  taxRate: 0.01
}

const getTotal = (subTotal, taxRate) => subTotal * (1 + taxRate);
const total = getTotal(order.subTotal, order.taxRate);

Note that the object is just data. It’s a record. And the function just takes a couple of arguments. If there needed to be a more complicated transformation internally, we could do that just as easily. Note that it also decouples the structure of the data from the actual computation (though we could pass in a record as well if we had a good reason to).

This makes it easily testable, for free. Want to make sure different tax rates get the correct output? Just… pass in a different tax rate. You don’t have to do any complicated work setting up an object instance first (which is especially important for more complex data types). It also makes it easier to chase down any bugs: the only thing you have to care about is that simple function body. There’s no other state to think about, because there’s no state at all here from the perspective of the function: just inputs and outputs.

This has one other really important consequence, which goes by the name referential transparency. All that means is that anywhere you see a pure function, you can always substitute the value it produces, or vice versa. This is quite unlike the Order::getTotal() method, where (a) it’s attached to an object instance and (b) it’s dependent on other things about that object. You can’t just substitute it in, or freely move it around, when you’re doing a refactor. Maybe you can, but you’d better hope that all the other state is shuffled around with it correctly. Whereas, with the standalone getTotal() function, all you need is its arguments, and you’ll always get the same thing back.

This is just like math: if you say, x = 5 when solving an algebraic equation, you can put 5 anywhere you see x; or, if it’s useful for factoring the equation or something, you can just as easily put x anywhere you see 5. And in math, that’s true for f(x) as well. When we use pure functions, it’s true for programming, too! That makes refactoring much easier.

As we’ll see in the example I walk through in a minute, it also lets us compose functions together far more easily. If all we have are inputs and outputs, then I can take the output from one function and use it as the input to the next.

Immutable data

Complementing the use of mostly pure functions is to use immutable data. Instead of having objects which we mutate, we create copies of the data as we transform it.

You’re probably wondering how in the world this can work (and also how you avoid it being incredibly computationally expensive). For the most part, we can rely on two things: smart compilers and runtimes, and the fact that we often don’t need to reuse the exact same data because we’re transforming it. However, as we’ll see below, in languages which don’t have native support for immutability, it can impose a performance penalty. Gladly, there are ways to work around this!

Pure JavaScript

First, here’s a pure-JavaScript approach, but a more functional one instead of an imperative one:

getBurgers().then(burgers => {
  const descriptionToCount = burgers
    .map(toCondiments)
    .map(toSortedCondiments)
    .map(toDescriptions)
    .reduce(toTotals, {})

  const entries = Object.entries(descriptionToCount);

  [...entries]
    .sort(byCount)
    .slice(0, 10)  // 4. Get the top 10.
    .map(descAndCountToOutput)
    .forEach(output);
});

First, the good: our transformation is no longer all jumbled together. In fact, our code reads a lot like our original description did. Also, notice that we just have a bunch of functions operating on data: none of the functions used here have any knowledge about where the data comes from that they operate on.

But then we also have a couple things that are a little bit clunky. The main thing that sticks out is that sudden stop in the chain in the middle.

When we’re dealing with the Array type, everything is fine, but when we convert our data into a Map, we no longer have that option, so we have to jump through some hoops to do the transformation back into the data type we need. We’re stuck if the object type doesn’t have the method we need. We’re kind of trying to mash together the imperative and functional styles, and it’s leaving us in a little bit of a weird spot.

There’s another issue here, though, and it’s the way that using the method-style calling convention obscures something important. When we call most of those methods, we’re doing something quite different from what most methods do. A method normally is an operation on an object. These methods—most of them—are operations that return new objects. So it’s nice from a syntax perspective, but if we’re not already familiar with the behavior of a given method, it won’t be clear at all that we’re actually generating a bunch of totally new data by calling those methods.

And… two of these methods (sort and forEach) are not doing that, but are modifying an array in place instead.

Lodash

The first step away from this problem is to use a tool like Lodash.

// More functional, with _:
// We tweak how a few of these work slightly to play nicely.
const _toDescriptions = condiments => _.join(condiments, ', ');
const _byCount = _.property(1);

getBurgers().then(burgers => {
  const condiments = _.map(burgers, toCondiments);
  const sortedCondiments = _.map(condiments, toSortedCondiments);
  const descriptions = _.map(sortedCondiments, _toDescriptions);
  const totals = _.reduce(descriptions, toTotals, {});
  const totalPairs = _.toPairs(totals);
  const sortedPairs = _.sortBy(totalPairs, _byCount);
  const sortedPairsDescending = _.reverse(sortedPairs);
  const topTen = _.take(sortedPairsDescending, 10);
  const forOutput = _.map(topTen, descAndCountToOutput)
  _.forEach(forOutput, output);
});

But it seems like we lost something when we moved away from the object-oriented approach. Being able to chain things, so that each item worked with the previous item, was actually pretty nice. And needing all these intermediate variables is not so nice.

One way around this is to use Lodash’s _.chain method. That would have let us write it like this:

getBurgers().then(burgers => {
  const foo = _.chain(burgers)
    .map(toCondiments)
    .map(toSortedCondiments)
    .map(_toDescriptions)
    .reduce(toTotals, {})
    .toPairs()
    .sortBy(_byCount)
    .reverse()
    .take(10)
    .map(descAndCountToOutput)
    .value()
    .forEach(output);
});

And that is a win. But it only works because JavaScript is incredibly dynamic and lets us change the behavior of the underlying Array type. (You’d have a much harder time doing that in Java or C^♯!)

Perhaps just as importantly, it requires us to make sure that we do that _.chain() call on on anything we want to tackle this way. So, can we get the benefits of this some other way? Well, obviously the answer is yes because I wouldn’t be asking otherwise.

With Ramda.

But we can actually go a bit further, and end up in a spot where we don’t need to modify the object prototype at all. We can just do this with a series of standalone functions which don’t depend on being attached to any object. If we use the Ramda library,¹ we can tackle this with nothing but functions.

const getTop10Burgers = R.pipe(
  R.map(R.prop('condiments')),
  R.map(R.sortBy(R.toString)),
  R.map(R.join(', ')),
  R.reduce(toTotals, {}),
  R.toPairs,
  R.sortBy(R.prop(1)),  // will give us least to greatest
  R.reverse,
  R.take(10),
  R.map(descAndCountToOutput)
);

return getBurgers()
  .then(getTop10Burgers)
  .then(R.forEach(output));

Notice the difference between here and even where we started with Lodash: we’re no longer dependent on a specific piece of data being present. Instead, we’ve created a standalone function which can operate on that data, simply by “piping” together—that is, composing—a bunch of other, smaller functions. The output from each one is used as the input for the next.

One of the many small niceties that falls out of this is that we can refactor this just by pulling it apart into smaller acts of compositions.

Here’s an example of how we might use that. We defined those simple transformations for the condiments as a set of three functions, which converted them from objects with condiments elements, sorted them, and joined them into a string. Now, let’s build those into meaningful functions for each step:

// 1. Convert condiments to descriptions.
const burgerRecordsToDescriptions = R.pipe(
  R.map(R.prop('condiments')),
  R.map(R.sortBy(R.toString)),
  R.map(R.join(', ')),
);

// 2. Build up a mapping from description to count.
const descriptionsToUniqueCounts = R.pipe(
  R.reduce(toTotals, {}),
  R.toPairs,
);

// 3. Sort that by count.
const uniqueCountsToSortedPairs = R.pipe(
  R.sortBy(R.prop(1)),
  R.reverse,
);

// For (4), to get the top 10, we'll just use `R.take(10)`.
// We could also alias that, but it doesn't gain us much.

// 5. Print it out
const sortedPairsToConsole = R.pipe(
  R.map(descAndCountToOutput),
  R.forEach(output)
);

Then we can put those together into another, top-level function to do exactly our steps.

const getTop10Burgers = R.pipe(
  burgerRecordsToDescriptions,  // (1)
  descriptionsToUniqueCounts,   // (2)
  uniqueCountsToSortedPairs,    // (3)
  R.take(10)                    // (4)
);

getBurgers()
  .then(getTop10Burgers)
  .then(sortedPairsToConsole);  // (5)

Notice that, because each step is just composing together functions, “refactoring” is easy. And, to be sure, you have to be mindful about what comes in and out of each function. But that’s true in the imperative approach, too: you always have to keep track of the state of the object you’re building up, but there you’re doing it in the middle of a loop, so you’re keeping track of a lot more state at any given time. Functions with simple inputs and outputs give us a more explicit way of specifying the structure and state of the data at any given time. That’s true even in JavaScript, but it goes double if we’re in a typed language like F^♯, Elm, etc., where we can specify those types for the function as a way of designing the flow of the program. (That’s such a helpful way of solving problems, in fact, that I may also do a talk on type-driven design in the future!)

Note, as well, that we’ve now completely isolated our input and output from everything else. The middle there is a chain of pure functions, built out of other pure functions, which neither know nor care that the data came in from an API call, or that we’re going to print it to the console when we finish.

So this takes us back around to that first question: why do we care? At the end of the day, is this really a win over the imperative style? Is the final version, using Ramda, really better than the pure-JavaScript mostly-functional version we used at first?

Obviously, I think the answers there are yes. The Ramda version there at the end is way better than the imperative version, and substantially better than even the first “functional” JavaScript versions we wrote.

For me, at least, the big takeaway here is this: we just built a small but reasonable transformation of data out of a bunch of really small pieces. That has two big consequences—consequences we’ve talked about all along the way, but which you’ve now seen in practice:

1. Those pieces are easy to test. If something isn’t working, I can easily take those pieces apart and test them individually, or test the result of any combination of them. As a result, I can test any part of that pipe chain, and I can fix pieces independent of each other. No part depends on being in the middle of a looper where transformations are done to other parts.

2. Because they’re small and do one simple things, I can recombine those pieces any way I like. And you see that in the Ramda examples in particular: most of what we’re doing in those examples is not even something we wrote ourselves. They’re also really basic building blocks, available in basically every standard library.

One last thing: if you’re curious about performance… you should know that it does matter for data at scale. In my tests (which are admittedly extremely unscientific; unfortunately, I couldn’t get JSPerf running nicely with this particular set of variations), I found that the time it took to run these varied depending on the approach and the library. With a ~10k-record data set:

• The imperative version, unsurprisingly, was the fastest, taking ~16–17ms.
• After that, the chained lodash version and the pure-JS version were comparable, at ~32–36ms, or around twice as long to finish as the imperative version.
• The plain lodash version was consistently a little slower yet, at ~38–43ms.
• Ramda is slow: both variations consistently took over 90ms to finish.

Those differences added up on larger data sets: dealing with ~10,000,000 records, the times ranged from ~12s for the imperative version, to ~15s for the lodash and pure-JS variants, to ~50s for the Ramda version.

They were all pretty darn quick. Compilers, including JavaScript JITs, are incredibly smart. Mostly you can just trust them; come back and profile before you even think about optimizing things. But you should know the performance characteristics of different libraries and consider the implications of what the language does well and what it doesn’t. Ramda is likely slower because of the way it curries every function—something that works well in languages with native support for it, e.g. F^♯ or Elm or Haskell, but imposes a penalty in languages which don’t… like JavaScript. That said, if you’re not in the habit of processing tens of thousands of records, you’re probably okay using any of them.