Chapter 1Introduction

1.1 What is differentiable programming?

A computer program is a sequence of elementary instructions for performing a task. In traditional computer programming, the program is typically manually written by a programmer. However, for certain tasks, particularly those involving intricate patterns and complex decision-making, such as image recognition or text generation, manually writing a program is extremely challenging, if not impossible.

In contrast, modern neural networks offer a different approach. They are constructed by combining parameterized functional blocks and are trained directly from data using gradient-based optimization. This end-to-end training process, where the network learns both feature extraction and task execution simultaneously, allows neural networks to tackle complex tasks that were previously considered insurmountable for traditional, hand-coded programs. This new programming paradigm has been referred to as “differentiable programming” or “software 2.0”. We give an informal definition below.

Neural networks as parameterized programs

In differentiable programming, as in regular computer programming, a program is defined as the composition of elementary operations, forming a computation graph. The key difference is that, as illustrated in Fig. 1.1, the program (such as a neural network) contains parameters that can be adjusted from data and can be differentiated end-to-end using automatic differentiation (autodiff). Typically, it is assumed that the program defines a mathematically valid function (a.k.a. pure function): the function should return identical values for identical arguments and should not have any side effects. Moreover, the function should have well-defined derivatives, ensuring that it can be used in a gradient-based optimization algorithm. Therefore, differentiable programming is not only the art of differentiating through programs but also of designing meaningful differentiable programs.

Why do we need derivatives?

Machine learning typically boils down to optimizing a certain objective function, which is the composition of a loss function and a model (network) function. Derivative-free optimization is called zero-order optimization. It only assumes that we can evaluate the objective function that we wish to optimize. Unfortunately, it is known to suffer from the curse of dimensionality, i.e., it only scales to low-dimensional problems, such as less than $10$ dimensions. Derivative-based optimization, on the other hand, is much more efficient and can scale to millions or billions of parameters. Algorithms that use first and second derivatives are known as first-order and second-order algorithms, respectively.

Why is autodiff so useful?

Before the autodiff revolution, researchers and practitioners needed to manually implement the gradient of the functions they wished to optimize. Manually deriving gradients can become very tedious for complicated functions. Moreover, every time the function is changed (for example, for trying out a new idea), the gradient needs to be re-derived. Autodiff is a game changer because it allows users to focus on quickly and creatively experimenting with functions for their tasks. An example of JAX code (Bradbury et al., 2018) is given in Fig. 1.2.

Differentiable programming is not just deep learning

While there is clearly overlap between deep learning and differentiable programming, their focus is different. Deep learning studies artificial neural networks composed of multiple layers, able to learn intermediate representations of the data. Neural network architectures have been proposed with various inductive biases. For example, convolutional neural networks are designed for images, and transformers are designed for sequences. On the other hand, differentiable programming studies the techniques for designing complex programs and differentiating through them. It is useful beyond deep learning: for instance in reinforcement learning, probabilistic programming and scientific computing in general.

Differentiable programming is not just autodiff

While autodiff is a key ingredient of differentiable programming, it is not the only one. Differentiable programming is also concerned with the design of principled differentiable operations. In fact, much research on differentiable programming has been devoted to making classical computer programming operations compatible with autodiff. As we shall see, many differentiable relaxations can be interpreted in a probabilistic framework. A core theme of this book is the interplay between optimization, probability and differentiation. Differentiation is useful for optimization and conversely, optimization can be used to design differentiable operators.

Our vision for differentiable programming

Computer programming offers powerful tools like control flows, data structures, and standard libraries, enabling users to construct complex programs for solving intricate problems. Our long-term vision is to achieve parity between traditional and differentiable programming, empowering programmers to seamlessly express differentiable programs (such as neural networks) using the full suite of tools they are accustomed to. However, as discussed earlier, differentiable programming is not simply a matter of applying automatic differentiation to existing code. Programs must be designed with differentiability in mind. This usually comes down to inducing a probability distribution over the program or its components. While significant work remains to fully realize this ambitious goal, we hope this book offers a solid foundation.

Where does “differentiable programming” come from?

While neural networks and autodiff have existed for several decades, the term “differentiable programming” is more recent. Olah (2015) discussed analogies between neural network architectures and higher-order functions in functional programming, and referred to it as a “new kind of programming”. Dalrymple (2016) wrote an essay titled “differentiable programming”, recognizing a paradigm where programs learn details through differentiation, with the expressiveness of functional programming. Plotkin (2018) gave a keynote talk titled “Some principles of differential programming languages” at POPL 2018. The “differentiable programming” and “software 2.0” terms were popularized, among others, by LeCun (2018) and Karpathy (2017). From this perspective, autodiff frameworks can be seen, not merely as libraries, but as domain-specific languages (DSLs) embedded into an existing programming language, such as Python. See also Imai (2019) for a review.

1.2 Book goals and scope

The present book aims to provide a comprehensive introduction to differentiable programming with an emphasis on core mathematical tools.

• In Part I, we review fundamentals: differentiation and probabilistic learning.

• In Part II, we review differentiable programs. This includes neural networks, sequence networks and control flows.

• In Part III, we review how to differentiate through programs. This includes automatic differentiation, but also differentiating through optimization and integration (in particular, expectations).

• In Part IV, we review smoothing programs. We focus on two main techniques: infimal convolution, which comes from the world of optimization, and convolution, which comes from the world of integration. We also strive to spell out the connections between them.

• In Part V, we review optimizing programs: basic optimization concepts, first-order algorithms, second-order algorithms and duality.

Our goal is to present the fundamental techniques useful for differentiable programming, not to survey how these techniques have been used in various applications.

1.3 Intended audience

This book is intended to be a graduate-level introduction to differentiable programming. Our pedagogical choices are made with the machine learning community in mind. Some familiarity with calculus, linear algebra, probability theory and machine learning is beneficial.

1.4 How to read this book?

This book does not need to be read linearly chapter by chapter. When needed, we indicate at the beginning of a chapter what chapters are recommended to be read as a prerequisite.

1.5 Related work

Differentiable programming builds upon a variety of connected topics. We review in this section relevant textbooks, tutorials and software.

Standard textbooks on backpropagation and automatic differentiation (autodiff) are those of Werbos (1994) and Griewank and Walther (2008). A tutorial with a focus on machine learning is provided by Baydin et al. (2018). Automatic differentiation is also reviewed as part of more general textbooks, such as those of Deisenroth et al. (2020), Murphy (2022) (from a linear algebra perspective) and Murphy (2023) (from a functional perspective; autodiff section authored by Roy Frostig). The present book was also influenced by Peyré (2020)’s textbook on data science. The history of reverse-mode autodiff is reviewed by Griewank (2012).

A tutorial on different perspectives of backpropagation is “There and Back Again: A Tale of Slopes and Expectations” (link), by Deisenroth and Ong. A tutorial on implicit differentiation is “Deep Implicit Layers - Neural ODEs, Deep Equilibrium Models, and Beyond” (link), by Kolter, Duvenaud, and Johnson.

The standard reference on inference in graphical models and its connection with exponential families is that of Wainwright and Jordan (2008). Differentiable programming is also related to probabilistic programming; see, e.g., Meent et al. (2018).

A review of smoothing from the infimal convolution perspective is provided by Beck and Teboulle (2012). A standard textbook on convex optimization is that of Nesterov (2018). A textbook on first-order optimization methods is that of Beck (2017).

Autodiff implementations that accelerated the autodiff revolution in machine learning are Theano (Bergstra et al., 2010) and Autograd (Maclaurin et al., 2015). Major modern implementations of autodiff include TensorFlow (Abadi et al., 2016), JAX (Bradbury et al., 2018), and PyTorch (Paszke et al., 2019). We in particular acknowledge the JAX team for influencing our view of autodiff.