The Elements of
Differentiable Programming

Mathieu Blondel and Vincent Roulet
Google DeepMind

Download the PDF on arXiv

Contents

• Abstract
• How to cite
• Acknowledgements
• Source code
• Notation
• 1 Introduction
• Part I: Fundamentals
• 2 Differentiation
• 3 Probabilistic learning
• Part II: Differentiable programs
• 4 Parameterized programs
• 5 Control flows
• 6 Data structures
• Part III: Differentiating through programs
• 7 Finite differences
• 8 Automatic differentiation
• 9 Second-order automatic differentiation
• 10 Inference in graphical models as differentiation
• 11 Differentiating through optimization
• 12 Differentiating through integration
• Part IV: Smoothing programs
• 13 Smoothing by optimization
• 14 Smoothing by integration
• Part V: Optimizing differentiable programs
• 15 Optimization basics
• 16 First-order optimization
• 17 Second-order optimization
• 18 Duality
• References