References

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv Preprint arXiv:1603.04467.

Abernethy, J., Lee, C., and Tewari, A. (2016). Perturbation techniques in online learning and optimization. Perturbations, Optimization, and Statistics, 233.

Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. (2024). Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv Preprint arXiv:2402.14740.

Aji, S. M., and McEliece, R. J. (2000). The generalized distributive law. IEEE Transactions on Information Theory, 46(2), 325–343.

Amari, S.-I. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251–276.

Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv Preprint arXiv:1607.06450.

Bach, F., Jenatton, R., Mairal, J., Obozinski, G., et al. (2012). Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning, 4(1), 1–106.

Ball, K., Carlen, E. A., and Lieb, E. H. (2002). Sharp uniform convexity and smoothness inequalities for trace norms. Inequalities: Selecta of Elliott H. Lieb, 171–190.

Ball, W. W. R. (1960). A short account of the history of mathematics. Courier Corporation.

Balog, M., Tripuraneni, N., Ghahramani, Z., and Weller, A. (2017). Lost relatives of the gumbel trick. International Conference on Machine Learning, 371–379.

Barndorff-Nielsen, O. (2014). Information and exponential families: In statistical theory. John Wiley & Sons.

Baston, R. A., and Nakatsukasa, Y. (2022). Stochastic diagonal estimation: Probabilistic bounds and an improved algorithm. arXiv Preprint arXiv:2201.10684.

Baum, L. E., and Petrie, T. (1966). Statistical inference for probabilistic functions of finite state markov chains. The Annals of Mathematical Statistics, 37(6), 1554–1563.

Baur, W., and Strassen, V. (1983). The complexity of partial derivatives. Theoretical Computer Science, 22(3), 317–330.

Bauschke, H., Heinz, and Combettes, L., Patrick. (2017). Convex analysis and monotone operator theory in hilbert spaces, 2011. In CMS books in mathematics. (2nd ed., pp. 978–971).

Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. (2018). Automatic differentiation in machine learning: A survey. Journal of Marchine Learning Research, 18, 1–43.

Baydin, A. G., Pearlmutter, B. A., Syme, D., Wood, F., and Torr, P. (2022). Gradients without backpropagation. arXiv Preprint arXiv:2202.08587.

Beck, A. (2017). First-order methods in optimization. SIAM.

Beck, A., and Teboulle, M. (2012). Smoothing and first order methods: A unified framework. SIAM Journal on Optimization, 22(2), 557–580.

Becker, S., and Le Cun, Y. (1988). Improving the convergence of back-propagation learning with second order methods. Proceedings of the 1988 Connectionist Models Summer School, 29–37.

Bekas, C., Kokiopoulou, E., and Saad, Y. (2007). An estimator for the diagonal of a matrix. Applied Numerical Mathematics, 57(11-12), 1214–1229.

Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: A CPU and GPU math expression compiler. Proceedings of the Python for Scientific Computing Conference (SciPy), 4, 1–7.

Berthet, Q., Blondel, M., Teboul, O., Cuturi, M., Vert, J.-P., and Bach, F. (2020). Learning with differentiable pertubed optimizers. Advances in Neural Information Processing Systems, 33, 9508–9519.

Bertsekas, D. P. (1997). Nonlinear programming. Journal of the Operational Research Society, 48(3), 334–334.

Blelloch, G. E. (1989). Scans as primitive parallel operations. IEEE Transactions on Computers, 38(11), 1526–1538.

Blondel, M. (2019). Structured prediction with projection oracles. Advances in Neural Information Processing Systems, 32.

Blondel, M., Berthet, Q., Cuturi, M., Frostig, R., Hoyer, S., Llinares-López, F., Pedregosa, F., and Vert, J.-P. (2021). Efficient and modular implicit differentiation. arXiv Preprint arXiv:2105.15183.

Blondel, M., Martins, A. F., and Niculae, V. (2020). Learning with fenchel-young losses. The Journal of Machine Learning Research, 21(1), 1314–1382.

Bolte, J., Boustany, R., Pauwels, E., and Pesquet-Popescu, B. (2022). On the complexity of nonsmooth automatic differentiation. The Eleventh International Conference on Learning Representations.

Bolte, J., and Pauwels, E. (2020). A mathematical model for automatic differentiation in machine learning. Advances in Neural Information Processing Systems, 33, 10809–10819.

Botev, A., Ritter, H., and Barber, D. (2017). Practical Gauss-Newton optimisation for deep learning. International Conference on Machine Learning, 557–565.

Boumal, N. (2023). An introduction to optimization on smooth manifolds. Cambridge University Press.

Boyd, S. P., and Vandenberghe, L. (2004). Convex optimization. Cambridge university press.

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: Composable transformations of Python+NumPy programs (Version 0.3.13) [Computer software]. http://github.com/google/jax

Braun, M., and Golubitsky, M. (1983). Differential equations and their applications (Vol. 2). Springer.

Brockhoff, D., Auger, A., Hansen, N., Arnold, D. V., and Hohm, T. (2010). Mirrored sampling and sequential selection for evolution strategies. Parallel Problem Solving from Nature, PPSN XI: 11th International Conference, Kraków, Poland, September 11-15, 2010, Proceedings, Part i 11, 11–21.

Broyden, C. G. (1970). The convergence of a class of double-rank minimization algorithms 1. General considerations. IMA Journal of Applied Mathematics, 6(1), 76–90.

Brucker, P. (1984). An $O(n)$ algorithm for quadratic knapsack problems. Operations Research Letters, 3(3), 163–166.

Butcher, J. C. (2016). Numerical methods for ordinary differential equations. John Wiley & Sons.

Cajori, F. (1993). A history of mathematical notations (Vol. 1). Courier Corporation.

Céa, J. (1986). Conception optimale ou identification de formes, calcul rapide de la dérivée directionnelle de la fonction coût. M2AN-Modélisation Mathématique Et Analyse Numérique, 20(3), 371–402.

Chaudhuri, S., and Solar-Lezama, A. (2010). Smooth interpretation. ACM Sigplan Notices, 45(6), 279–291.

Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31.

Chen, X., Kayal, N., Wigderson, A., et al. (2011). Partial derivatives in arithmetic complexity and beyond. Foundations and Trends in Theoretical Computer Science, 6(1–2), 1–138.

Clarke, F. H. (1975). Generalized gradients and applications. Transactions of the American Mathematical Society, 205, 247–262.

Clarke, F. H., Ledyaev, Y. S., Stern, R. J., and Wolenski, P. R. (2008). Nonsmooth analysis and control theory (Vol. 178). Springer Science & Business Media.

Cohn, D. L. (2013). Measure theory (Vol. 5). Springer.

Condat, L. (2016). Fast projection onto the simplex and the $\ell_1$ ball. Mathematical Programming, 158(1-2), 575–585.

Dalrymple, D. (2016). Differentiable programming [Computer software]. https://www.edge.org/response-detail/26794

Dangel, F., Kunstner, F., and Hennig, P. (2019). Backpack: Packing more into backprop. arXiv Preprint arXiv:1912.10985.

Danskin, J. M. (1966). The theory of max-min, with applications. SIAM Journal on Applied Mathematics, 14(4), 641–664.

Davis, J. Q., Choromanski, K., Varley, J., Lee, H., Slotine, J.-J., Likhosterov, V., Weller, A., Makadia, A., and Sindhwani, V. (2020). Time dependence in non-autonomous neural odes. arXiv Preprint arXiv:2005.01906.

DeGroot, M. H. (1962). Uncertainty, information, and sequential experiments. The Annals of Mathematical Statistics, 33(2), 404–419.

Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. (2023). Scaling vision transformers to 22 billion parameters. International Conference on Machine Learning, 7480–7512.

Deisenroth, M. P., Faisal, A. A., and Ong, C. S. (2020). Mathematics for machine learning. Cambridge University Press.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186.

Drusvyatskiy, D., and Paquette, C. (2019). Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming, 178, 503–558.

Duchi, J. C., Jordan, M. I., Wainwright, M. J., and Wibisono, A. (2015). Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5), 2788–2806.

Duchi, J. C., Shalev-Shwartz, S., Singer, Y., and Chandra, T. (2008). Efficient projections onto the $\ell_1$-ball for learning in high dimensions. Proc. Of ICML.

Dufter, P., Schmitt, M., and Schütze, H. (2022). Position information in transformers: An overview. Computational Linguistics, 48(3), 733–763.

Eisner, J. (2016). Inside-outside and forward-backward algorithms are just backprop (tutorial paper). Proceedings of the Workshop on Structured Prediction for NLP, 1–17.

Elsayed, M., and Mahmood, A. R. (2022). HesScale: Scalable computation of hessian diagonals. arXiv Preprint arXiv:2210.11639.

Epperly, E. N., Tropp, J. A., and Webber, R. J. (2023). XTrace: Making the most of every sample in stochastic trace estimation. arXiv Preprint arXiv:2301.07825.

Farinhas, A., Aziz, W., Niculae, V., and Martins, A. F. (2021). Sparse communication via mixed distributions. arXiv Preprint arXiv:2108.02658.

Flanders, H. (1973). Differentiation under the integral sign. The American Mathematical Monthly, 80(6), 615–627.

Fleming, W. H., and Rishel, R. W. (2012). Deterministic and stochastic optimal control (Vol. 1). Springer Science & Business Media.

Fletcher, R. (1970). A new approach to variable metric algorithms. The Computer Journal, 13(3), 317–322.

Foerster, J., Farquhar, G., Al-Shedivat, M., Rocktäschel, T., Xing, E., and Whiteson, S. (2018). Dice: The infinitely differentiable monte carlo estimator. International Conference on Machine Learning, 1529–1538.

Forney, G. D. (1973). The viterbi algorithm. Proceedings of the IEEE, 61(3), 268–278.

Franceschi, L., Donini, M., Frasconi, P., and Pontil, M. (2017). Forward and reverse gradient-based hyperparameter optimization. International Conference on Machine Learning, 1165–1173.

Frey, B. J., Kschischang, F. R., Loeliger, H.-A., and Wiberg, N. (1997). Factor graphs and algorithms. Proceedings of the Annual Allerton Conference on Communication Control and Computing, 35, 666–680.

Frigyik, B. A., Srivastava, S., and Gupta, M. R. (2008). An introduction to functional derivatives. Dept. Electr. Eng., Univ. Washington, Seattle, WA, Tech. Rep, 1.

Frostig, R., Johnson, M. J., Maclaurin, D., Paszke, A., and Radul, A. (2021). Decomposing reverse-mode automatic differentiation. arXiv Preprint arXiv:2105.09469.

Gautschi, W. (2011). Numerical analysis. Springer Science & Business Media.

Getreuer, P. (2013). A survey of gaussian convolution algorithms. Image Processing On Line, 2013, 286–310.

Geweke, J. (1988). Antithetic acceleration of monte carlo integration in bayesian inference. Journal of Econometrics, 38(1-2), 73–89.

Gholaminejad, A., Keutzer, K., and Biros, G. (2019). ANODE: Unconditionally accurate memory-efficient gradients for neural ODEs. International Joint Conferences on Artificial Intelligence.

Gini, C. (1912). Variabilità e mutabilità. Reprinted in Memorie Di Metodologica Statistica (Ed. Pizetti E, Salvemini, T). Rome: Libreria Eredi Virgilio Veschi.

Girard, A. (1989). A fast ‘monte-carlo cross-validation’procedure for large least squares problems with noisy data. Numerische Mathematik, 56, 1–23.

Goldfarb, D. (1970). A family of variable-metric methods derived by variational means. Mathematics of Computation, 24(109), 23–26.

Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. (2017). The reversible residual network: Backpropagation without storing activations. Advances in Neural Information Processing Systems, 30.

Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. arXiv Preprint arXiv:1410.5401.

Greig, D. M., Porteous, B. T., and Seheult, A. H. (1989). Exact maximum a posteriori estimation for binary images. Journal of the Royal Statistical Society Series B: Statistical Methodology, 51(2), 271–279.

Griewank, A. (1992). Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optimization Methods and Software, 1(1), 35–54. https://doi.org/10.1080/10556789208805505

Griewank, A. (2003). A mathematical view of automatic differentiation. Acta Numerica, 12, 321–398.

Griewank, A. (2012). Who invented the reverse mode of differentiation. Documenta Mathematica, Extra Volume ISMP, 389400.

Griewank, A., and Walther, A. (2008). Evaluating derivatives: Principles and techniques of algorithmic differentiation. SIAM.

Grimm, J., Pottier, L., and Rostaing-Schmidt, N. (1996). Optimal time and minimum space-time product for reversing a certain class of programs [PhD thesis]. INRIA.

Grünwald, P. D., and Dawid, A. P. (2004). Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. Annals of Statistics, 1367–1433.

Hallman, E., Ipsen, I. C., and Saibaba, A. K. (2023). Monte carlo methods for estimating the diagonal of a real symmetric matrix. SIAM Journal on Matrix Analysis and Applications, 44(1), 240–269.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778.

Helfrich, K., Willmott, D., and Ye, Q. (2018). Orthogonal recurrent neural networks with scaled cayley transform. International Conference on Machine Learning, 1969–1978.

Hestenes, M. R., Stiefel, E., et al. (1952). Methods of conjugate gradients for solving linear systems (Vol. 49). NBS Washington, DC.

Hewitt, E. (1948). Rings of real-valued continuous functions. i. Transactions of the American Mathematical Society, 64(1), 45–99.

Hida, T., and Hitsuda, M. (1976). Gaussian processes (Vol. 120). American Mathematical Soc.

Hiriart-Urruty, J.-B., and Lemaréchal, C. (1993). Convex analysis and minimization algorithms II (Vol. 305). Springer science & business media.

Hutchinson, M. F. (1989). A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3), 1059–1076.

Imai, T. (2019). Where did “differentiable programming” come from? [Computer software]. https://medium.com/@bonotake/where-did-differentiable-programming-come-from-27b385fb6d6d

Ioffe, S., and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, 448–456.

Jaggi, M. (2013). Revisiting frank-wolfe: Projection-free sparse convex optimization. International Conference on Machine Learning, 427–435.

Jang, E., Gu, S., and Poole, B. (2016). Categorical reparameterization with gumbel-softmax. arXiv Preprint arXiv:1611.01144.

Jayaram, B., and Baczynski, M. (2008). Fuzzy implications (Vol. 231). Springer Science & Business Media.

Kakade, S., Shalev-Shwartz, S., Tewari, A., et al. (2009). On the duality of strong convexity and strong smoothness: Learning appl ications and matrix regularization. Tech Report, 2(1), 35.

Karpathy, A. (2017). Software 2.0.

Kelley, C. T. (1995). Iterative methods for linear and nonlinear equations. SIAM.

Kingma, D. P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv Preprint arXiv:1412.6980.

Kingma, D. P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv Preprint arXiv:1312.6114.

Klir, G., and Yuan, B. (1995). Fuzzy sets and fuzzy logic (Vol. 4). Prentice hall New Jersey.

Kobyzev, I., Prince, S., and Brubaker, M. A. (2019). Normalizing flows: Introduction and ideas. Stat, 1050, 25.

Kool, W., Hoof, H. van, and Welling, M. (2019). Buy 4 reinforce samples, get a baseline for free!

Kreikemeyer, J. N., and Andelfinger, P. (2023). Smoothing methods for automatic differentiation across conditional branches. IEEE Access.

Krieken, E., Tomczak, J., and Ten Teije, A. (2021). Storchastic: A framework for general stochastic automatic differentiation. Advances in Neural Information Processing Systems, 34, 7574–7587.

Kunstner, F., Hennig, P., and Balles, L. (2019). Limitations of the empirical fisher approximation for natural gradient descent. Advances in Neural Information Processing Systems, 32.

Lafferty, J., McCallum, A., and Pereira, F. C. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data.

Lan, G. (2012). An optimal method for stochastic composite optimization. Mathematical Programming, 133(1-2), 365–397.

LeCun, Y. (1988). A theoretical framework for back-propagation. Proceedings of the 1988 Connectionist Models Summer School, 1, 21–28.

LeCun, Y. (2018). Deep learning est mort. Vive differentiable programming!

Levenberg, K. (1944). A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2(2), 164–168.

Liu, D. C., and Nocedal, J. (1989). On the limited memory method for large scale optimization. Mathematical Programming, 45, 503–528.

Liu, H., Li, Z., Hall, D., Liang, P., and Ma, T. (2023). Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv Preprint arXiv:2305.14342.

Loeliger, H.-A. (2004). An introduction to factor graphs. IEEE Signal Processing Magazine, 21(1), 28–41.

Loshchilov, I., and Hutter, F. (2016). SGDR: Stochastic gradient descent with warm restarts. International Conference on Learning Representations.

Lucet, Y. (1997). Faster than the fast legendre transform, the linear-time legendre transform. Numerical Algorithms, 16, 171–185.

Maclaurin, D., Duvenaud, D., and Adams, R. P. (2015). Autograd: Effortless gradients in numpy. ICML 2015 AutoML Workshop, 238.

Maddison, C. J., Mnih, A., and Teh, Y. W. (2016). The concrete distribution: A continuous relaxation of discrete random variables. arXiv Preprint arXiv:1611.00712.

Marquardt, D. W. (1963). An algorithm for least-squares estimation of nonlinear parameters. Journal of the Society for Industrial and Applied Mathematics, 11(2), 431–441.

Martens, J. (2020). New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(1), 5776–5851.

Martens, J., and Grosse, R. (2015). Optimizing neural networks with Kronecker-factored approximate curvature. International Conference on Machine Learning, 2408–2417.

Martins, A., and Astudillo, R. (2016). From softmax to sparsemax: A sparse model of attention and multi-label classification. International Conference on Machine Learning, 1614–1623.

Martins, J. R., Sturdza, P., and Alonso, J. J. (2003). The complex-step derivative approximation. ACM Transactions on Mathematical Software (TOMS), 29(3), 245–262.

Meent, J.-W. van de, Paige, B., Yang, H., and Wood, F. (2018). An introduction to probabilistic programming. arXiv Preprint arXiv:1809.10756.

Mensch, A., and Blondel, M. (2018). Differentiable dynamic programming for structured prediction and attention. International Conference on Machine Learning, 3462–3471.

Messerer, F., Baumgärtner, K., and Diehl, M. (2021). Survey of sequential convex programming and generalized Gauss-Newton methods. ESAIM: Proceedings and Surveys, 71, 64–88.

Meyer, R. A., Musco, C., Musco, C., and Woodruff, D. P. (2021). Hutch++: Optimal stochastic trace estimation. Symposium on Simplicity in Algorithms (SOSA), 142–155.

Michelot, C. (1986). A finite algorithm for finding the projection of a point onto the canonical simplex of $\mathbb{R}^n$. Journal of Optimization Theory and Applications, 50(1), 195–200.

Mohamed, S., Rosca, M., Figurnov, M., and Mnih, A. (2020). Monte carlo gradient estimation in machine learning. The Journal of Machine Learning Research, 21(1), 5183–5244.

Mohri, M., Pereira, F., and Riley, M. (2008). Speech recognition with weighted finite-state transducers. Springer Handbook of Speech Processing, 559–584.

Morgenstern, J. (1985). How to compute fast a function and all its derivatives: A variation on the theorem of baur-strassen. ACM SIGACT News, 16(4), 60–62.

Morrey Jr, C. B. (2009). Multiple integrals in the calculus of variations. Springer Science & Business Media.

Murphy, K. P. (2022). Probabilistic machine learning: An introduction. MIT Press. http://probml.github.io/book1

Murphy, K. P. (2023). Probabilistic machine learning: Advanced topics. MIT Press. http://probml.github.io/book2

Mutze, U. (2013). An asynchronous leapfrog method II. arXiv Preprint arXiv:1311.6602.

Nemirovski, A., and Yudin, D. (1983). Problem complexity and method efficiency in optimization.

Nesterov, Y. (2005). Smooth minimization of non-smooth functions. Mathematical Programming, 103, 127–152.

Nesterov, Y. (2007). Modified gauss–newton scheme with worst case guarantees for global performance. Optimisation Methods and Software, 22(3), 469–483.

Nesterov, Y. (2018). Lectures on convex optimization (Vol. 137). Springer.

Nesterov, Y., and Spokoiny, V. (2017). Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17, 527–566.

Olah, C. (2015). Neural networks, types, and functional programming [Computer software]. https://colah.github.io/posts/2015-09-NN-Types-FP/

Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2021). Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1), 2617–2680.

Parikh, N., Boyd, S., et al. (2014). Proximal algorithms. Foundations and Trends in Optimization, 1(3), 127–239.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., … Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems 32 (pp. 8024–8035).

Paulus, M., Choi, D., Tarlow, D., Krause, A., and Maddison, C. J. (2020). Gradient estimation with stochastic softmax tricks. Advances in Neural Information Processing Systems, 33, 5691–5704.

Petersen, F., Borgelt, C., Kuehne, H., and Deussen, O. (2021). Learning with algorithmic supervision via continuous relaxations. Advances in Neural Information Processing Systems, 34, 16520–16531.

Peyré, G. (2020). Mathematical foundations of data sciences. Rn, 1, 2.

Peyré, G., and Cuturi, M. (2019). Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning, 11(5-6), 355–607.

Plotkin, G. (2018). Some principles of differential programming languages [Computer software]. https://popl18.sigplan.org/details/POPL-2018-papers/76/Some-Principles-of-Differential-Programming-Languages

Pollock, S., and Rebholz, L. G. (2021). Anderson acceleration for contractive and noncontractive operators. IMA Journal of Numerical Analysis, 41(4), 2841–2872.

Polyak, B. (1963). Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics, 3(4), 864–878.

Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. Ussr Computational Mathematics and Mathematical Physics, 4(5), 1–17.

Pontryagin, L. S. (1985). The mathematical theory of optimal processes and differential games. Trudy Mat. Inst. Steklov, 169, 119–158.

Press, O., and Wolf, L. (2016). Using the output embedding to improve language models. arXiv Preprint arXiv:1608.05859.

Rabiner, L. R. (1989). A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.

Rademacher, H. (1919). Über partielle und totale differenzierbarkeit von funktionen mehrerer variabeln und über die transformation der doppelintegrale. Mathematische Annalen, 79(4), 340–359.

Radul, A., Paszke, A., Frostig, R., Johnson, M., and Maclaurin, D. (2022). You only linearize once: Tangents transpose to gradients. arXiv Preprint arXiv:2204.10923.

Rahimi, A., and Recht, B. (2007). Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 20.

Recht, B. (2016). Mates of costate.

Recht, B., and Frostig, R. (2017). Nesterov’s punctuated equilibrium.

Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. International Conference on Machine Learning, 1278–1286.

Rockafellar, R. T., and Wets, R. J.-B. (2009). Variational analysis (Vol. 317). Springer Science & Business Media.

Rodriguez, O. H., and Lopez Fernandez, J. M. (2010). A semiotic reflection on the didactics of the chain rule. The Mathematics Enthusiast, 7(2), 321–332.

Roulet, V., and Harchaoui, Z. (2022). Differentiable programming à la Moreau. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3498–3502.

Saad, Y., and Schultz, M. H. (1986). GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing, 7(3), 856–869.

Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. (2017). Evolution strategies as a scalable alternative to reinforcement learning. arXiv Preprint arXiv:1703.03864.

Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. (2021a). Momentum residual neural networks. International Conference on Machine Learning, 9276–9287.

Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. (2021b). Momentum residual neural networks. International Conference on Machine Learning, 9276–9287.

Satterthwaite, F. (1942). Generalized poisson distribution. The Annals of Mathematical Statistics, 13(4), 410–417.

Schlag, I., Irie, K., and Schmidhuber, J. (2021). Linear transformers are secretly fast weight programmers. International Conference on Machine Learning, 9355–9366.

Schölkopf, B., and Smola, A. J. (2002). Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT press.

Schulman, J., Heess, N., Weber, T., and Abbeel, P. (2015). Gradient estimation using stochastic computation graphs. Advances in Neural Information Processing Systems, 28.

Schwartz, J. (1954). The formula for change in variables in a multiple integral. The American Mathematical Monthly, 61(2), 81–85.

Schwarz, H. (1873). Communication. Archives Des Sciences Physiques Et Naturelles, 48, 38–44.

Sengupta, S., Harris, M. J., Garland, M., and Owens, J. D. (2010). Efficient parallel scan algorithms for manycore GPUs.

Shanno, D. F. (1970). Conditioning of quasi-newton methods for function minimization. Mathematics of Computation, 24(111), 647–656.

Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379–423.

Shawe-Taylor, J., and Cristianini, N. (2004). Kernel methods for pattern analysis. Cambridge university press.

Squire, W., and Trapp, G. (1998). Using complex variables to estimate derivatives of real functions. SIAM Review, 40(1), 110–112.

Stoer, J., Bulirsch, R., Bartels, R., Gautschi, W., and Witzgall, C. (1980). Introduction to numerical analysis (Vol. 1993). Springer.

Stumm, P., and Walther, A. (2010). New algorithms for optimal online checkpointing. SIAM Journal on Scientific Computing, 32(2), 836–854.

Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024). Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568, 127063.

Sutherland, D. J., and Schneider, J. (2015). On the error of random fourier features. arXiv Preprint arXiv:1506.02785.

Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). On the importance of initialization and momentum in deep learning. International Conference on Machine Learning, 1139–1147.

Sutton, C., McCallum, A., et al. (2012). An introduction to conditional random fields. Foundations and Trends in Machine Learning, 4(4), 267–373.

Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12.

Taylor, M. (2002). Differential forms and the change of variable formula for multiple integrals. Journal of Mathematical Analysis and Applications, 268(1), 378–383.

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288.

Tignol, J.-P. (2015). Galois’ theory of algebraic equations. World Scientific Publishing Company.

Tsallis, C. (1988). Possible generalization of boltzmann-gibbs statistics. Journal of Statistical Physics, 52, 479–487.

van Krieken, E. (2024). Optimisation in neurosymbolic learning systems [PhD thesis]. Vrije Universiteit Amsterdam.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.

Vaswani, S., Mishkin, A., Laradji, I., Schmidt, M., Gidel, G., and Lacoste-Julien, S. (2019). Painless stochastic gradient: Interpolation, line-search, and convergence rates. Advances in Neural Information Processing Systems, 32.

Verdu, S., and Poor, H. V. (1987). Abstract dynamic programming models under commutativity conditions. SIAM Journal on Control and Optimization, 25(4), 990–1006.

Vicol, P., Metz, L., and Sohl-Dickstein, J. (2021). Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. International Conference on Machine Learning, 10553–10563.

Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., … SciPy 1.0 Contributors. (2020). SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17, 261–272.

Viterbi, A. (1967). Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2), 260–269.

Vorst, H. A. van der, and Vorst, H. A. van der. (1992). Bi-CGSTAB: A fast and smoothly converging variant of Bi-CG for the solution of nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing, 13(2), 631–644. http://dx.doi.org/10.1137/0913035

Wainwright, M. J., and Jordan, M. I. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2), 1–305.

Wang, Q., Moin, P., and Iaccarino, G. (2009). Minimal repetition dynamic checkpointing algorithm for unsteady adjoint calculation. SIAM Journal on Scientific Computing, 31(4), 2549–2567.

Wei, C., Kakade, S., and Ma, T. (2020). The implicit and explicit regularization effects of dropout. International Conference on Machine Learning, 10181–10192.

Werbos, P. J. (1990). Backpropagation through time: What it does and how to do it. Proceedings of the IEEE, 78(10), 1550–1560.

Werbos, P. J. (1994). The roots of backpropagation: From ordered derivatives to neural networks and political forecasting (Vol. 1). John Wiley & Sons.

Wright, S., and Nocedal, J. (1999). Numerical optimization. Springer Science, 35(67-68), 7.

Xu, P., Roosta, F., and Mahoney, M. W. (2020). Second-order optimization for non-convex machine learning: An empirical study. Proceedings of the 2020 SIAM International Conference on Data Mining, 199–207.

Yuan, M., and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1), 49–67.

Zhang, A., Lipton, Z. C., Li, M., and Smola, A. J. (2021). Dive into deep learning. arXiv Preprint arXiv:2106.11342.

Zhou, X. (2018). On the fenchel duality between strong convexity and lipschitz continuou s gradient. arXiv Preprint arXiv:1803.06573.

Zhuang, J., Dvornek, N. C., Tatikonda, S., and Duncan, J. S. (2021). Mali: A memory efficient and reverse accurate integrator for neural odes. arXiv Preprint arXiv:2102.04668.

Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv Preprint arXiv:1909.08593.