![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
https://www.cs.purdue.edu/homes/rompf/papers/wang-preprint201811.pdf
In this paper, we take a fresh look at automatic differentiation (AD) techniques, and especially aim to demystify the reverse-mode form of AD that generalizes back-propagation in neural networks.We uncover a tight connection between reverse-mode AD and delimited continuations, which permits implementing reverse-mode AD purely via operator overloading and with-out managing any auxiliary data structures.
In this paper, we take a fresh look at automatic differentiation (AD) techniques, and especially aim to demystify the reverse-mode form of AD that generalizes back-propagation in neural networks.We uncover a tight connection between reverse-mode AD and delimited continuations, which permits implementing reverse-mode AD purely via operator overloading and with-out managing any auxiliary data structures.
