Chapter 5Control flows

Control flows, such as conditionals or loops, are an essential part of computer programming, as they allow us to express complex programs. It is therefore natural to ask whether these constructs can be included in a differentiable program. This is what we study in this chapter.

5.1 Comparison operators

Control flows rely on comparison operators, a.k.a. relational operators. Formally, we can define a comparison operator $\pi = \mathrm{op}(u_1, u_2)$ as a function from $u_1 \in \RR$ and $u_2 \in \RR$ to $\pi \in \{0,1\}$. The binary (Boolean) output $\pi$ can then be used within a conditional statement (see Section 5.6, Section 5.7) to decide whether to execute one branch or another. We define the following operators, illustrated in Fig. 5.1:

• greater than: $$\begin{align*} \mathrm{gt}(u_1, u_2) &\coloneqq \begin{cases} 1 &\mbox{ if } u_1 \ge u_2 \\ 0 &\mbox{ otherwise } \end{cases} \\ &= \mathrm{step}(u_1 - u_2) \end{align*}$$

• less than: $$\begin{align*} \mathrm{lt}(u_1, u_2) &\coloneqq \begin{cases} 1 &\mbox{ if } u_1 \le u_2 \\ 0 &\mbox{ otherwise } \end{cases} \\ &= \mathrm{step}(u_2 - u_1) \end{align*}$$

• equal: $$\begin{align*} \mathrm{eq}(u_1, u_2) &\coloneqq \begin{cases} 1 &\mbox{ if } |u_1 - u_2| = 0 \\ 0 &\mbox{ otherwise } \end{cases} \\ &= \mathrm{gt}(u_2, u_1) \cdot \mathrm{gt}(u_1, u_2) \\ &= \mathrm{step}(u_2 - u_1) \cdot \mathrm{step}(u_1 - u_2) \\ \end{align*}$$

• not equal: $$\begin{align*} \mathrm{neq}(u_1, u_2) &\coloneqq \begin{cases} 1 &\mbox{ if } |u_1 - u_2| > 0 \\ 0 &\mbox{ otherwise } \end{cases} \\ &= 1 - \mathrm{eq}(u_1, u_2) \\ &= 1 - \mathrm{step}(u_2 - u_1) \cdot \mathrm{step}(u_1 - u_2), \end{align*}$$

where $\mathrm{step} \colon \RR \to \{0,1\}$ is the Heaviside step function $$\mathrm{step}(u) \coloneqq \begin{cases} 1 &\mbox{ if } u \ge 0 \\ 0 &\mbox{ otherwise} \end{cases}.$$ The Heaviside step function is piecewise constant. At $u=0$, the function is discontinuous. At $u \neq 0$, it is continuous and has null derivative. Since the comparison operators we presented are all expressed in terms of the $\mathrm{step}$ function, they are all continuous and differentiable almost everywhere, with null derivative. Therefore, while their derivatives are well-defined almost everywhere, they are uninformative and prevent gradient backpropagation.

5.2 Soft inequality operators

5.2.1 Heuristic definition

To obtain a continuous relaxation of inequality operators, we can heuristically replace the step function in the expression of “greater than” and “less than” by a sigmoid function $\mathrm{sigmoid}_\sigma$, where $\sigma > 0$ is a scaling parameter. Such a sigmoid function should satisfy the following properties:

• $\mathrm{sigmoid}_\sigma(-u) = 1 - \mathrm{sigmoid}_\sigma(u)$,

• $\lim_{u \to \infty} \mathrm{sigmoid}_\sigma(u) = 1$,

• $\lim_{u \to -\infty} \mathrm{sigmoid}_\sigma(u) = 0$,

• $\mathrm{sigmoid}_\sigma(0) = \frac{1}{2}$.

Two examples of sigmoids satisfying the aforementioned properties are the logistic function (the CDF of the standard logistic distribution) $$\mathrm{sigmoid}_\sigma(u) \coloneqq \logistic_\sigma(u) \coloneqq \frac{1}{1+ e^{-u/\sigma}} \in (0, 1)$$ and the standard Gaussian’s CDF, defined in Eq. (3.1), $$\mathrm{sigmoid}_\sigma(u) \coloneqq \Phi(u/\sigma).$$ We may then define the soft “greater than” $$\begin{align*} \mathrm{gt}(\mu_1, \mu_2) &= \mathrm{step}(\mu_1 - \mu_2) \\ &\approx \sigmoid_\sigma(\mu_1 - \mu_2) \\ &\eqqcolon \mathrm{gt}_\sigma(\mu_1, \mu_2) \end{align*}$$ and the soft “less than” $$\begin{align*} \mathrm{lt}(\mu_1, \mu_2) &= \mathrm{step}(\mu_2-\mu_1) \\ &\approx \sigmoid_\sigma(\mu_2 - \mu_1) \\ &\eqqcolon \mathrm{lt}_\sigma(\mu_1, \mu_2) \\ &= 1 - \sigmoid_\sigma(\mu_1 - \mu_2) \\ &= 1 - \mathrm{gt}_\sigma(\mu_1, \mu_2). \end{align*}$$ In the limit, we have that $\sigmoid_\sigma(\mu_1 - \mu_2) \to 1$ when $\mu_1 - \mu_2 \to \infty$. In the limit, $\sigmoid_\sigma$ therefore outputs a value of $1$ if $\mu_1$ and $\mu_2$ are infinitely apart. Besides the logistic function and the standard Gaussian’s CDF, other sigmoid functions are possible, as discussed in Section 13.6. In particular, with sparse sigmoids, there exists a finite value $\tau$ such that $\mu_1 - \mu_2 \ge \tau \Longrightarrow \sigmoid_\sigma(\mu_1 - \mu_2) = 1$.

5.2.2 Stochastic process perspective

When the sigmoid used to replace the step function is the logistic function or the standard Gaussian’s CDF, we can revisit the previous heuristic definition of $\mathrm{gt}_\sigma(\mu_1,\mu_2)$ and $\mathrm{lt}_\sigma(\mu_1,\mu_2)$ from a more formal perspective. Indeed, to real values $\mu_1 \in \RR$ and $\mu_2 \in \RR$, we can associate random variables $$U_1 \sim p_{\mu_1,\sigma_1} \quad \text{and} \quad U_2 \sim p_{\mu_2,\sigma_2},$$ thereby forming a stochastic process (we assume that $\sigma_1$ and $\sigma_2$ are fixed). Alternatively, we can also write $$(U_1, U_2)\sim p_{\mu_1,\sigma_1}\otimes p_{\mu_2,\sigma_2},$$ where for two distributions $p_1$ and $p_2$, we denote by $p_1 \otimes p_2$ their outer product $(p_1 \otimes p_2)(u_1, u_2) \coloneqq p_1(u_1)p_2(u_2)$. We can then define $$\begin{align*} \mathrm{gt}_\sigma(\mu_1, \mu_2) &= \EE\left[\mathrm{gt}(U_1, U_2)\right] \\ &= \EE\left[\mathrm{step}(U_1 - U_2)\right] \\ &= \PP(U_1 - U_2 \ge 0) \\ &= \PP(U_2 - U_1 \le 0) \\ &= F_{U_2 - U_1}(0), \end{align*}$$ where $F_X$ is the cumulative distribution function (CDF) of the random variable $X$, and $\sigma$ is a function of $\sigma_1$ and $\sigma_2$. Similarly, we obtain $$\begin{align*} \mathrm{lt}_\sigma(\mu_1, \mu_2) &= \EE\left[\mathrm{lt}(U_1, U_2)\right] \\ &= \EE\left[\mathrm{step}(U_2 - U_1)\right] \\ &= \PP(U_1 - U_2 \le 0) \\ &= F_{U_1 - U_2}(0). \end{align*}$$ We see that the soft inequality operators are based on the CDF of the difference between $U_1$ and $U_2$.

For location-scale family distributions (Section 12.4.1), from a perturbation perspective, we can also define noise variables $Z_1 \sim p_{0,1}$ and $Z_2 \sim p_{0,1}$ such that $U_1 = \mu_1 + \sigma_1 Z_1$ and $U_2 = \mu_2 + \sigma_2 Z_2$ (Section 12.4.1). We then have $$\begin{align*} \mathrm{gt}_\sigma(\mu_1, \mu_2) &= \EE \left[\mathrm{gt}(\mu_1 + \sigma_1 Z_1, \mu_2 + \sigma_2 Z_2)\right] \\ \mathrm{lt}_\sigma(\mu_1, \mu_2) &= \EE \left[\mathrm{lt}(\mu_1 + \sigma_1 Z_1, \mu_2 + \sigma_2 Z_2)\right]. \end{align*}$$

Gaussian case

When $U_1 \sim \mathrm{Normal}(\mu_1, \sigma_1^2)$ and $U_2 \sim \mathrm{Normal}(\mu_2, \sigma_2^2)$, we have $$U_1 - U_2 \sim \mathrm{Normal}(\mu_1 - \mu_2, \sigma_1^2 + \sigma_2^2). \tag{5.1}$$ Denoting $\Phi$ the standard Gaussian’s CDF, we then obtain $$\begin{align*} \mathrm{gt}_\sigma(\mu_1, \mu_2) &= F_{U_2 - U_1}(0) \\ &= \Phi\left(\frac{0 - (\mu_2 - \mu_1)}{\sigma}\right) \\ &= \Phi\left(\frac{\mu_1-\mu_2}{\sigma}\right) \\ \mathrm{lt}_\sigma(\mu_1, \mu_2) &= F_{U_1 - U_2}(0) \\ &= \Phi\left(\frac{0 - (\mu_1 - \mu_2)}{\sigma}\right) \\ &= \Phi\left(\frac{\mu_2-\mu_1}{\sigma}\right), \end{align*}$$ where $\sigma \coloneqq \sqrt{\sigma_1^2 + \sigma_2^2}$. The corresponding distribution for $Z_1$ and $Z_2$ is Gaussian noise.

Logistic case

When $U_1 \sim \mathrm{Gumbel}(\mu_1, \sigma)$ and $U_2 \sim \mathrm{Gumbel}(\mu_2, \sigma)$, we have $$U_1 - U_2 \sim \mathrm{Logistic}(\mu_1 - \mu_2, \sigma). \tag{5.2}$$ We then obtain (see also Proposition 14.3) $$\begin{align*} \mathrm{gt}_\sigma(\mu_1, \mu_2) &= \mathrm{logistic}\left(\frac{\mu_1-\mu_2}{\sigma}\right) \\ \mathrm{lt}_\sigma(\mu_1, \mu_2) &= \mathrm{logistic}\left(\frac{\mu_2-\mu_1}{\sigma}\right). \end{align*}$$ The corresponding distribution for $Z_1$ and $Z_2$ is Gumbel noise.

Recovering hard inequality operators

We easily recover the “hard” inequality operator by $$\mathrm{gt}(\mu_1, \mu_2) = \EE\left[\mathrm{gt}(U_1, U_2)\right],$$ where $U_i \sim \delta_{\mu_i}$ and where $\delta_{\mu_i}$ is the delta distribution that assigns a probability of $1$ to $\mu_i$.

5.3 Soft equality operators

5.3.1 Heuristic definition

The equality operator $\mathrm{eq}(\mu_1,\mu_2)$ can be seen as an extreme kind of similarity function between numbers, that can only output the values $0$ or $1$. To define soft equality operators, a natural idea is therefore to replace the equality operator by a more general similarity function. A similarity function should achieve its maximum at $\mu_1 = \mu_2$ and it should decrease as $\mu_1$ and $\mu_2$ move apart. A common family of similarity functions are kernels. Briefly, a kernel $k(\mu_1, \mu_2)$ can be seen as the inner product $$k(\mu_1, \mu_2) \coloneqq \langle \phi(\mu_1), \phi(\mu_2) \rangle$$ between the embeddings $\phi(\mu_1)$ and $\phi(\mu_2)$ of $\mu_1$ and $\mu_2$ in some (potentially infinite-dimensional) space $\cH$, a reproducing kernel Hilbert space to be precise; see Schölkopf and Smola (2002; Shawe-Taylor and Cristianini, 2004) for an in-depth review of kernels. To obtain a similarity measure between $0$ and $1$ approximating the equality operator, we can normalize to obtain $$\begin{align*} \mathrm{eq}(\mu_1, \mu_2) & \approx \frac{k(\mu_1,\mu_2)} {\sqrt{k(\mu_1,\mu_1)k(\mu_2,\mu_2)}} \\ & = \frac{\langle \phi(\mu_1), \phi(\mu_2) \rangle} {\|\phi(\mu_1)\|\|\phi(\mu_2)\|}, \end{align*}$$ where $\|\phi(\mu)\| \coloneqq \sqrt{\langle \phi(\mu), \phi(\mu)\rangle} = \sqrt{\kappa(\mu, \mu)}$. This is the cosine similarity between $\phi(\mu_1)$ and $\phi(\mu_2)$.

A particular kind of kernel is isotropic kernels of the form $$k(\mu_1, \mu_2) \coloneqq \kappa(\mu_1 - \mu_2),$$ that depend only on the difference between inputs. When the kernel has a scale parameter $\sigma > 0$, we use the notation $\kappa_\sigma$. We can then define a soft equality operator as $$\begin{align*} \mathrm{eq}(\mu_1, \mu_2) \approx \mathrm{eq}_\sigma(\mu_1, \mu_2) \coloneqq \frac{\kappa_\sigma(\mu_1 - \mu_2)}{\kappa_\sigma(0)}. \end{align*}$$ Several isotropic kernels can be chosen such as the Gaussian kernel $$\kappa_\sigma(t) \coloneqq \exp\left(-\frac{t^2}{2\sigma^2}\right)$$ or the logistic kernel $$\kappa_\sigma(t) \coloneqq \mathrm{sech}^2\left(\frac{t}{2\sigma}\right),$$ where we defined the hyperbolic secant $$\mathrm{sech}(u) \coloneqq 2 / (\exp(u) + \exp(-u)).$$ As their names suggest, these kernels arise naturally from a probabilistic perspective, that we present below.

The soft equality operators obtained with these kernels are illustrated in Fig. 5.2. Intuitively, we replaced a bar located at $\mu_1=\mu_2$ with a bump function. The soft equality operator obtained with the logistic kernel coincides with the expression Petersen et al. (2021) arrive at (see their Eq. 9), in a different manner.

5.3.2 Stochastic process perspective

We again adopt the stochastic process perspective, in which we associate random variables $$U_1 \sim p_{\mu_1,\sigma} \quad \text{and} \quad U_2 \sim p_{\mu_2,\sigma}$$ to real values $\mu_1 \in \RR$ and $\mu_2 \in \RR$. However, to handle the equality operator, we cannot simply use the expectation of $\mathrm{eq}(U_1, U_2)$ since $$\EE[\mathrm{eq}(U_1, U_2)] = \PP(U_1=U_2)= 0,$$ $U_1$ and $U_2$ being independent continuous variables. While we cannot use the probability of $U_1=U_2$, or equivalently of $U_1-U_2=0$, we can consider using the probability density function (PDF) $f_{U_1-U_2}$ of $U_1-U_2$ evaluated at $0$. To ensure that the maximum is achieved at $0$ with value $1$, we can normalize the PDF to define $$\mathrm{eq}_\sigma(\mu_1,\mu_2) = \frac{f_{U_1 - U_2}(0)}{f_{U_0}(0)},$$ where $U_0 \sim p_{0, \sigma}$ (we assume that $f_{U_0}$ attains its maximum at $0$). It is well-known that the PDF of the sum of two random variables is the convolution of their respective PDFs. We therefore have $$\begin{align*} f_{U_1-U_2}(t) &= (f_{U_1} \ast f_{-U_2})(t) \\ &= \int_{-\infty}^\infty f_{U_1}(\tau) f_{-U_2}(t - \tau) d\tau. \end{align*}$$ In particular, with $t=0$, if $f_X$ is the PDF of a location-scale family distributed random variable, we obtain $$\begin{align*} f_{U_1-U_2}(0) &= (f_{U_1} \ast f_{-U_2})(0) \\ &= \int_{-\infty}^\infty f_{U_1}(\tau) f_{-U_2}(-\tau) d\tau \\ &= \int_{-\infty}^\infty f_{U_1}(\tau) f_{U_2}(\tau) d\tau \\ &\coloneqq \langle f_{U_1}, f_{U_2} \rangle \\ &\coloneqq \kappa(\mu_1,\mu_2). \end{align*}$$ We indeed recover an inner product and therefore a kernel.

CDF and PDF of absolute difference

While $\PP(U_1=U_2) = 0$, we can also consider using $\PP(|U_1-U_2| \le \varepsilon) = F_{|U_1-U_2|}(\varepsilon)$ as an alternative notion of soft equality. For any random variable $X$, we have $$\begin{align*} F_{|X|}(x) &= \PP(|X| \le x) \\ &= \PP(-x \le X \le x) \\ &= \PP(X \le x) - \PP(X \le -x) \\ &= F_X(x) - F_X(-x). \end{align*}$$ Therefore, $$\PP(|U_1-U_2| \le \varepsilon) = F_{U_1-U_2}(\varepsilon) - F_{U_1-U_2}(-\varepsilon).$$ We can also derive the PDF of $|X|$ as $$\begin{align*} f_{|X|}(x) &= F'_X(x) - F'_{X}(-x) \\ &= f_X(x) + f_{X}(-x) \end{align*}$$ and in particular $$f_{|X|}(0) = 2 f_X(0).$$ Therefore $$f_{|U_1-U_2|}(0) = 2 f_{U_1-U_2}(0),$$ further justifying using the PDF of $U_1 - U_2$ evaluated at $0$. When $X$ follows a normal distribution, $|X|$ follows the so-called folded normal distribution.

Gaussian case

When $U_1 \sim \mathrm{Normal}(\mu_1, \sigma_1^2)$ and $U_2 \sim \mathrm{Normal}(\mu_2, \sigma_2^2)$, we obtain from Eq. (5.1) $$f_{U_1 - U_2}(t) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{(t - (\mu_1 -\mu_2))^2}{2(\sigma_1^2 + \sigma_2^2)}\right)$$ so that $$\mathrm{eq}_\sigma(\mu_1,\mu_2) = \exp\left(-\frac{(\mu_1 -\mu_2)^2}{2(\sigma_1^2 + \sigma_2^2)}\right) \in [0,1].$$ We indeed recover $\kappa_\sigma(\mu_1 - \mu_2) / \kappa_\sigma(0)$, where $\kappa_\sigma$ is the Gaussian kernel with $\sigma = \sqrt{\sigma_1^2 + \sigma_2^2}$. For the CDF of the absolute difference, we obtain $$\PP(|U_1-U_2| \le \varepsilon) = \Phi\left(\frac{\varepsilon-(\mu_1-\mu_2)}{\sigma}\right) - \Phi\left(\frac{-\varepsilon-(\mu_1-\mu_2)}{\sigma}\right).$$

Logistic case

When $U_1 \sim \mathrm{Gumbel}(\mu_1, \sigma)$ and $U_2 \sim \mathrm{Gumbel}(\mu_2, \sigma)$, recalling that $$\mathrm{sech}(u) \coloneqq 2 / (\exp(u) + \exp(-u)),$$ we obtain from Eq. (5.2) $$f_{U_1 - U_2}(t) = \frac{1}{4\sigma} \mathrm{sech}^2\left(\frac{t - (\mu_1 - \mu_2)}{2 \sigma}\right)$$ so that $$\mathrm{eq}_\sigma(\mu_1,\mu_2) = \mathrm{sech}^2\left(\frac{\mu_1 - \mu_2}{2 \sigma}\right) \in [0,1].$$ We indeed recover $\kappa_\sigma(\mu_1 - \mu_2) / \kappa_\sigma(0)$, where $\kappa_\sigma$ is the logistic kernel with $\sigma = \sigma_1 = \sigma_2$.

5.3.3 Gaussian process perspective

The previous approach relied on mapping $\mu_1$ and $\mu_2$ to two independent random variables $U_1 \sim p_{\mu_1,\sigma_1}$ and $U_2 \sim p_{\mu_2,\sigma_2}$ (we assume that $\sigma_1$ and $\sigma_2$ are fixed). Instead, we can consider mapping $\mu_1$ and $\mu_2$ to two dependent random variables $U_1$ and $U_2$, whose covariance depends on the similarity between $\mu_1$ and $\mu_2$. We can do so by using a Gaussian process (Hida and Hitsuda, 1976).

A Gaussian process on $\RR$ is a stochastic process $\{U_\mu: \mu\in \RR\}$ indexed by $\mu \in \RR$ such that any subset of $K$ random variables $(U_{\mu_1}, \dots, U_{\mu_K})$ associated with $(\mu_1, \ldots, \mu_K) \in \RR^K$ is a multivariate Gaussian random variable. The Gaussian process is characterized by the mean function $\mu \mapsto \EE[U_{\mu}]$, and its covariance function $(\mu_i,\mu_j) \mapsto \Cov(U_{\mu_i}, U_{\mu_j})$. For the mean function, we may simply choose $\EE[U_{\mu}]=\mu$. For the covariance function, we need to ensure that the variance of any combination of random variables in the Gaussian process is non-negative. This property is satisfied by kernel functions. We can therefore define $$\Cov(U_{\mu_i}, U_{\mu_j}) \coloneqq k(\mu_i, \mu_j),$$ for some kernel $k$. Equipped with such a mapping from real numbers to random variables, we need a measure of similarity between random variables. A natural choice is their correlation $$\mathrm{corr}(U_{\mu_i}, U_{\mu_j}) \coloneqq \frac{\Cov(U_{\mu_i}, U_{\mu_j})}{\sqrt{\Var(U_{\mu_i})\Var(U_{\mu_j})}} \in [-1, 1].$$ We therefore obtain $$\begin{align*} \mathrm{corr}(U_{\mu_i}, U_{\mu_j}) &= \frac{k(\mu_i,\mu_j)}{\sqrt{k(\mu_i,\mu_i)k(\mu_j,\mu_j)}} \\ &= \frac{\langle \phi(\mu_i),\phi(\mu_j) \rangle}{\|\phi(\mu_i)\|\|\phi(\mu_j)\|}, \end{align*}$$ which coincides with the cosine similarity measure we saw before. In the particular case $K=2$ and when $k_\sigma(\mu_1, \mu_2) = \kappa(\mu_1 - \mu_2)$, we then recover the previous heuristically defined soft equality operator $$\mathrm{eq}_\sigma(\mu_1, \mu_2) = \mathrm{corr}(U_{\mu_1}, U_{\mu_2}) = \frac{\kappa(\mu_1 -\mu_2)}{\kappa(0)}.$$

5.4 Logical operators

Logical operators can be used to perform Boolean algebra. Formally, we can define them as functions from $\{0,1\} \times \{0,1\}$ to $\{0,1\}$. The and (logical conjunction a.k.a. logical product), or (logical disjunction a.k.a. logical addition) and not (logical negation a.k.a. logical complement) operators, for example, are defined by $$\begin{align*} \mathrm{and}(\pi, \pi') &\coloneqq \begin{cases} 1 &\mbox{ if } \pi = \pi' = 1 \\ 0 &\mbox{ otherwise } \end{cases} \\ \mathrm{or}(\pi, \pi') &\coloneqq \begin{cases} 1 &\mbox{ if } 1 \in \{\pi, \pi'\} \\ 0 &\mbox{ otherwise } \end{cases} \\ \mathrm{not}(\pi) &\coloneqq \begin{cases} 0 &\mbox{ if } \pi = 1 \\ 1 &\mbox{ if } \pi = 0 \end{cases}. \end{align*}$$ Classical properties of these operators include

• Commutativity: $$\begin{align*} \mathrm{and}(\pi, \pi') = \mathrm{and}(\pi', \pi) \\ \mathrm{or}(\pi, \pi') = \mathrm{or}(\pi', \pi) \end{align*}$$

• Associativity: $$\begin{align*} \mathrm{and}(\pi, \mathrm{and}(\pi', \pi'')) &= \mathrm{and}(\mathrm{and}(\pi, \pi'), \pi'') \\ \mathrm{or}(\pi, \mathrm{or}(\pi', \pi'')) &= \mathrm{or}(\mathrm{or}(\pi, \pi'), \pi'') \end{align*}$$

• Distributivity of and over or: $$\mathrm{and}(\pi, \mathrm{or}(\pi', \pi'')) = \mathrm{or}(\mathrm{and}(\pi, \pi'), \mathrm{and}(\pi, \pi''))$$

• Neutral element: $$\begin{align*} \mathrm{and}(\pi, 1) &= \pi \\ \mathrm{or}(\pi, 0) &= \pi \end{align*}$$

• De Morgan’s laws: $$\begin{align*} \mathrm{not}(\mathrm{or}(\pi, \pi')) &= \mathrm{and}(\mathrm{not}(\pi), \mathrm{not}(\pi')) \\ \mathrm{not}(\mathrm{and}(\pi, \pi')) &= \mathrm{or}(\mathrm{not}(\pi), \mathrm{not}(\pi')). \end{align*}$$

More generally, for a binary vector $\piv = (\pi_1, \dots, \pi_K) \in \{0,1\}^K$, we can define $\textbf{all}$ (universal quantification, $\forall$) and $\textbf{any}$ (existential quantification, $\exists$) operators, which are functions from $\{0,1\}^K$ to $\{0,1\}$, as $$\mathrm{all}(\piv) \coloneqq \begin{cases} 1 &\mbox{ if } \pi_1 = \dots = \pi_K = 1 \\ 0 &\mbox{ otherwise } \end{cases}$$ and $$\mathrm{any}(\piv) \coloneqq \begin{cases} 1 &\mbox{ if } 1 \in \{\pi_1, \dots, \pi_K\} \\ 0 &\mbox{ otherwise } \end{cases}.$$

5.5 Continuous extensions of logical operators

5.5.1 Probabilistic continuous extension

We can equivalently write the and, or and not operators as $$\begin{align*} \mathrm{and}(\pi, \pi') &= \pi \cdot \pi' \\ \mathrm{or}(\pi, \pi') &= \pi + \pi' - \pi \cdot \pi' \\ \mathrm{not}(\pi) &= 1 - \pi. \end{align*}$$ These are extensions of the previous definitions: we can use them as functions from $[0, 1] \times [0,1] \to [0, 1]$, as illustrated in Fig. 5.3. This means that we can use the soft comparison operators defined in Section 5.2 to obtain $\pi, \pi' \in [0,1]$. Likewise, we can define continuous extensions of all and any, which are functions from $[0,1]^K$ to $[0,1]$, as $$\begin{align*} \mathrm{all}(\piv) &= \prod_{i=1}^K \pi_i \\ \mathrm{any}(\piv) &= 1 - \prod_{i=1}^K (1 - \pi_i). \end{align*}$$

From a probabilistic perspective, if we let $Y$ and $Y'$ be two independent random variables distributed according to Bernoulli distributions with parameters $\pi$ and $\pi'$, then $$\begin{align*} \mathrm{and}(\pi, \pi') &= \PP(Y = 1 \cap Y' = 1) = \PP(Y = 1) \cdot \PP(Y' = 1) \\ \mathrm{or}(\pi, \pi') &= \PP(Y = 1 \cup Y' = 1) \\ &= \PP(Y=1) + \PP(Y'=1) - \PP(Y = 1 \cap Y' = 1) \\ &= \PP(Y=1) + \PP(Y'=1) - \PP(Y = 1) \PP(Y' = 1) \\ \mathrm{not}(\pi) &= \PP(Y \neq 1) = 1 - \PP(Y = 1). \end{align*}$$ In probability theory, these correspond to the product rule of two independent variables, the addition rule, and the complement rule.

Likewise, if we let $Y = (Y_1, \dots, Y_K) \in \{0,1\}^K$ be a random variable distributed according to a multivariate Bernoulli distribution with parameters $\piv = (\pi_1, \dots, \pi_K)$, then $$\begin{align*} \textrm{all}(\piv) &= \PP(Y_1 = 1 \cap \dots \cap Y_K = 1) \\ &= \prod_{i=1}^K \PP(Y_i = 1) \\ \textrm{any}(\piv) &= \PP(Y_1 = 1 \cup \dots \cup Y_K = 1) \\ &= 1 - \PP(\neg(Y_1 = 1 \cup \dots \cup Y_K = 1)) \\ &= 1 - \PP(Y_1 \neq 1 \cap \dots \cap Y_K \neq 1) \\ &= 1 - \prod_{i=1}^K (1 - \PP(Y_i=1)). \end{align*}$$ These are the chain rule of probability and the addition rule of probability for $K$ independent variables.

5.5.2 Triangular norms and co-norms

More generally, in the fuzzy logic literature (Jayaram and Baczynski, 2008; Klir and Yuan, 1995), the concepts of triangular norms and co-norms have been introduced to provide continuous relaxations of the and and or operators, respectively.

The previously-defined probabilistic extensions of $\textbf{and}$ and or are examples of triangular norms and conorms. More examples are given in Table 5.1. Thanks to the associative property of these operators, we can generalize them to vectors $\piv \in [0,1]^K$ to define continuous extensions of the all and any operators, as shown in Table 5.2. For more examples and analysis, see for instance van Krieken (2024, 2 and 3).

5.6 If-else statements

An if-else statement executes different code depending on a condition. Formally, we can define the $\mathrm{ifelse} \colon \{0,1\} \times \cV \times \cV \to \cV$ function by $$\begin{align*} \mathrm{ifelse}(\pi, \v_1, \v_0) &\coloneqq \begin{cases} \v_1 &\mbox{ if } \pi = 1 \\ \v_0 &\mbox{ if } \pi = 0 \end{cases} \tag{5.3}\\ &= \pi \cdot \v_1 + (1 - \pi) \cdot \v_0. \end{align*}$$ The $\pi$ variable is called the predicate. It is a binary (Boolean) variable, making the function $\mathrm{ifelse}$ undefined if $\pi \not \in \{0,1\}$. The function is therefore discontinuous and nondifferentiable w.r.t. $\pi \in \{0,1\}$. On the other hand, $\v_0 \in \cV$ and $\v_1 \in \cV$, which correspond to the false and true branches, can be continuous variables. If $\pi=1$, the function is linear w.r.t. $\v_1$ and constant w.r.t. $\v_0$. Conversely, if $\pi=0$, the function is linear w.r.t. $\v_0$ and constant w.r.t. $\v_1$. We now discuss how to differentiate through $\mathrm{ifelse}$.

5.6.1 Differentiating through branch variables

For $\pi \in \{0,1\}$ fixed, $\mathrm{ifelse}(\pi, \v_1, \v_0)$ is a valid function w.r.t. $\v_1 \in \cV$ and $\v_0 \in \cV$, and can therefore be used as a node in a computational graph (Section 8.3). Due to the linearity w.r.t. $\v_1$ and $\v_0$, we obtain that the Jacobians w.r.t. $\v_1$ and $\v_0$ are $$\begin{align*} \partial_{\v_0} \mathrm{ifelse}(\pi, \v_1, \v_0) &\coloneqq \begin{cases} 0 &\mbox{ if } \pi = 1 \\ I &\mbox{ if } \pi = 0 \end{cases} \\ &= (1-\pi) \cdot I \end{align*}$$ and $$\begin{align*} \partial_{\v_1} \mathrm{ifelse}(\pi, \v_1, \v_0) &\coloneqq \begin{cases} I &\mbox{ if } \pi = 1 \\ 0 &\mbox{ if } \pi = 0 \end{cases} \\ &= \pi \cdot I, \end{align*}$$ where $I$ is the identity matrix of appropriate size. Most of the time, if-else statements are composed with other functions. Let $g_1 \colon \cU_1 \to \cV$ and $g_0 \colon \cU_0 \to \cV$ be differentiable functions. We then define $\v_1 \coloneqq g_1(\u_1)$ and $\v_0 \coloneqq g_0(\u_0)$, where $\u_1 \in \cU_1$ and $\u_0 \in \cU_0$. The composition of $\mathrm{ifelse}$, $g_1$ and $g_0$ is then the function $f \colon \{0,1\} \times \cU_1 \times \cU_0 \to \cV$ defined by $$\begin{align*} f(\pi, \u_1, \u_0) &\coloneqq \mathrm{ifelse}(\pi, g_1(\u_1), g_0(\u_0)) \\ &= \pi \cdot g_1(\u_1) + (1-\pi) \cdot g_0(\u_0). \end{align*}$$ We obtain that the Jacobians are $$\partial_{\u_1} f(\pi, \u_1, \u_0) = \pi \cdot \partial g_1(\u_1)$$ and $$\partial_{\u_0} f(\pi, \u_1, \u_0) = (1-\pi) \cdot \partial g_0(\u_0).$$ As long as $g_1$ and $g_0$ are differentiable functions, we can therefore differentiate through the branch variables $\u_1$ and $\u_0$ without any issue. More problematic is the predicate variable $\pi$, as we now discuss.