Notation

Table 1: Naming conventions

Notation	Description
$\cX \subseteq \RR^D$	Input space (e.g., features)
$\cY \subseteq \RR^M$	Output space (e.g., classes)
$\cS_k \subseteq \RR^{D_k}$	Output space on layer or state $k$
$\cW \subseteq \RR^P$	Weight space
$\Lambda \subseteq \RR^Q$	Hyperparameter space
$\Theta \subseteq \RR^R$	Distribution parameter space, logit space
$N$	Number of training samples
$T$	Number of optimization iterations
$\x \in \cX$	Input vector
$\y \in \cY$	Target vector
$\s_k \in \cS_k$	State vector $k$
$\w \in \cW$	Network (model) weights
$\lambdav \in \Lambda$	Hyperparameters
$\thetav \in \Theta$	Distribution parameters, logits
$\pi \in [0,1]$	Probability value
$\piv \in \triangle^M$	Probability vector

Table 2: Naming conventions (continued)

Notation	Description
$f$	Network function
$f(\cdot; \x)$	Network function with $\x$ fixed
$L$	Objective function
$\ell$	Loss function
$\kappa$	Kernel function
$\phi$	Output embedding, sufficient statistic
$\mathrm{step}$	Heaviside step function
$\logistic_\sigma$	Logistic function with temperature $\sigma$
$\logistic$	Shorthand for $\logistic_1$
$p_\thetav$	Model distribution with parameters $\thetav$
$\rho$	Data distribution over $\cX \times \cY$
$\rho_\cX$	Data distribution over $\cX$
$\muv, \sigma^2$	Mean and variance
$Z$	Random noise variable