Better Neural Network Expressivity: Subdividing the Simplex

4 Geometric Interpretation by Subdivisions

Definition 46 Convex hull
#

The convex hull \(\operatorname {conv}(S)\) of a set \(S\subseteq \mathbb {R}^n\) is the smallest convex set containing \(S\).

Definition 47 Polytope

A polytope in \(\mathbb {R}^n\) is the convex hull of finitely many points of \(\mathbb {R}^n\).

Definition 48 Support function and Newton polytope

The support function of a polytope \(P\subseteq \mathbb {R}^n\) is

\[ h_P(x)=\max \{ \langle x,p\rangle :p\in P\} . \]

Conversely, when a convex positively homogeneous CPWL function \(f\) is equal to \(h_P\), the unique such polytope \(P\) is called the Newton polytope \(\operatorname {Newt}(f)\).

Lemma 49 Support functions and polytopes

Support functions of polytopes are exactly the convex, positively homogeneous CPWL functions, and the representing polytope is unique.

Lemma 50 Newton polytope of \(\operatorname {MAX}_n\)

The Newton polytope of \(\operatorname {MAX}_n\) is the simplex \(\Delta _{n-1}=\operatorname {conv}\{ e_1,\ldots ,e_n\} \subseteq \mathbb {R}^n\).

Proof

For every \(x\in \mathbb {R}^n\),

\[ h_{\Delta _{n-1}}(x) =\max \{ \langle x,p\rangle :p\in \operatorname {conv}(e_1,\ldots ,e_n)\} . \]

A linear functional attains its maximum over a simplex at a vertex, so this equals \(\max _i\langle x,e_i\rangle =\max _i x_i=\operatorname {MAX}_n(x)\). Hence \(\operatorname {Newt}(\operatorname {MAX}_n)=\Delta _{n-1}\) by uniqueness in the definition of Newton polytope.

Definition 51 Minkowski sum and convex-hull join

For subsets \(P,Q\subseteq \mathbb {R}^n\), the Minkowski sum is

\[ P+Q=\{ p+q:p\in P,\ q\in Q\} . \]

The convex hull of their union is denoted

\[ P*Q=\operatorname {conv}(P\cup Q). \]
Definition 52 The polytope classes \(\mathcal{P}_k\)

Let \(\mathcal{P}_0\) be the set of one-point polytopes in a fixed ambient dimension. For \(k\ge 0\), define

\[ \mathcal{P}_{k+1}= \left\{ \sum _i P_i*Q_i:\text{ the sum is finite and } P_i,Q_i\in \mathcal{P}_k\right\} . \]

The complexity of a polytope \(P\) is the least \(k\) with \(P\in \mathcal{P}_k\).

Lemma 53 \(\mathcal{P}_k\) is closed under finite Minkowski sums

For each \(k\), a finite Minkowski sum of polytopes in \(\mathcal{P}_k\) again belongs to \(\mathcal{P}_k\).

Proof

For \(k=0\), a finite sum of one-point polytopes is a one-point polytope. For \(k+1\), write each summand as a finite sum of joins \(P_i*Q_i\) with \(P_i,Q_i\in \mathcal{P}_k\). Concatenating all these finite lists gives a single finite sum of such joins, which is precisely an element of \(\mathcal{P}_{k+1}\).

Definition 54 Formal Minkowski difference

A polytope \(X\) is a formal Minkowski difference of polytopes \(P\) and \(Q\) if

\[ X+P=Q. \]
Lemma 55 Lemma 8: ReLU depth as formal Minkowski difference

For every polytope \(X\), the required number of hidden layers for representing \(h_X\) is the minimum \(k\) such that there exist \(P,Q\in \mathcal{P}_k\) with \(X+P=Q\).

Lemma 56 Valuation identity

If \(P\) and \(Q\) are polytopes and \(P\cup Q\) is a polytope, then

\[ (P\cup Q)+(P\cap Q)=P+Q. \]
Proof

It suffices to compare support functions, because a polytope is determined by its support function. For every direction \(u\), the support function of \(P\cup Q\) is \(\max (h_P(u),h_Q(u))\). Under the hypothesis that \(P\cup Q\) is convex, the smaller of the two exposed support values is attained on the common part \(P\cap Q\), so \(h_{P\cap Q}(u)=\min (h_P(u),h_Q(u))\). Hence

\[ h_{P\cup Q}(u)+h_{P\cap Q}(u) =\max (h_P(u),h_Q(u))+\min (h_P(u),h_Q(u)) =h_P(u)+h_Q(u). \]

Support functions add under Minkowski sum, so the two Minkowski sums have the same support function and are equal.

Lemma 57 Lemma 9: full additivity

Let \(X,Q_1,\ldots ,Q_m\) be polytopes in \(\mathbb {R}^n\) with \(X=\bigcup _{i\in [m]}Q_i\). Let \(\mathcal{T}_0\) be the collection of nonempty subsets \(S\subseteq [m]\) of even size and \(\mathcal{T}_1\) the collection of nonempty subsets of odd size. For \(Q_S=\bigcap _{i\in S}Q_i\),

\[ X+\sum _{S\in \mathcal{T}_0}Q_S =\sum _{S\in \mathcal{T}_1}Q_S. \]
Lemma 58 Lemma 10: subdivision upper bound

The required number of hidden layers for \(\operatorname {MAX}_n\) is at most the minimum \(k\) for which there are polytopes \(Q_1,\ldots ,Q_m\) satisfying \(\Delta _{n-1}=\bigcup _{i\in [m]}Q_i\) and \(Q_S=\bigcap _{i\in S}Q_i\in \mathcal{P}_k\) for every nonempty \(S\subseteq [m]\).

Proof

Apply Lemma 57 to \(X=\Delta _{n-1}\) and the cover \(Q_1,\ldots ,Q_m\). It yields

\[ \Delta _{n-1}+\sum _{S\in \mathcal{T}_0}Q_S =\sum _{S\in \mathcal{T}_1}Q_S. \]

By hypothesis every \(Q_S\) lies in \(\mathcal{P}_k\), and Lemma 53 keeps both finite sums in \(\mathcal{P}_k\). Thus \(\Delta _{n-1}\) is a formal Minkowski difference of two polytopes in \(\mathcal{P}_k\). Lemma 55 converts this into a \(k\)-hidden-layer representation of \(h_{\Delta _{n-1}}\), which is \(\operatorname {MAX}_n\) by Lemma 50.

Lemma 59 Faces preserve \(\mathcal{P}_k\) in the subdivision method

If each cell \(Q_i\) of a polyhedral subdivision lies in \(\mathcal{P}_k\) and every nonempty intersection \(Q_S\) is a face of one of the cells, then every such \(Q_S\) lies in \(\mathcal{P}_k\).

Proof

The classes \(\mathcal{P}_k\) are closed under taking faces: a face of a one-point polytope is one point; a face of a finite Minkowski sum is the sum of the corresponding exposed faces; and a face of a join is the join of the exposed faces. Induction on \(k\) proves the closure claim. Since each \(Q_S\) is assumed to be a face of some cell \(Q_i\in \mathcal{P}_k\), it also belongs to \(\mathcal{P}_k\).

Construction 60 Rhombic-pyramid subdivision of \(\Delta _3\)

Let

\[ x_1=(-1,-1,-1),\quad x_2=(1,1,-1),\quad x_3=(-1,1,1),\quad x_4=(1,-1,1), \]

and let \(x_{ij}\) be the midpoint of \(x_i\) and \(x_j\). The simplex \(\operatorname {conv}\{ x_1,x_2,x_3,x_4\} \) is subdivided into four rhombic pyramids with apex/base data

\[ \begin{array}{c|c} \text{apex} & \text{base}\\ \hline x_{12} & x_1,x_{14},x_{34},x_{13}\\ x_{12} & x_2,x_{23},x_{34},x_{24}\\ x_{34} & x_3,x_{13},x_{12},x_{23}\\ x_{34} & x_4,x_{14},x_{12},x_{24}. \end{array} \]

Each piece has the form \(p_i*Z_i\) where \(p_i\in \mathcal{P}_0\) is a point and \(Z_i\in \mathcal{P}_1\) is a rhombus.

Proof

The four vertices form an affine copy of \(\Delta _3\). Orthogonal projection to the plane perpendicular to the third coordinate sends them to the four vertices of a square. Cutting the simplex by the two coordinate planes perpendicular to the first and second coordinates separates this square into four quadrants, and lifting these cuts back to the simplex gives the four listed cells. In each cell, the four base points form a parallelogram, hence a rhombus affinely equivalent to the convex hull of two line segments and so in \(\mathcal{P}_1\). Joining that base to the listed apex gives the corresponding rhombic pyramid.

Construction 61 Subdivision of \(\Delta _4\) into \(\mathcal{P}_2\) cells

Let \(X\) be a three-dimensional face of \(\Delta _4\) and \(x_5\) the remaining vertex, so \(\Delta _4=x_5*X\). Subdivide \(X\) into the four rhombic pyramids \(p_i*Z_i\) from Construction 60. Then

\[ x_5*(p_i*Z_i)=(x_5*p_i)*Z_i\in \mathcal{P}_2 \]

gives a subdivision of \(\Delta _4\) into four full-dimensional cells in \(\mathcal{P}_2\).

Proof

Since \(\Delta _4\) is the join of \(x_5\) with the face \(X\), joining \(x_5\) to each cell of a subdivision of \(X\) subdivides all of \(\Delta _4\). For each cell \(p_i*Z_i\), the segment \(x_5*p_i\) belongs to \(\mathcal{P}_1\), and \(Z_i\) belongs to \(\mathcal{P}_1\) by Construction 60. Their join is therefore an allowed summand in the definition of \(\mathcal{P}_2\).

The four full-dimensional cells in Construction 61 have support functions corresponding to \(P_1,P_2,P_3,P_4\), and their lower-dimensional intersections have support functions corresponding to \(Q,R_{13},R_{14},R_{23},R_{24}\). Lemma 57 therefore yields the signed formula defining \(M\).

Proof

Taking support functions turns convex hulls into maxima and Minkowski sums into sums. The four cells \((x_5*p_i)*Z_i\) therefore produce the four functions \(P_1,P_2,P_3,P_4\), while the nonempty pairwise and higher intersections of the cells are precisely the lower-dimensional faces whose support functions are \(Q\) and the four \(R\) terms. The alternating inclusion-exclusion identity of Lemma 57 is exactly the signed half-sum in Definition 24.

Remark 63 Higher-dimensional geometric intuition

The rhombic-pyramid subdivision replaces a triangular piece, encoded by three point-like alternatives, by one parallelogram-like object of lower polytope complexity. Algebraically, this is the passage from \(T_{a,b+4}\) to \(\mathcal{T}_{a+1,b+1}\) in Claim 5. Repeating the operation reduces the number of alternatives by a factor of three per layer, producing the \(\log _3\) upper bound of Theorem 40.

Proof

Construction 60 is the geometric move: a triangle in the join description is replaced by a parallelogram cell. Claim 5 is the algebraic encoding of the same move, converting four extra scalar alternatives into one new paired-sum block and one extra scalar. Claim 7 then implements the paired-sum blocks by one additional ReLU layer. Iterating these two steps gives the depth bound of Theorem 40.

Conjecture 64 Open constant-depth maximum question

It remains open whether a constant number of hidden layers, perhaps even two hidden layers, suffices to compute \(\operatorname {MAX}_m\) for all \(m\).

Conjecture 65 Open two-layer case for \(\operatorname {MAX}_6\)

It remains open whether \(\operatorname {MAX}_6\) can be represented with two hidden layers.