Fundamental Limitations on Subquadratic Alternatives to Transformers

3 Hardness of document similarity

Lemma 186 Zero cosine is exactly orthogonality

For nonzero binary vectors \(v,w\), their cosine similarity is \(0\) if and only if \(\left\langle v,w\right\rangle =0\).

Proof

Since \(v\) and \(w\) are nonzero, \(\| v\| \| w\| {\gt}0\). Dividing by this positive denominator preserves whether the numerator is zero, so \(\left\langle v,w\right\rangle /(\| v\| \| w\| )=0\) exactly when \(\left\langle v,w\right\rangle =0\).

Assuming SETH or OVC, for every \(\varepsilon {\gt}0\) there exists a constant \(c{\gt}0\) such that for every \(\gamma \geq 1\), \(\gamma \)-\(\mathrm{LSD}_{n,\ell }\) cannot be solved in \(O(n^{2-\varepsilon })\) time when \(\ell =c\log n\).

Proof

Suppose, toward contradiction, that an algorithm \(A\) solves \(\gamma \)-\(\mathrm{LSD}_{n,c\log n}\) in \(O(n^{2-\varepsilon })\) time for the constants needed by OVC. Given an OV instance, first check in \(O(n\ell )\) time whether any vector is zero; if so, answer yes. Otherwise all vectors are nonzero and the LSD objective is well-defined. Run \(A\) and inspect the returned pair. By Lemma 186, the optimum LSD value is \(0\) exactly when the OV instance has an orthogonal pair. Any multiplicative \(\gamma \)-approximation to an optimum of \(0\) must also return value \(0\), while if no orthogonal pair exists every cosine similarity is positive. Thus this decides \(\mathrm{OV}_{n,c\log n}\) in \(O(n\ell +n^{2-\varepsilon })=O(n^{2-\varepsilon '})\) time, contradicting OVC, and hence SETH through Theorem 106.

Corollary 188 LSD variant hardness, Corollary 3.2

Assuming SETH or OVC, for every \(\varepsilon {\gt}0\) there exists \(c{\gt}0\) such that \(\mathrm{LSD}_{n,\ell }\) cannot be solved in \(O(n^{2-\varepsilon })\) time for \(\ell \geq c\log n\). The same lower bound holds for bichromatic \(\gamma \)-\(\mathrm{LSD}\) for all \(\gamma \geq 1\) and for \(\mathrm{LSD}_{n,\ell ,t}\) for some \(t\in [0,1]\).

Proof

Exact LSD is at least as hard as approximate LSD because an exact optimum is a valid approximation. Bichromatic LSD is harder than monochromatic LSD by Lemma 185. If every threshold problem \(\mathrm{LSD}_{n,\ell ,t}\) were truly subquadratic, then Lemma 184 would give a truly subquadratic exact LSD algorithm, contradicting Theorem 187. Therefore at least one threshold \(t\) is hard.

Definition 189 Bichromatic additive MSD

Given \(A,B\subseteq \{ 0,1\} ^{\ell }\), \(\alpha \in [0,1]\), and an additive gap \(\delta {\gt}0\), bichromatic \(\delta \)-Additive-MSD distinguishes yes instances, where some \((a,b)\in A\times B\) has cosine similarity at least \(\alpha \), from no instances, where every \((a,b)\) has cosine similarity less than \(\alpha -\delta \).

Lemma 190 Kronecker powers amplify cosine similarity

For nonzero vectors \(v,w\) and positive integer \(q\),

\[ \frac{\left\langle v^{\otimes q},w^{\otimes q}\right\rangle }{\| v^{\otimes q}\| \| w^{\otimes q}\| } = \left(\frac{\left\langle v,w\right\rangle }{\| v\| \| w\| }\right)^q . \]
Proof

Substitute Lemma 9 into the numerator and Lemma 10 into both denominator factors. The resulting ratio is \(\left\langle v,w\right\rangle ^q/(\| v\| ^q\| w\| ^q)\), which is the \(q\)th power of the original cosine similarity.

Lemma 191 Tensor boosting for approximate MSD, Lemma B.2

If \(\gamma \)-\(\mathrm{MSD}_{n,\ell }\) in dimension \(\ell =(\log n)^{c\log n/(\log \log n)^2}\) is solvable in \(O(n^{2-\varepsilon })\) time for all constants \(c{\gt}0\) and \(\gamma \leq (1+1/\log \log n)^{\log n/(\log \log n)^2}\), then \((1+1/\log \log n)\)-\(\mathrm{MSD}_{n,(\log n)^k}\) is solvable in truly subquadratic time for every constant \(k{\gt}0\).

Proof

Given vectors in dimension \((\log n)^k\), set \(q=\log n/(\log \log n)^2\) and replace every vector \(v_i\) by \(v_i^{\otimes q}\). The new dimension is \((\log n)^{k\log n/(\log \log n)^2}\), which has the form required by the assumed algorithm. By Lemma 190, every cosine similarity is raised to the \(q\)th power. Therefore a \(\gamma \)-approximation after tensoring becomes a \(\gamma ^{1/q}\)-approximation before tensoring, and the stated bound on \(\gamma \) makes this at most \(1+1/\log \log n\).

Theorem 192 Karthik–Manurangsi reduction to additive Max-IP, Theorem B.4

If \((1+1/\log \log n)\)-\(\mathrm{MSD}_{n,(\log n)^k}\) is truly subquadratic for every constant \(k{\gt}0\), then bichromatic \((\log n)\)-Additive-Max-IP in dimension \(c\log n\) is truly subquadratic for every constant \(c{\gt}0\).

Lemma 193 Equal-weight padding preserves cross inner products

Let \(a,b\in \{ 0,1\} ^{\ell }\). Define

\[ a'=(a,1^{\ell -\| a\| _1},0^{\ell +\| a\| _1}),\qquad b'=(b,0^{\ell +\| b\| _1},1^{\ell -\| b\| _1}). \]

Then \(a',b'\in \{ 0,1\} ^{3\ell }\), each has exactly \(\ell \) ones, and \(\left\langle a',b'\right\rangle =\left\langle a,b\right\rangle \).

Proof

The appended block of \(a'\) containing ones is paired with a zero block of \(b'\), and the appended block of \(b'\) containing ones is paired with a zero block of \(a'\). Hence all appended coordinates contribute zero to the cross inner product, while the first \(\ell \) coordinates contribute exactly \(\left\langle a,b\right\rangle \). The number of ones in \(a'\) is \(\| a\| _1+(\ell -\| a\| _1)=\ell \), and the same calculation gives \(\ell \) ones for \(b'\).

Lemma 194 Equal-weight padding converts inner product gaps to cosine gaps

With \(a'\) and \(b'\) as in Lemma 193,

\[ \frac{\left\langle a',b'\right\rangle }{\| a'\| \| b'\| }=\frac{\left\langle a,b\right\rangle }{\ell }. \]

Consequently an additive Max-IP threshold \(\alpha \) and gap \(g\) become the additive MSD threshold \(\alpha /\ell \) and gap \(g/\ell \).

Proof

By Lemma 193, both padded vectors have exactly \(\ell \) binary one-coordinates, so their Euclidean norms are \(\sqrt{\ell }\). The same lemma gives \(\left\langle a',b'\right\rangle =\left\langle a,b\right\rangle \), and substituting these identities into the cosine-similarity formula gives \(\left\langle a,b\right\rangle /(\sqrt{\ell }\sqrt{\ell })=\left\langle a,b\right\rangle /\ell \). Scaling both the threshold and the additive gap by this common factor preserves the promised yes/no separation.

Lemma 195 Additive MSD and additive Max-IP equivalence, Lemma B.5

Bichromatic \((\log n)/\ell \)-Additive-MSD in dimension \(\ell =O(\log n)\) and bichromatic \((\log n)\)-Additive-Max-IP in dimension \(O(\log n)\) are subquadratic equivalent.

Proof

Given a bichromatic additive Max-IP instance \(A,B\subseteq \{ 0,1\} ^{\ell }\) with threshold \(\alpha \), replace each \(a\in A\) by \(a'=(a,1^{\ell -\| a\| _1},0^{\ell +\| a\| _1})\) and each \(b\in B\) by \(b'=(b,0^{\ell +\| b\| _1},1^{\ell -\| b\| _1})\). Then every new vector has exactly \(\ell \) ones, and Lemma 194 says that the cosine similarity of \(a',b'\) is \(\left\langle a,b\right\rangle /\ell \). Thus the additive Max-IP threshold \(\alpha \) becomes the additive MSD threshold \(\alpha /\ell \), with the additive gap scaled by \(1/\ell \).

Conversely, given an additive MSD instance, apply the same padding so all new vectors have equal norm \(\sqrt{\ell }\) while preserving cross inner products. A cosine threshold \(\alpha \) becomes the Max-IP threshold \(\alpha \ell \), and an additive cosine gap of \((\log n)/\ell \) becomes an additive inner-product gap of \(\log n\). The dimension only triples.

Theorem 196 Chen additive Max-IP hardness, Lemma B.6

If bichromatic \((\log n)\)-Additive-Max-IP in dimension \(c\log n\) is truly subquadratic for every constant \(c{\gt}0\), then \(\mathrm{OV}_{n,c'\log n}\) is truly subquadratic for every constant \(c'{\gt}0\), contradicting OVC and SETH.

A truly subquadratic algorithm for the approximate MSD regime of Theorem 198 would imply a truly subquadratic OV algorithm in logarithmic dimension.

Proof

Lemma 191 converts the assumed high-dimensional approximate MSD algorithm into a truly subquadratic \((1+1/\log \log n)\)-approximate MSD algorithm in every polylogarithmic dimension. Theorem 192 then yields a truly subquadratic additive Max-IP algorithm. Lemma 195 transfers between the additive MSD and additive Max-IP formulations with only logarithmic dimension blowup. Finally Theorem 196 converts this into a truly subquadratic OV algorithm, contradicting OVC.

Theorem 198 MSD hardness, Theorem 3.3/B.1

Assuming SETH or OVC, for every \(\varepsilon {\gt}0\) there is a constant \(c{\gt}0\) such that \(\gamma \)-\(\mathrm{MSD}_{n,\ell }\) cannot be solved in \(O(n^{2-\varepsilon })\) time when

\[ \ell \geq (\log n)^{c\log n/(\log \log n)^2} \quad \text{and}\quad \gamma \leq \left(1+\frac1{\log \log n}\right)^{\log n/(\log \log n)^2}. \]
Proof

If such an MSD algorithm existed for all constants required by OVC, then Lemma 197 would imply a truly subquadratic algorithm for \(\mathrm{OV}_{n,c'\log n}\) for every constant \(c'{\gt}0\). This contradicts OVC, and contradicts SETH through Theorem 106.

Corollary 199 MSD variant hardness, Corollary 3.4

Assuming SETH or OVC, for every \(\varepsilon {\gt}0\) there is a constant \(c{\gt}0\) such that \(\mathrm{MSD}_{n,\ell }\) cannot be solved in \(O(n^{2-\varepsilon })\) time when \(\ell \geq (\log n)^{c\log n/(\log \log n)^2}\). The same lower bound holds for bichromatic \(\gamma \)-\(\mathrm{MSD}\) in the same approximation range and for \(\mathrm{MSD}_{n,\ell ,t}\) for some \(t\in [0,1]\).

Proof

Exact MSD is at least as hard as approximate MSD. Bichromatic variants are harder by Lemma 185. If every decision threshold were truly subquadratic, Lemma 179 would make exact MSD truly subquadratic, contradicting Theorem 198.

Theorem 200 Introductory LSD lower bound, Theorem 1.1

The least-similar-document problem and its listed variants require quadratic time under SETH in the logarithmic-dimensional regime stated in the introduction.

Proof

The exact statement is the combination of Theorem 187 for approximate LSD and Corollary 188 for exact, threshold, and bichromatic variants.

Theorem 201 Introductory MSD lower bound, Theorem 1.2

The most-similar-document problem and its threshold and approximation variants require quadratic time under SETH in the quasipolylogarithmic-dimensional regime stated in the introduction.

Proof

This is precisely Theorem 198 together with Corollary 199.

Theorem 202 Bichromatic MSD logarithmic lower bound, Theorem 1.3

Assuming SETH, for every \(\varepsilon {\gt}0\) there exists \(c{\gt}0\) such that bichromatic \(\mathrm{MSD}_{n,\ell }\) cannot be solved in \(O(n^{2-\varepsilon })\) time when \(\ell =c\log n\).

Proof

The bichromatic setting avoids the monochromatic tensor-power loss. The paper routes logarithmic-dimensional bichromatic MSD through the corresponding bichromatic Max-IP hardness: equal-norm padding preserves cross inner products as cosine similarities, and the cited additive/Max-IP hardness supplies the OV contradiction in logarithmic dimension.