Span

The span of a set of vectors $S$ is everything you can reach from $S$ using only addition and scaling. There are two ways to say this precisely:

• from below: all finite linear combinations of elements of $S$;
• from above: the intersection of every subspace that contains $S$.

They always give the same subspace (Theorem LA2.5), for any $S$: finite, infinite or empty. The word finite in the first definition is not optional, and seeing why is the main lesson of this unit.

Needs: F1 (families, intersections, support), LA1 (subspaces). Used by: linear independence, bases and dimension (next units), column spaces, least squares.

Linear combinations

Definition LA2.1Linear combination of a list#

For vectors $v_1, \dots, v_n \in V$ and scalars $a_1, \dots, a_n \in \mathbb{F}$, the vector $a_1 v_1 + \dots + a_n v_n$ is a linear combination of $v_1, \dots, v_n$. For $n = 0$ (the empty list) the combination is the empty sum, which by convention is $0$.

For a finite list that is all you need. For an arbitrary set $S$, possibly infinite, we keep every combination finite and let the coefficients be a finitely supported function on $S$ (F1.10):

Definition LA2.2Linear combination of elements of a set#

Let $S \subseteq V$. A linear combination of elements of $S$ is a vector

$$\sum_{s \in \operatorname{supp} c} c(s)\, s$$

where $c : S \to \mathbb{F}$ is finitely supported. Equivalently: a vector $a_1 s_1 + \dots + a_n s_n$ for some $n \ge 0$, some $s_1, \dots, s_n \in S$ and some scalars $a_i$. If $c = 0$ (empty support), the combination is $0$.

Full explanationWhy “finite” is part of the definition

The axioms of a vector space give you $u + v$ for two vectors. By repeating, you get any finite sum $v_1 + v_2 + \dots + v_n$, and associativity and commutativity (VS1, VS2) guarantee the order and grouping do not matter. That is all. Nothing in the axioms says what $v_1 + v_2 + v_3 + \cdots$ forever means. An infinite sum is a limit of finite sums, and limits need a notion of closeness (a norm, a metric, or more generally a topology), which a bare vector space does not have.

So even when $S$ is infinite, each linear combination uses only finitely many elements of $S$. Different combinations can use different elements, so together they can involve all of $S$. The finitely supported function $c$ is just careful bookkeeping for this: it says which finitely many vectors are used and with which coefficients, without needing to list $S$ in any order (which may be impossible: $S$ can be uncountable).

Span, defined two ways

Definition LA2.3Span, from above#

For $S \subseteq V$, the span of $S$ is

$$\operatorname{span} S := \bigcap \{ W : W \text{ is a subspace of } V \text{ and } S \subseteq W \}.$$

Compare: Hoffman & Kunze, §2.2 (this is their definition of the subspace spanned by S)

This is well defined: the family always contains $V$, so it is never empty (F1.12). Subspaces are closed under intersection (LA1.14), so by F1.15, $\operatorname{span} S$ is the smallest subspace of $V$ containing $S$: it contains $S$, and it is contained in every subspace that contains $S$.

Definition LA2.4Span, from below#

$L(S)$ is the set of all linear combinations of elements of $S$ (in the sense of LA2.2).

Theorem LA2.5The two definitions agree#

For every subset $S$ of a vector space $V$, finite, infinite or empty,

$$\operatorname{span} S = L(S).$$

Compare: Hoffman & Kunze, §2.2, Theorem 3 (stated there for non-empty S)

Proofof LA2.5

Step 1: $L(S)$ is a subspace containing $S$. Use the subspace test (LA1.12).

• $0 \in L(S)$: take $c = 0$.
• Sums: let $u = \sum c(s)\, s$ and $w = \sum d(s)\, s$, with $c, d$ finitely supported. Both sums can be written over the finite set $T = \operatorname{supp} c \cup \operatorname{supp} d$, because extra terms $0 \cdot s$ are zero (LA1.10). Then VS1, VS2 and VS8 give $u + w = \sum_{s \in T} (c(s) + d(s))\, s$, a linear combination with coefficient function $c + d$, which is finitely supported.
• Scaling: $a u = \sum (a\, c(s))\, s$ by VS7 and VS5, and $a c$ is finitely supported.
• $S \subseteq L(S)$: each $s \in S$ equals $1 \cdot s$ (VS6), a combination whose coefficient function is $1$ at $s$ and $0$ elsewhere.

Step 2: every subspace $W$ with $S \subseteq W$ contains $L(S)$. Take $u = a_1 s_1 + \dots + a_n s_n$ with $s_i \in S \subseteq W$. Each $a_i s_i \in W$ because $W$ is closed under scaling. A sum of $n$ vectors of $W$ is in $W$, by induction on $n$ using closure under addition; for $n = 0$ the sum is $0 \in W$. So $u \in W$.

Step 3: double inclusion (F1.3). By Step 1, $L(S)$ is one of the subspaces being intersected, so $\operatorname{span} S \subseteq L(S)$ (F1.14(1)). By Step 2, $L(S)$ is contained in every one of them, so $L(S) \subseteq \operatorname{span} S$ (F1.14(2)).

Example LA2.6The span of the empty set#

$\operatorname{span} \varnothing = \{0\}$.

From above: $\varnothing$ is a subset of every subspace (F1.5), so we intersect all subspaces of $V$. Each contains $0$, and $\{0\}$ is one of them, so the intersection is $\{0\}$. No convention needed.

From below: the only combination of elements of $\varnothing$ is the empty one, which is $0$, by convention.

Full explanationWhich definition is better? Both, for different jobs

The intersection (top-down) definition never mentions sums, coefficients or finiteness, so it needs no conventions: the empty set takes care of itself. It hands you minimality for free (“smallest subspace containing $S$”), which is what you use in proofs. And it generalises with no changes: intersect closed subspaces and you get the closed span (below); intersect convex sets and you get the convex hull; intersect σ-algebras and you get the σ-algebra generated by a family of events (for example by a random variable, via its preimages). Same pattern, F1.15, every time.

The combination (bottom-up) definition tells you what is actually inside. To decide whether a vector belongs to the span, you look for coefficients, which in $\mathbb{F}^n$ means solving a linear system. Computation always goes through this one.

Theorem LA2.5 is the bridge. In a proof, use whichever side is easier and cross over when needed. That is also why textbooks disagree on which to call “the” definition: Hoffman & Kunze start from the intersection; Axler works with finite lists and linear combinations throughout. Both are right because of LA2.5. (Hoffman & Kunze state their Theorem 3 for non-empty $S$ only, because their linear combinations have at least one term; allowing the empty combination, as we do, removes the exception.)

Proposition LA2.7Three working rules#

For subsets $S, T \subseteq V$:

1. if $S \subseteq T$ then $\operatorname{span} S \subseteq \operatorname{span} T$;
2. $\operatorname{span} S = S$ if and only if $S$ is a subspace;
3. $\operatorname{span}(\operatorname{span} S) = \operatorname{span} S$.

Proofof LA2.7

1. $\operatorname{span} T$ is a subspace containing $T$, hence containing $S$; by minimality it contains $\operatorname{span} S$.
2. If $S$ is a subspace, it is one of the subspaces intersected in LA2.3, so $\operatorname{span} S \subseteq S$; and $S \subseteq \operatorname{span} S$ always. Conversely, $\operatorname{span} S$ is a subspace, so if it equals $S$ then $S$ is a subspace.
3. $\operatorname{span} S$ is a subspace; apply 2 to it.

The finite case: span in Fⁿ

Example LA2.8A plane in ℝ³, and how to test membership#

Let $v_1 = (1, 0, 1)$ and $v_2 = (0, 1, 1)$. A combination is $a v_1 + b v_2 = (a, b, a + b)$, so

$$\operatorname{span}\{v_1, v_2\} = \{(x, y, z) \in \mathbb{R}^3 : z = x + y\},$$

a plane through the origin. Is $(2, 3, 5)$ in it? Yes: $5 = 2 + 3$, and indeed $2v_1 + 3v_2 = (2, 3, 5)$. Is $(1, 1, 1)$? We would need $a = 1$, $b = 1$, and then $a + b = 2 \neq 1$. No.

In matrix form: put the vectors as the columns of a matrix $A = [\,v_1 \ \cdots \ v_k\,]$. Then a linear combination is exactly $A x$ for a coefficient vector $x \in \mathbb{F}^k$, and

$$\operatorname{span}\{v_1, \dots, v_k\} = \{A x : x \in \mathbb{F}^k\},$$

the column space of $A$. Deciding whether $b$ is in the span is deciding whether $Ax = b$ has a solution. Systems of linear equations are therefore a question about spans, not the definition of linear algebra.

Where this shows up in AI

Linear regression lives in a span. With a data matrix $X$, every possible vector of predictions $Xw$ lies in the column space of $X$. Least squares picks the point of that span closest to the targets $y$: a projection, coming up in a later unit and in week 1 of the AI course. A layer can only output its column space. For $x \mapsto Wx$, the outputs are exactly $\operatorname{span}$ of the columns of $W$; low rank means a small span. LoRA fine-tunes with an update $\Delta W = BA$ where $B$ has $r$ columns, so every column of $\Delta W$ lies in the span of those $r$ vectors. Attention: each head’s output, before the output projection, is $\sum_j w_j v_j$ with softmax weights $w_j \ge 0$ and $\sum_j w_j = 1$, so it lies in the span of the value vectors, and in fact in their convex hull.

Check What is span{(1, 2)} in ℝ²? And span{(1, 0), (0, 1), (1, 1)}?

The first is the line $\{(t, 2t) : t \in \mathbb{R}\}$, that is $y = 2x$. The second is all of $\mathbb{R}^2$: already $(1, 0)$ and $(0, 1)$ reach everything, and $(1, 1)$ adds nothing new. A span does not care about redundancy; measuring redundancy is the job of linear independence, the next unit.

The infinite case

Here the word finite in LA2.2 starts to bite.

Example LA2.9Unit sequences span less than you might think#

In $V = \mathbb{F}^{\mathbb{N}}$ (all sequences), let $e_k$ be the sequence with $1$ in position $k$ and $0$ elsewhere. Then

$$\operatorname{span}\{e_k : k \in \mathbb{N}\} = \{x \in \mathbb{F}^{\mathbb{N}} : x_k \neq 0 \text{ for only finitely many } k\}.$$

In particular $(1, 1, 1, \dots)$ is not in the span, even though it looks like "$e_0 + e_1 + e_2 + \cdots$".

Proofof LA2.9

A combination $a_1 e_{k_1} + \dots + a_n e_{k_n}$ is zero outside the finite set $\{k_1, \dots, k_n\}$, so it has finitely many non-zero terms. Conversely, a sequence $x$ with finitely many non-zero terms equals $\sum_{k \in \operatorname{supp} x} x_k e_k$, a finite combination. The all-ones sequence has infinitely many non-zero terms, so it is not in the span.

Example LA2.10Powers of x span the polynomials, and no more#

In $V = \mathbb{R}^{\mathbb{R}}$, let $S = \{1, x, x^2, x^3, \dots\}$ (the functions $t \mapsto t^k$). Then $\operatorname{span} S = \mathcal{P}(\mathbb{R})$, the polynomials. The exponential $e^x$ is not in the span, although $e^x = \sum_{k \ge 0} \frac{x^k}{k!}$.

Proofof LA2.10

A linear combination of finitely many powers is a polynomial, and every polynomial is such a combination, so $\operatorname{span} S = \mathcal{P}(\mathbb{R})$. Suppose $e^x = p(x)$ for all $x$, with $p$ a polynomial of degree at most $d$. Differentiate both sides $d + 1$ times. The left side stays $e^x$, which is never zero. The right side becomes the zero function, because each differentiation lowers the degree by one. Contradiction.

The series $\sum x^k / k!$ is a statement about limits: its partial sums are polynomials, and they get closer and closer to $e^x$. Linear algebra alone cannot express “closer”. Add a way to measure distance and a new, larger object appears.

Remark LA2.11Closed span, and Hamel versus Schauder#

In a space with a norm (a notion of length, introduced in a later unit) one defines the closed span of $S$ as the intersection of all closed subspaces containing $S$: the same top-down pattern with a different family. It equals the closure of $\operatorname{span} S$: everything you can approximate arbitrarily well by finite combinations. In $C([0, 1])$, the continuous functions on $[0, 1]$ with the maximum norm, the closed span of $\{1, x, x^2, \dots\}$ is all of $C([0, 1])$ (the Weierstrass approximation theorem). So $e^x$, restricted to $[0, 1]$, is in the closed span but not in the span: the argument of LA2.10 works unchanged on $(0, 1)$.

This splits the idea of a basis in two. A Hamel basis writes every vector as a finite combination in exactly one way (the purely algebraic notion this site uses by default). A Schauder basis is a sequence $(e_k)$ such that every vector is a norm-convergent series $\sum_k a_k e_k$ with unique coefficients; it depends on the norm, and exists only in separable spaces.

Full explanationUniversal approximation is a statement about a closed span

A neural network with one hidden layer of $N$ units and a linear output computes

$$G(x) = \sum_{j=1}^{N} \alpha_j\, \sigma(y_j^\top x + \theta_j),$$

with $y_j \in \mathbb{R}^n$ and $\alpha_j, \theta_j \in \mathbb{R}$. Read it with this unit’s eyes: $G$ is a finite linear combination of functions from the set

$$S_\sigma = \{\, x \mapsto \sigma(y^\top x + \theta) : y \in \mathbb{R}^n,\ \theta \in \mathbb{R} \,\}.$$

So the functions all such networks can compute, over every width $N$, form exactly $\operatorname{span} S_\sigma$.

Cybenko’s 1989 theorem says that for a continuous sigmoidal $\sigma$ (one that tends to $1$ at $+\infty$ and $0$ at $-\infty$), these finite sums are dense in $C([0, 1]^n)$ with the maximum norm. In this unit’s language: the closed span of $S_\sigma$ is every continuous function on the cube. Any continuous target can be approximated as closely as you like by some finite network.

What it does not say is just as instructive: nothing about how large $N$ must be, nothing about how to find the weights, nothing about generalising from finite data. Density in a closed span is an existence statement. Training is a different problem.

Check Is every element of span S a combination of finitely many elements of S, even when S is uncountable?

Yes. That is LA2.5 combined with LA2.2: $\operatorname{span} S = L(S)$, and every element of $L(S)$ uses a finitely supported coefficient function.

Check The function x ↦ |x| on [−1, 1]: is it in the span of {1, x, x², …}? In the closed span under the maximum norm?

Not in the span: polynomials are differentiable everywhere, and $|x|$ is not differentiable at $0$. In the closed span, yes: $|x|$ is continuous on $[-1, 1]$, and by the Weierstrass approximation theorem polynomials approximate every continuous function on a closed interval uniformly. Being in the closed span does not make $|x|$ a power series $\sum a_k x^k$: the approximating polynomials change entirely as the accuracy improves.

Check If span S = V and S ⊆ T ⊆ V, what is span T?

$V$. By LA2.7(1), $V = \operatorname{span} S \subseteq \operatorname{span} T \subseteq V$.

Sources

• Kenneth Hoffman & Ray Kunze, Linear Algebra, 2nd ed. (1971), §2.2: the subspace spanned by $S$ defined as an intersection; Theorem 3 (equality with linear combinations, for non-empty $S$).
• Sheldon Axler, Linear Algebra Done Right, 4th ed. (open access), §2A: linear combinations and span of a finite list.
• Wikipedia, Linear span: both definitions, finite combinations for infinite sets, closed linear span.
• nLab, basis in functional analysis: Hamel versus Schauder bases.
• George Cybenko, “Approximation by superpositions of a sigmoidal function”, Mathematics of Control, Signals, and Systems 2 (1989), 303–314, Theorem 2.