Vector spaces

A vector space is a set where you can add elements and scale them by numbers, and where those two operations obey the rules you already use without thinking. Nothing else: no lengths, no angles, no coordinates, no arrows. Every model in machine learning lives in one, because data, weights, gradients and even functions can be added and scaled.

Needs: F1 Sets, families and intersections. Used by: LA2 Span, and everything after.

The numbers: fields

You scale vectors by numbers, so first we fix what “numbers” are allowed to be.

Definition LA1.1Field#

A field is a set $\mathbb{F}$ with two operations $\mathbb{F} \times \mathbb{F} \to \mathbb{F}$, addition and multiplication, such that:

• addition and multiplication are associative and commutative;
• there exist elements $0 \neq 1$ in $\mathbb{F}$ such that $a + 0 = a$ and $1 \cdot a = a$ for all $a$;
• every $a$ has an additive inverse $-a$ with $a + (-a) = 0$, and every $a \neq 0$ has a multiplicative inverse $a^{-1}$ with $a a^{-1} = 1$;
• multiplication distributes over addition: $a(b + c) = ab + ac$ for all $a, b, c$.

Compare: Axler, LADR 4e, “Digression on fields” (ch. 1A)

Set	Field?	Why
$\mathbb{Q}$, $\mathbb{R}$, $\mathbb{C}$	yes	all the rules hold
$\mathbb{F}_2 = \{0, 1\}$, arithmetic mod 2	yes	$1 + 1 = 0$; check the rest by hand
$\mathbb{Z}$	no	$2$ has no inverse: $\tfrac{1}{2} \notin \mathbb{Z}$
float64 numbers	no	addition is not associative (below)

Convention LA1.2F means ℝ or ℂ#

On this site $\mathbb{F}$ stands for $\mathbb{R}$ or $\mathbb{C}$ unless a page says otherwise. Elements of $\mathbb{F}$ are called scalars.

Full explanationWhy a field, and why your computer doesn't have one

The field rules are exactly what you need to solve for things. Linear algebra constantly finds unknown coefficients, and finding them means dividing. Integers fail here: you cannot solve $2a = 1$ in $\mathbb{Z}$.

Floating-point numbers also fail, in a way that matters in practice. In float64,

$$(0.1 + 0.2) + 0.3 = 0.6000000000000001, \qquad 0.1 + (0.2 + 0.3) = 0.6.$$

Associativity is broken by rounding. In float32, $(10^8 + 1) - 10^8 = 0$ while $(10^8 - 10^8) + 1 = 1$: added to the large number first, the $1$ is absorbed by rounding. This is why summing the same numbers in a different order (on another GPU, or with another batch split) can give slightly different results, and why numerical code is written to keep rounding under control. The mathematics on this site is exact, over $\mathbb{R}$ or $\mathbb{C}$; computers approximate it.

Who uses which field. $\mathbb{R}$ is the default in machine learning. $\mathbb{C}$ appears wherever there are rotations and frequencies: Fourier transforms, signal processing, quantum computing, and rotary position embeddings in transformers, which rotate pairs of coordinates by an angle that grows with position (the same thing as multiplying complex numbers by $e^{i\theta}$). $\mathbb{F}_2$ is the field of bits: error-correcting codes, cryptography and hashing do linear algebra over it.

The definition

Definition LA1.3Vector space#

A vector space over $\mathbb{F}$ is a set $V$ together with

• an addition $V \times V \to V$, written $(u, v) \mapsto u + v$, and
• a scalar multiplication $\mathbb{F} \times V \to V$, written $(a, v) \mapsto av$,

such that for all $u, v, w \in V$ and all $a, b \in \mathbb{F}$:

	Axiom	Name
VS1	$u + v = v + u$	commutativity
VS2	$(u + v) + w = u + (v + w)$	associativity
VS3	there is $0 \in V$ with $v + 0 = v$ for all $v$	additive identity
VS4	for each $v$ there is $w \in V$ with $v + w = 0$	additive inverse
VS5	$(ab)v = a(bv)$	compatibility of scaling
VS6	$1v = v$	scaling by one
VS7	$a(u + v) = au + av$	distributivity over vectors
VS8	$(a + b)v = av + bv$	distributivity over scalars

Elements of $V$ are called vectors.

Compare: Axler, LADR 4e, Definition 1.20; Hoffman & Kunze, §2.1

Full explanationHow to read the definition

The operations are part of the data. A vector space is not just a set; it is a set with a particular addition and scaling. The same set with different operations can be a vector space or not (see LA1.8 and LA1.9).

Closure is built in. Writing addition as a function $V \times V \to V$ already says that the sum of two vectors is again in $V$. Some books list “closure” as two extra axioms. Here it lives in the arrows.

What is not there. Nothing in the definition mentions length, angle, distance, coordinates, multiplying two vectors together, or infinite sums. Those are extra structure added later (norms, inner products, bases). Keeping them out is what lets the same theorems apply to arrows in the plane, to polynomials and to probability-weighted functions.

“Vector” is a role, not a shape. A vector is anything that lives in a vector space. A function, a matrix or an image is a vector as soon as you put it in a space where it can be added and scaled.

Examples: one space that contains the others

Example LA1.4Fⁿ#

$\mathbb{F}^n$, the $n$-tuples of scalars, with componentwise operations: $(x_1, \dots, x_n) + (y_1, \dots, y_n) = (x_1 + y_1, \dots, x_n + y_n)$ and $a(x_1, \dots, x_n) = (ax_1, \dots, ax_n)$. The zero vector is $(0, \dots, 0)$.

Rather than checking eight axioms again for matrices, sequences, images and functions, prove them once for the space that contains all of these as special cases.

Theorem LA1.5Functions into a field form a vector space#

For any set $X$, the set $\mathbb{F}^X$ of all functions $X \to \mathbb{F}$ is a vector space with pointwise operations:

$$(f + g)(x) := f(x) + g(x), \qquad (af)(x) := a\, f(x).$$

Its zero vector is the constant function $0$, and the additive inverse of $f$ is $x \mapsto -f(x)$.

Proofof LA1.5

Each axiom, evaluated at a point $x \in X$, becomes a rule of the field $\mathbb{F}$. For VS2: $((f + g) + h)(x) = (f(x) + g(x)) + h(x) = f(x) + (g(x) + h(x)) = (f + (g + h))(x)$, using associativity in $\mathbb{F}$. Two functions that agree at every point are equal, so $(f + g) + h = f + (g + h)$. VS1 and VS5–VS8 go the same way, each using the matching field rule. VS3: $(f + 0)(x) = f(x) + 0 = f(x)$. VS4: $(f + (-f))(x) = f(x) - f(x) = 0$.

Now choose the index set $X$ (recall F1, “A vector is a function”):

Choose $X$	$\mathbb{F}^X$ is	Who uses it
$\{1, \dots, n\}$	$\mathbb{F}^n$	every feature vector, embedding and gradient
$\{1, \dots, m\} \times \{1, \dots, n\}$	$m \times n$ matrices	weight matrices, covariance matrices, attention scores
a pixel grid $\times \{\text{R, G, B}\}$	colour images	computer vision
$\mathbb{N}$	infinite sequences	signals, time series, generating functions
$\mathbb{R}$ or $[0, 1]$	real functions	function approximation, kernels, Gaussian processes

Example LA1.6Polynomials#

$\mathcal{P}(\mathbb{F})$ is the set of polynomial functions $p(x) = a_0 + a_1 x + \dots + a_d x^d$ with coefficients in $\mathbb{F}$, and $\mathcal{P}_d(\mathbb{F})$ those of degree at most $d$ (including the zero polynomial). With the usual addition and scaling of functions both are vector spaces: in fact they sit inside $\mathbb{F}^{\mathbb{F}}$ as subspaces (LA1.13).

Full explanationWhere polynomials fit, and a trap over small fields

Polynomials are the first function space most people meet, and the right picture is a subspace of the big space $\mathbb{F}^{\mathbb{F}}$ of all functions: the ones you can build from $1, x, x^2, \dots$ with finitely many additions and scalings. LA2 makes “build from” precise; it is called the span.

They matter because they are the functions a computer can evaluate exactly with $+$ and $\times$. Taylor expansions approximate smooth functions by polynomials; polynomial regression fits $\mathcal{P}_d(\mathbb{R})$ to data; the polynomial kernel $k(x, y) = (x^\top y + 1)^d$ in kernel methods and SVMs corresponds to a feature space of polynomials. Over finite fields, polynomials are the backbone of Reed–Solomon codes, which protect data on QR codes and storage disks.

A trap. A polynomial (a list of coefficients) and a polynomial function (what it does to inputs) are the same thing over $\mathbb{R}$ and $\mathbb{C}$, because a non-zero polynomial has only finitely many roots. Over $\mathbb{F}_2$ they differ: $x$ and $x^2$ are different polynomials, but as functions on $\{0, 1\}$ they are identical, since $0^2 = 0$ and $1^2 = 1$. This is why Axler restricts to $\mathbb{R}$ and $\mathbb{C}$ when he defines polynomials as functions.

Example LA1.7The zero space#

$\{0\}$, a single vector with $0 + 0 = 0$ and $a0 = 0$, is a vector space. It is the smallest one, and it turns up as the span of nothing (LA2).

What is not a vector space

A definition is understood when you can break it. Each non-example below breaks the definition at a specific point, often closure (the arrows in LA1.3). Example 2 isolates a single axiom, which shows that axiom is doing real work.

Non-example LA1.8Broken examples, one axiom at a time#

1. The positive reals $(0, \infty)$ with ordinary $+$ and scaling. Scaling by $-1$ leaves the set: $(-1) \cdot 2 = -2$. Scalar multiplication is not a function $\mathbb{F} \times V \to V$. There is no zero vector either: with ordinary addition, $v + e = v$ forces $e$ to be the real number $0$, which is not in $(0, \infty)$, so VS3 fails.
2. $\mathbb{R}^2$ with ordinary addition but $a \odot (x, y) := (ax, 0)$. VS5, VS7 and VS8 still hold (check them), but VS6 fails: $1 \odot (1, 1) = (1, 0) \neq (1, 1)$. So VS6 is not a consequence of the others.
3. $\mathbb{Z}^n$ ($n \ge 1$) as a space over $\mathbb{R}$. $\tfrac{1}{2} \cdot (1, 0, \dots, 0) \notin \mathbb{Z}^n$: scaling leaves the set.
4. Probability vectors $\{p \in \mathbb{R}^n : p_i \ge 0, \ \sum_i p_i = 1\}$ with ordinary operations. Adding two of them gives total $2$, not $1$; and with ordinary addition the zero vector would have to be $(0, \dots, 0)$, which is not in the set.
5. Unit vectors $\{v \in \mathbb{R}^n : \|v\| = 1\}$, with $\|v\|$ the Euclidean length and $n \ge 1$. The sum of two unit vectors is rarely a unit vector, and the zero vector (forced to be $(0, \dots, 0)$, as in 4) is missing.

Example LA1.9The same set, different operations: now it is a vector space#

Take the positive reals again, as a space over $\mathbb{R}$, but define “addition” as $x \oplus y := xy$ and “scaling” as $a \odot x := x^a$. Every axiom holds. The zero vector is $1$ (since $x \oplus 1 = x$), the inverse of $x$ is $1/x$, and VS8 reads $x^{a+b} = x^a x^b$.

Full explanationWhat the positive-reals example teaches

The set did not change; the operations did, and that decided everything. The logarithm translates this strange space into ordinary $\mathbb{R}$: $\log(xy) = \log x + \log y$ and $\log(x^a) = a \log x$. Products become sums and powers become scalings. Later you will call such a translation an isomorphism.

You use this translation constantly in machine learning: likelihoods are products of probabilities, and taking logs turns them into sums of log-probabilities, which are easier to differentiate and do not underflow. Non-examples 4 and 5 matter for the same reason. Softmax outputs live on the probability simplex, and normalised embeddings live on the unit sphere. Neither is a vector space. On the simplex, sums, differences and scalings leave the set, although averages stay inside (the simplex is convex); on the unit sphere even averages leave it. That is why models apply softmax or re-normalise after doing linear algebra in the full space $\mathbb{R}^n$.

What the axioms already imply

Some facts look too obvious to prove. Proving them is how you find out which axioms they depend on, and it is the first practice at reasoning from definitions alone.

Theorem LA1.10First consequences#

In every vector space $V$, for all $v \in V$ and $a \in \mathbb{F}$:

1. the zero vector is unique;
2. each $v$ has exactly one additive inverse, written $-v$;
3. $0v = 0$ (scalar zero times $v$ is the zero vector);
4. $a0 = 0$;
5. $(-1)v = -v$.

Proofof LA1.10

1. Let $0$ and $0'$ both satisfy VS3. Then $0' = 0' + 0 = 0 + 0' = 0$, using VS3 for $0$, VS1, then VS3 for $0'$.
2. Let $w$ and $w'$ both be inverses of $v$. Then $w = w + 0 = w + (v + w') = (w + v) + w' = 0 + w' = w'$, using VS3, VS2 and VS1.
3. $0v = (0 + 0)v = 0v + 0v$ by VS8. Add $-(0v)$ to both sides and simplify with VS2, VS4 and VS3: $0 = 0v$.
4. $a0 = a(0 + 0) = a0 + a0$ by VS7; add $-(a0)$ to both sides as in 3.
5. $v + (-1)v = 1v + (-1)v = (1 + (-1))v = 0v = 0$, using VS6, VS8 and part 3. So $(-1)v$ is an inverse of $v$, and by part 2 it is the inverse, $-v$.

Check In the strange space of LA1.9 (positive reals over ℝ, x ⊕ y = xy, a ⊙ x = x^a), what does part 3 of LA1.10, 0v = 0, say in ordinary notation?

The scalar $0$ times $x$ is $x^0 = 1$, and the zero vector of this space is $1$. So "$0v = 0$" becomes $x^0 = 1$: a familiar fact, here forced by the axioms.

Subspaces

Definition LA1.11Subspace#

A subset $U \subseteq V$ is a subspace of $V$ if $U$ is itself a vector space with the addition and scalar multiplication of $V$.

Theorem LA1.12Subspace test#

$U \subseteq V$ is a subspace if and only if

1. $0 \in U$;
2. $u, w \in U$ implies $u + w \in U$;
3. $a \in \mathbb{F}$ and $u \in U$ implies $au \in U$.

Compare: Axler, LADR 4e, §1C; Hoffman & Kunze, §2.2, Theorem 1

Proofof LA1.12

($\Rightarrow$) If $U$ is a subspace, its operations are those of $V$ and land in $U$, which gives 2 and 3. $U$ has its own zero $0_U$ with $0_U + 0_U = 0_U$. Adding the inverse of $0_U$ in $V$ to both sides gives $0_U = 0$, so $0 \in U$.

($\Leftarrow$) By 2 and 3 the operations of $V$ restrict to functions $U \times U \to U$ and $\mathbb{F} \times U \to U$. Axioms VS1, VS2 and VS5–VS8 hold for all vectors of $V$, so in particular for those in $U$. VS3 holds by 1. For VS4, take $u \in U$: by 3, $(-1)u \in U$, and $(-1)u = -u$ by LA1.10, so $u$ has its inverse inside $U$.

Example LA1.13Subspaces, and look-alikes that aren't#

Subspaces. $\{0\}$ and $V$ itself. Lines and planes through the origin in $\mathbb{R}^3$. The solution set $\{x \in \mathbb{R}^n : Ax = 0\}$ of a homogeneous linear system. $\mathcal{P}_d(\mathbb{F}) \subseteq \mathcal{P}(\mathbb{F}) \subseteq \mathbb{F}^{\mathbb{F}}$. Continuous functions inside $\mathbb{R}^{\mathbb{R}}$. Sequences with only finitely many non-zero terms inside $\mathbb{F}^{\mathbb{N}}$.

Not subspaces. A line that misses the origin, such as $\{(x, y) : y = x + 1\}$ (no $0$). The solutions of $Ax = b$ with $b \neq 0$ (no $0$). The union of the $x$-axis and the $y$-axis (not closed under addition). The unit disc (not closed under scaling). Polynomials of degree exactly $2$ ($x^2 + (-x^2) = 0$ has no degree $2$).

Theorem LA1.14Intersections of subspaces are subspaces#

The intersection of any non-empty family of subspaces of $V$ is a subspace of $V$.

Compare: Hoffman & Kunze, §2.2, Theorem 2

Proofof LA1.14

Let $\mathcal{U}$ be a non-empty family of subspaces and $W = \bigcap \mathcal{U}$. Apply the subspace test. (1) Every $U \in \mathcal{U}$ contains $0$, so $0 \in W$. (2) If $u, w \in W$, then $u, w \in U$ for every $U \in \mathcal{U}$; each $U$ is closed under addition, so $u + w \in U$ for every $U$, hence $u + w \in W$. (3) Scaling is identical.

Unions do not behave this way: the two axes are subspaces, their union is not. To get the smallest subspace containing both, you intersect all the subspaces that contain them, exactly as in F1.15. That subspace is the span, the subject of LA2.

Where this shows up in AI

Null spaces are blind spots. For a linear layer $x \mapsto Wx$, the subspace $\{x : Wx = 0\}$ is the set of input directions the layer cannot see: adding any of them to an input changes nothing downstream. Affine sets are subspaces shifted. The solutions of $Ax = b$ form a subspace moved away from the origin, which is why constraints like “the weights sum to one” are handled differently from “the weights are orthogonal to $v$”. Models are vectors. Gradient boosting adds functions (trees) together and scales them by a learning rate: it is gradient descent in a space of functions, $\mathbb{R}^X$.

Check Is {(x, y, z) ∈ ℝ³ : x + y + z = 0} a subspace? And {(x, y, z) : x + y + z = 1}?

The first is: it contains $0$, and sums and multiples of solutions are solutions. It is a plane through the origin. The second is not: $0$ is missing ($0 + 0 + 0 \neq 1$). It is the same plane shifted.

Check ℂ is a vector space over ℂ and also over ℝ. Is ℝ a subspace of ℂ?

It depends on the field. Over $\mathbb{R}$, yes: real multiples of reals are real. Over $\mathbb{C}$, no: $i \cdot 1 = i \notin \mathbb{R}$. “Subspace” always means subspace over a given field.

Check In non-example LA1.8(2), verify that VS7 holds: a ⊙ (u + v) = a ⊙ u + a ⊙ v.

With $u = (x_1, y_1)$, $v = (x_2, y_2)$: $a \odot (u + v) = (a(x_1 + x_2), 0) = (ax_1, 0) + (ax_2, 0) = a \odot u + a \odot v$. Only VS6 breaks.

Sources

• Sheldon Axler, Linear Algebra Done Right, 4th ed. (open access, 2024): fields (ch. 1A), Definition 1.20 (vector space), §1C (subspaces).
• Kenneth Hoffman & Ray Kunze, Linear Algebra, 2nd ed. (1971): §2.1 (vector spaces), §2.2 Theorems 1–2 (subspace test, intersections of subspaces).
• Deisenroth, Faisal & Ong, Mathematics for Machine Learning (2020): §2.4 (vector spaces), for a second, more applied angle.