Vector Operations: Navigating Space
1. Introduction
A vector is not just a static list of numbers; it describes a movement in space. If a vector represents “Walk 5 meters North,” then vector operations allow us to answer questions like:
- “If I walk North, then East, where do I end up?” (Addition)
- “If I walk twice as far?” (Scalar Multiplication)
- “What is the path from A to B?” (Subtraction)
In Machine Learning, these operations are the engine of learning. When a model “learns,” it is literally moving its weight vector down a hill (Gradient Descent) towards a better solution.
2. Vector Addition (The Head-to-Tail Rule)
When adding two vectors u and v, we place the tail of v at the head of u. The result (sum) is the vector from the start of u to the end of v.
- u = [ 2, 1 ]T
- v = [ 1, 3 ]T
u + v = [ 2+1, 1+3 ]T = [ 3, 4 ]T
[!TIP] ML Application: In Gradient Descent, we update our current position (weights) by adding a step vector (the negative gradient).
new_weights = old_weights + (-learning_rate * gradient)
3. Vector Subtraction
Subtracting a vector is adding its negative: u - v = u + (-v). Geometrically, u - v is the vector that points from v to u.
u - v = [ 2-1, 1-3 ]T = [ 1, -2 ]T
[!TIP] ML Application: This is used to calculate Error or Loss. If y is the prediction and t is the target (truth), the error vector is y - t. We want to make this vector as small as possible.
4. Scalar Multiplication
Multiplying a vector by a Scalar (single number) scales its magnitude without changing its direction (unless the scalar is negative, which flips it).
2 × [ 2, 1 ]T = [ 4, 2 ]T
This “stretches” or “shrinks” the vector. In ML, the Learning Rate acts as a scalar multiplier, controlling how big of a step we take.
5. Interactive Visualizer: Vector Playground
Drag the Blue (u) and Green (v) vector heads to see how operations change the Resultant Red (r) vector.
6. Summary
- Addition: Combining forces (e.g., Wind + Velocity).
- Subtraction: Finding the difference or path between two points.
- Scalar Multiplication: Scaling the magnitude without changing direction.