Python, NumPy
& Tensor Programming.
Unpack the operations inside a neural network. Understand each number, shape and multiplication by hand—then assemble them into fast NumPy and PyTorch code.
Why arrays are the language of neural networks
A deep learning model does not see a photograph as a dog or a sentence as a thought. It sees numbers arranged in shapes. Layers transform those numbers through mathematical operations. Our goal is to open these operations, inspect their moving parts, then rebuild them correctly.
Hours studied: 2
Attendance: 0.8
Assignments: 5
Three numbers describe one record.
Three records, each containing three features.
Python primitives → reusable model components
Before using neural-network frameworks, understand why we use functions, loops and classes. Python lists store objects; adding two lists concatenates them rather than adding elements.
import numpy as np # Fast operations over numerical arrays
first = [1, 2, 3]
second = [4, 5, 6]
print(first + second) # [1,2,3,4,5,6]: list concatenation
x = np.array(first, dtype=np.float64) # Make numeric arrays
y = np.array(second, dtype=np.float64)
print(x + y) # [5. 7. 9.]: element-wise addition
def predict(hours: float, weight: float, bias: float) -> float:
# Express the mathematical rule y_hat = w*x + b
return weight * hours + bias
class LinearPredictor:
def __init__(self, weight: float, bias: float):
self.weight = weight # Store model parameter
self.bias = bias # Store intercept
def forward(self, x):
return self.weight * x + self.bias
model = LinearPredictor(weight=5.0, bias=30.0)
print(model.forward(6)) # 60.0__init__ prepares persistent state. forward computes a result. The object remembers the parameters between calls.
PyTorch’s nn.Module builds on the same class pattern, adding parameter registration, training modes, and other framework behavior.
Tensors: dimensions, axes and shapes
A scalar is one number. A vector is a one-dimensional array. A matrix is two-dimensional. “Tensor” is the general name for an array with any number of axes.
Read (32,10) as 32 records × 10 features per record. Always write shapes before writing operations.
Live lab · Shape transformer
Select a target shape. Watch the same 12 numbered values rearrange without gaining or losing elements.
import numpy as np
x = np.arange(12) # Integers 0 through 11
print(x.ndim) # 1: number of axes
print(x.shape) # (12,): lengths of the axes
print(x.size) # 12: total elements
print(x.dtype) # Usually int64 or int32, platform-dependent
matrix = x.reshape(3, 4) # 3 rows, 4 columns
print(matrix)
print(matrix[0, :]) # First row (four values)
print(matrix[:, 1]) # Second column (three values)
print(matrix.mean(axis=0)) # Column means, shape (4,)
print(matrix.mean(axis=1)) # Row means, shape (3,)
# Common bug: (3,) minus (3,1) broadcasts into (3,3).
y_hat = np.array([1, 2, 3])
y = np.array([[1], [2], [3]])
print((y_hat - y).shape) # (3,3), NOT (3,)Broadcasting: one rule reused across a batch
Disassemble: Each of three students receives the same two-feature bias. Instead of writing a loop, NumPy lines up dimensions from the right. An aligned pair is compatible if its sizes are equal or one is 1. Missing leading axes behave as though their size is 1.
Live lab · Broadcasting the bias
Change the two bias values. See every row update, and compare the repeated bias with the output.
Reassemble: This is exactly how a neural network applies a single learned bias vector to every sample in a batch.
import numpy as np
X = np.array([[1, 2], [3, 4], [5, 6]]) # Shape (3,2)
b = np.array([10, 5]) # Shape (2,)
print(X + b) # Add bias to every row: shape (3,2)
# Shapes are compared from the right, not from the left.
A = np.ones((4, 1)) # Shape (4,1)
B = np.array([1, 2, 3]) # Shape (3,)
print((A + B).shape) # (4,3)
# Incompatible rightmost dimensions 3 and 4 lead to an error.
# np.ones((2,3)) + np.ones((4,))Matrix multiplication: unroll every multiply and add
Element-wise multiplication and matrix multiplication mean different things. A single matrix multiplication output cell is a dot product: take one row, take one column, multiply corresponding entries, then add the products.
Animated lab · Open the dot product
Step through the computation of [[1,2],[3,4]] @ [[5,6],[7,8]]. Each click reveals one component of the selected cell.
For a mini-batch of 32 students, 10 inputs and a layer with 5 neurons, matrix multiplication is (32,10) @ (10,5) → (32,5). The middle dimensions must match.
import numpy as np
def matmul_from_scratch(A, B):
# Assumes nonempty rectangular lists of numbers.
rows_a, cols_a = len(A), len(A[0])
rows_b, cols_b = len(B), len(B[0])
if cols_a != rows_b:
raise ValueError("Inner dimensions must match")
out = [[0.0] * cols_b for _ in range(rows_a)]
for i in range(rows_a): # Pick output row
for j in range(cols_b): # Pick output column
for k in range(cols_a): # Walk across row and column
out[i][j] += A[i][k] * B[k][j]
return out
A = [[1, 2], [3, 4]]
B = [[5, 6], [7, 8]]
print(matmul_from_scratch(A, B)) # [[19,22],[43,50]]
print(np.array(A) @ np.array(B)) # Same result using NumPyVectorization and reduction: replacing loops with intent
Instead of telling Python how to visit every number, tell NumPy what operation to perform on the array. This reduces interpreter overhead and often uses compiled routines.
Visit each input, multiply, add, append. Readable, but repeated interpreter work for large batches.
prediction = 3 * X + 2 expresses the computation over the entire array.
mean(X, axis=1) → one result per row
For a (3,2) table, reducing axis 0 gives shape (2,). Reducing axis 1 gives shape (3,). keepdims=True preserves reduced dimensions at size one, which often makes later broadcasting easier.
import numpy as np
X = np.array([[70, 80], [60, 90], [85, 95]], dtype=float)
print(X.mean(axis=0)) # Mean for each exam: shape (2,)
print(X.mean(axis=1)) # Mean for each student: shape (3,)
column_means = X.mean(axis=0, keepdims=True) # (1,2)
centered = X - column_means # Broadcast mean to each student
print(centered.shape) # (3,2)
values = np.arange(1_000_000, dtype=float)
result = 3 * values + 2 # Vectorized linear predictionNumerical stability: when correct math fails in a computer
Mathematics assumes ideal real numbers. Computers use finite precision. For large inputs, the expression exp(x) may overflow. Stable formulas use algebra to avoid dangerous intermediate values.
Disassemble softmax
For logits [1000, 1001, 1002], direct exponentiation overflows in common floating-point formats. Subtract the largest value, 1002, from every logit. The probability distribution remains unchanged, because the common exponential factor cancels.
Live lab · Stable softmax probabilities
import numpy as np
def stable_softmax(logits, axis=-1):
logits = np.asarray(logits, dtype=np.float64)
shifted = logits - logits.max(axis=axis, keepdims=True)
exponentials = np.exp(shifted) # Largest exponential is 1
return exponentials / exponentials.sum(axis=axis, keepdims=True)
logits = np.array([[1000,1001,1002], [1,2,3]])
probs = stable_softmax(logits, axis=1)
print(probs) # Both rows have identical probabilities
print(probs.sum(axis=1)) # [1., 1.]
# For classification training in PyTorch, pass raw logits to
# torch.nn.CrossEntropyLoss; do NOT pre-apply softmax.Reassemble the pieces into one neural-network layer
Everything now fits into a single expression. Take a batch X, multiply by weights W, add a bias b, and apply ReLU:
- Input: Each row is one example, each column one feature.
X.shape = (3,3). - Weights: Four output units each connect to three inputs. Our NumPy convention is
W.shape = (3,4). - Multiply:
X @ Wforms four weighted combinations per example, shape(3,4). - Bias: Add
b.shape = (4,)to every row using broadcasting. - Activation: ReLU changes negative outputs to zero. Shape stays
(3,4).
import numpy as np
class DenseReLU:
def __init__(self, input_features, output_features, seed=42):
rng = np.random.default_rng(seed) # Independent random generator
# He-style scale: often suitable for ReLU layers
self.W = rng.normal(0, np.sqrt(2 / input_features),
size=(input_features, output_features))
self.b = np.zeros(output_features) # A bias for each output unit
def forward(self, X):
Z = X @ self.W + self.b # Linear transform; bias broadcasts
return np.maximum(0, Z) # Element-wise ReLU activation
X = np.array([[2., .8, 5.], [4., .9, 7.], [6., .95, 9.]])
layer = DenseReLU(3, 4)
output = layer.forward(X)
print(output.shape) # (3,4)
print(output)import torch
from torch import nn
torch.manual_seed(42)
X = torch.tensor([[2., .8, 5.], [4., .9, 7.], [6., .95, 9.]])
model = nn.Sequential(
nn.Linear(3, 4), # Three input features, four output units
nn.ReLU() # Rectified linear activation
)
out = model(X)
print(out.shape) # torch.Size([3, 4])
# PyTorch stores Linear weights as (out_features, in_features).
# So model[0].weight.shape == (4,3), not (3,4).
print(model[0].weight.shape)
# This is a forward pass. There is NO training loop here.Mini-project: train student-score regression end to end
We will build a small synthetic regression model from scratch. Features are study hours, attendance fraction, and completed assignments. The target is generated from a known formula plus noise, so we can verify the learning algorithm without claiming it predicts real academic performance.
L = (1/N) · Σ(ŷ − y)²
Break down the gradients
Write the error vector e = Xw + b − y. MSE differentiates into:
∂L/∂b = (2/N) Σe
Read the weight formula as “collect the error contributed by each feature across all samples.” The transpose aligns the samples with the feature dimension. We will derive this rigorously in the calculus chapter.
import numpy as np
rng = np.random.default_rng(42)
# 1. Synthetic data: 500 students, 3 explanatory variables
n = 500
X = np.column_stack([
rng.uniform(0, 10, n), # Study hours
rng.uniform(.5, 1.0, n), # Attendance fraction
rng.integers(0, 11, n) # Completed assignments
])
y = X @ np.array([4., 20., 1.5]) + 10 + rng.normal(0, 2, n)
# 2. Hold-out split: evaluate on data never used for optimization
order = rng.permutation(n)
train_idx, test_idx = order[:400], order[400:]
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# 3. Standardize with train statistics ONLY to prevent leakage
mu = X_train.mean(axis=0)
sigma = X_train.std(axis=0)
sigma = np.where(sigma == 0, 1, sigma) # Guard zero-variance columns
X_train = (X_train - mu) / sigma
X_test = (X_test - mu) / sigma
# 4. Start with a deliberately untrained linear model
w = np.zeros(X_train.shape[1])
b = 0.0
lr = 0.05
# 5. Repeatedly predict → measure error → calculate gradients → update
for epoch in range(1200):
predictions = X_train @ w + b
error = predictions - y_train
loss = np.mean(error ** 2)
dw = (2 / len(X_train)) * (X_train.T @ error)
db = 2 * error.mean()
w -= lr * dw
b -= lr * db
if epoch % 300 == 0:
print(f"epoch={epoch:4d}, train_mse={loss:.4f}")
# 6. Evaluation happens after training, on held-out examples
predictions_test = X_test @ w + b
test_mse = np.mean((predictions_test - y_test) ** 2)
print("Test MSE:", round(test_mse, 4))
# 7. The learned weights are in standardized feature coordinates
print("Standardized coefficients:", w)
print("Intercept in standardized coordinates:", b)import torch
from torch import nn
# Run the NumPy project first; X_train, X_test, y_train, y_test exist.
torch.manual_seed(42)
train_x = torch.tensor(X_train, dtype=torch.float32)
train_y = torch.tensor(y_train, dtype=torch.float32).reshape(-1, 1)
test_x = torch.tensor(X_test, dtype=torch.float32)
test_y = torch.tensor(y_test, dtype=torch.float32).reshape(-1, 1)
model = nn.Linear(3, 1) # Register trainable weight and bias
loss_fn = nn.MSELoss() # Mean squared error
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
for epoch in range(1200):
model.train() # Enter training mode
prediction = model(train_x) # Forward pass
loss = loss_fn(prediction, train_y)
optimizer.zero_grad() # Clear gradient values from last step
loss.backward() # Calculate derivatives via autograd
optimizer.step() # Move parameters against gradients
model.eval() # Evaluation mode
with torch.no_grad(): # Disable gradient tracking for inference
test_mse = loss_fn(model(test_x), test_y)
print("PyTorch test MSE:", test_mse.item())Debugging: diagnose broken shapes and learning
A model running without errors does not prove that its mathematics is correct. The most dangerous bugs are silent: unintended broadcasts, data leakage, incorrect target dimensions, or numerical instability.
Prediction is (N,), target is (N,1).
Two axes expand and compare every prediction against every target.
Make both arrays (N,) or both (N,1), intentionally.
import numpy as np
def checked_mse(prediction, target):
# Do not let broadcasting silently change the meaning of the loss.
if prediction.shape != target.shape:
raise ValueError(f"Shape mismatch: {prediction.shape} vs {target.shape}")
if not np.isfinite(prediction).all():
raise ValueError("Prediction contains NaN or infinity")
if not np.isfinite(target).all():
raise ValueError("Target contains NaN or infinity")
return np.mean((prediction - target) ** 2)
print(checked_mse(np.array([1.,2.]), np.array([1.5,2.5])))Practice lab: your implementation checklist
Before moving on, try these without consulting the solutions above. Tick completed items; the page remembers them in this browser.
Self-test: explain it like an engineer
Primary documentation & what to read next
The handbook’s explanations are original, but these official references are the source of truth for current array and framework behavior.