Neural Networks as Composed Functions | COSMO - Community of Science and Math Outreach

Computer ScienceMachine LearningNeural NetworksLinear Algebra Guide July 29, 2026

A neural network is a function built by composing simpler functions. The notation can look unfamiliar, but each layer repeats two operations: an affine transformation and a nonlinear activation.

For an input vector $x$ , one hidden layer computes

z=W_1x+b_1,\qquad h=\sigma(z),

and an output layer computes

\hat y=W_2h+b_2.

The matrices $W_1,W_2$ contain weights; the vectors $b_1,b_2$ contain biases. Training adjusts these numbers. The architecture determines which compositions are possible.

A single neuron

For $x=(x_1,x_2)$ , one neuron may compute

z=0.8x_1-0.3x_2+0.5.

With the rectified linear unit,

\operatorname{ReLU}(z)=\max(0,z),

the neuron outputs zero on one side of a line and a positive linear value on the other. Different neurons draw different boundaries. Later layers combine their outputs into more complicated regions.

Why a nonlinear activation is essential

If every layer were only linear, then two layers would collapse into one:

W_2(W_1x)= (W_2W_1)x.

Adding more linear layers would not create new kinds of functions. Nonlinear activations prevent this collapse. They let the network combine many simple pieces into a function with bends, thresholds, and interactions.

Dimensions are a useful debugging tool

Suppose $x\in\mathbb{R}^4$ , the hidden layer has 6 neurons, and the output has 3 scores. Then

W_1\in\mathbb{R}^{6\times4},\quad b_1\in\mathbb{R}^6,

and

W_2\in\mathbb{R}^{3\times6},\quad b_2\in\mathbb{R}^3.

Checking these shapes catches many mistakes before any calculation begins. The inner dimensions must match when matrices multiply.

From scores to probabilities

For a classification problem with $m$ classes, the network often produces scores $s_1,\ldots,s_m$ . Softmax converts them into positive numbers that sum to one:

p_i=\frac{e^{s_i}}{\sum_{j=1}^{m}e^{s_j}}.

These values are model outputs, not guaranteed truths. A model can assign high probability and still be wrong, especially on inputs unlike its training data.

A classroom forward pass

Give students a two-input, two-hidden-neuron network with small integer weights. Calculate $z$ , apply ReLU, then calculate the final score. Change one input and ask which hidden neurons switch on. This reveals the network as a deterministic calculation rather than a mysterious box.

Then ask what the network can represent and what evidence would be needed to trust it. Expressive power is not the same as accuracy, fairness, or reliability. Those properties must be tested on data that reflects the intended use.