Convolutional Neural Networks

Vismay Shanbhag

Lecture notes on convolutional neural networks, pooling, feature hierarchies and batch normalization.

Every figure can be played, or stepped through with Next.

Part 1Why Not Dense Layers for Images?

Loss of spatial structure: Flattening 2D images into 1D vectors destroys pixel adjacency, discarding shapes, lines, and textures.
Parameter explosion: Connecting every pixel to hidden units produces millions of weights, causing severe memory overhead and overfitting.
No spatial weight sharing: Shifting an object forces dense layers to relearn it from scratch, as weights are tied to fixed pixel coordinates.

Part 2The Convolution Operation & ReLU

Filter (Kernel): A small matrix of weights (e.g. $3 \times 3$ ) that slides over pixels to detect visual features.
Stride (S): The step size in pixels the filter shifts horizontally and vertically across the image.
Padding (P): Extra border pixels (usually zeros) added around image edges to prevent spatial shrinkage.
Feature Map: The 2D output grid storing the dot-product response at each filter position.
ReLU: Non-linear activation $ReLU(x) = max(0, x)$ that zeros out negative values.

Image Scene: Filter (Kernel): Stride: Padding: Apply ReLU: max(0, x)

1. Visual Picture (6×6)

2. Pixel Numbers (0-255)

∗

3. Filter (3×3)

4. Output Map (4×4)

Click 'Play' or 'Next' to slide the 3×3 filter and calculate dot products.

Step 0 / 16

Output Size = ⌊ W − K + 2P S ⌋ + 1

Formula Breakdown: Input size

W = 6

, Filter size

K = 3

, Padding

P = 0

, Stride

S = 1

.
Plugging in the numbers:

⌊ (6 - 3 + 0) / 1 ⌋ + 1 = 3 + 1 = 4 \times 4 output map

Part 2bRGB 3-Channel Convolution

3D input volumes: Color images consist of 3 stacked 2D channels ( $H \times W \times 3$ for Red, Green, and Blue).
Cross-channel summation: Dot products across all 3 color planes sum together into a single 2D feature map.

Image Preset: 3D Filter Preset: Channel View: Apply ReLU: max(0, x)

■ Red Channel (8×8)

■ Green Channel (8×8)

■ Blue Channel (8×8)

Combined RGB (8×8)

ONE Shared 3D Filter (

3 \times 3 \times 3

= 27 weights): Reused across all 36 patch positions (click weights to edit)

Output Map (6×6)

Click 'Play' or 'Next' to slide the 3×3×3 filter volume over the RGB image.

Step 1 / 36

Part 3Feature Hierarchies

Hierarchical abstraction: Early layers detect raw visual primitives; deeper layers compose them into complex semantics.
Layer 1 (Low-level): Extracts oriented edges, color-opponent contrasts, and basic gradient bars.
Layer 2 (Mid-level): Combines edges into corners, textures, curves, and surface patterns.
Layer 3+ (High-level): Assembles mid-level contours into recognizable object parts (wheels, windows, faces).

CNN Feature Hierarchy Flow: Yellow Car to Low, Mid, High Features and Classifier

End-to-End Visual Hierarchy: An input image (yellow car) passes through successive convolution layers, building representations from oriented edges to complex semantic parts.

Layer 1 • Low-Level

Edges & Opposing Colors

Layers 2–3 • Mid-Level

Textures, Curves & Motifs

Layers 4–5 • High-Level

Object Parts & Assemblies

Part 4Pooling & Spatial Downsampling

Spatial downsampling: Reduces dimensions ( $H, W$ ) by selecting local window maximums for translation invariance, preserving channel depth ( $C$ ) with zero learnable parameters.

Pooling Mode:

Feature Map (6×6)

→

Pooled Output (3×3)

Click 'Play' or 'Next' to step the 2×2 pooling window.

Ready. Press 'Play' or 'Next' to downsample the feature map.

Step 0 / 9

Part 5Batch Normalization

Internal covariate shift: Upstream parameter updates cause downstream feature distributions to drift continuously.
Mini-batch standardization: Normalizes each feature dimension across the batch to zero mean and unit variance.

Drift Scenario:

Training Iteration: Iteration 1 / 6 (8 batch activation points traveling through Layer 2)

μ = 0.0

Batch Mean (μ)

0.0

Batch Spread (σ)

1.0

Downstream Layer 2 Status

Optimal Inputs

Click 'Play' or drag the iteration slider to watch the activations drift across training updates.

Iteration 1 / 6

Standardization step: Subtracts batch mean $μ B$ , divides by standard deviation $σ B$ , then applies $γ$ and $β$ .

Batch Preset:

Normalized = γ × x − μ_B √(σ_B² + ε) + β

μ = 22.3

Mean (μ_B): 22.3

Spread (σ_B): 20.6

Current Step: 1. Raw Inputs

Ready. Click any step below or press 'Auto Play' to see how BatchNorm standardizes wild activations into stable numbers.

Step 1 / 4

Part 5bBatchNorm in CNNs (4D Tensors)

4D tensor dimensions: Convolutional activations form a 4D tensor of shape $N \times C \times H \times W$ .
Channel-wise pooling: Computes one mean $μ c$ and variance $σ c 2$ across all $N \times H \times W$ activations for each channel.
Consistent spatial scaling: Adheres to convolutional weight sharing by scaling all pixels in a channel identically.

Inspect Channel:

Mini-Batch of

N = 4

Images ×

H \times W = 3 \times 3

Activations (36 total values in Channel 0):

Activation Distribution (Channel 0) Mean: +14.2 • Std: 5.8

Step 1 (Raw Activations): Showing mini-batch activations across N=4 images. Notice how the channel is off-center with wide variance.

Step 1 / 5

Optimization conditioning: Smooths the loss surface, allowing substantially higher learning rates without divergence.
Gradient health: Mitigates vanishing and exploding gradients throughout deep network architectures.

Training Dynamics: With vs. Without BatchNorm

Drag the learning rate slider to see how BatchNorm prevents gradient explosions:

Learning Rate (η): 0.010

                With BatchNorm: Stable convergence (Final Loss: 0.08)

                Without BatchNorm: Slow descent (Final Loss: 0.35)

— With BatchNorm — Without BatchNorm