From pixels to representations
Fine-tune a general image classifier to recognize 37 pet breeds, then explore how its predictions and internal representations relate to the images.
Which images have similar representations?
Each image becomes 512 numbers: one spatial average per channel in layer4. UMAP arranges those vectors in two dimensions. Nearby images have similar response patterns, with some distortion from the projection.
◎ Dark rings mark spatial coverage · scroll to zoom · drag to pan
Colors indicate true breed. Click a point, or use the image selector for overlapping points. Axes have no physical units.
Nearest images in the original 512-dimensional space
These neighbours use cosine distance in the full representation, independently of the 2D layout.
One image, a changing prediction
The same evaluation crop is passed through eight saved checkpoints. Follow how the predicted breed and its probability change as the classifier, then the full network, learns.
Breed probabilities
Probability of the true breed through training
Teal: this image’s true-breed probability. Gray dashed line: accuracy across all 740 validation images. Click a checkpoint on the curve to inspect it. Lines connect saved measurements.
Full-network fine-tuning starts after step 184. The next saved validation accuracy is lower, followed by recovery. At this transition, training begins to update the feature-producing layers as well as the classifier. What changes during fine-tuning, and how should this dip be interpreted?
This timeline changes predictions only. The image map and spatial feature views show the final checkpoint.
Probabilities compare the model’s breed scores; they are not calibrated certainty. The fixed crop isolates changes in the model from changes in the input.
Reading the network: Channels, responses and feature maps · Why ResNet uses residual blocks · What ReLU does to a response · From a representation to class scores · Scores, softmax and confidence
How this image moves through the network
Convolutions combine nearby responses across channels. Successive stages use smaller spatial grids and more channels to describe patterns over larger regions. Each stage here contains two residual blocks, which add a transformed input to a shortcut path.
Parameter tensor shapes
Each tile is one channel’s feature map. Color represents response strength on a fixed scale for that channel; brightness is not comparable between channels. These are responses, not pixel attributions.
Select a channel above
Global average: one number per channel
Each of layer4’s 512 maps has 7 × 7 cells. Averaging those 49 values gives a single channel response. Together, these averages form the vector used by both the classifier and the image map.
One bar per channel, numbered 0–511. Channels are learned response dimensions, not named objects.
A pretrained network adapted to 37 pet breeds
ResNet-18 was fine-tuned on 2,940 training images, with 740 images held out for validation. One epoch trained the new classifier alone; two further epochs fine-tuned the full network. Batch-normalization running statistics stayed frozen. The official test set remains unscored and is not included in this report.
Accuracy is the fraction of the same 740 validation crops classified correctly. The representation view uses only the final checkpoint. It describes how this trained model organizes the images, rather than how that organization developed during training.
What is shown
All 740 validation images have saved final layer4 vectors and predictions. Final-checkpoint spatial maps cover 74 images (10% of validation): two per breed from the original stratified probe panel, chosen before training. Prediction histories cover all 740 images at all eight checkpoints. Their channel maps are rendered images; pooled responses, classifier weights and scores retain their saved float32 values. Input crops use web compression for display only.
From image to breed score
From pixels to a representation
Three channels hold red, green and blue values at each pixel.
Each learned channel produces a grid of responses. A brighter cell means a stronger response at that location.
Channels jointly describe the image. Averaging each final map produces the 512 numbers used by the classifier.
Read shapes as channels × rows × columns: 256 × 14 × 14 means 256 maps with 196 cells each. Channels are learned features, not named objects. The same channel number in different layers has no shared meaning.
Each convolution learns filters that combine local positions across incoming channels. A channel is one output of this computation, with a response at every spatial location: its feature map. ReLU clips negative values to zero. Residual blocks add a shortcut to their transformed input, allowing the network to refine a representation while preserving an easier path for information and gradients. Downsampling trades spatial detail for a wider set of channels.
Layer4 still has spatial structure: 512 maps of 7 × 7 cells. Global averaging removes that spatial grid, leaving 512 values per image. The classifier uses a learned weighted sum of those values for each breed. UMAP is a separate visualization of the same vector; it does not participate in prediction.
Implementation and training
PyTorch is the tensor and automatic-differentiation library used to run the network and compute gradients. TorchVision supplies ResNet-18 with ImageNet-pretrained weights (IMAGENET1K_V1) and image transforms. The original classifier is replaced with a randomly initialized linear layer with 37 outputs.
| Data split | Official trainval partition: 2,940 training images and 740 validation images (20 per breed); seed 20261028. Official test images are excluded. |
|---|---|
| Training input | Random resized crop to 224 × 224, area scale 0.7–1.0, random horizontal flip; RGB normalization with ImageNet mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. |
| Evaluation input | Resize short edge to 256, center crop 224 × 224, same normalization. Each checkpoint sees the same crop. |
| Optimization | Cross-entropy loss; AdamW; batch size 16; weight decay 0.0001. One classifier-only epoch at learning rate 0.001, then two full-network epochs at 0.0001 for the backbone and 0.0003 for the classifier. Optimizer reset at the transition; no learning-rate scheduler. |
| Precision and normalization | FP16 autocast with gradient scaling during training; FP32 evaluation and feature extraction. Batch-normalization running statistics remain frozen during training. |
Cross-entropy penalizes assigning low probability to the true breed. Backpropagation computes how each trainable parameter affects this loss; AdamW uses these gradients to update weights, with weight decay discouraging large weights. Batch normalization rescales channels using stored statistics; freezing those statistics keeps them fixed while learned scale and offset parameters can still be fine-tuned.
What fine-tuning changes
Fine-tuning means continuing to train an already trained network on a new task. Here, ResNet-18 starts with weights learned from ImageNet images. Its convolutional layers—the backbone—already transform pixels into useful features. A new classifier head learns how to combine the final 512 features into scores for the 37 pet breeds. The network architecture and number of channels stay the same during this pilot; training changes the numerical weights that determine their responses.
| Phase | What can change? | What this means for the same image |
|---|---|---|
| Classifier only 1 epoch, steps 1–184 | The new head’s weights and biases. Backbone parameters are frozen. | Its feature maps and pooled vector stay fixed under the same evaluation crop, while the mapping from that vector to breed scores learns. |
| Full-network fine-tuning 2 epochs, steps 185–552 | Backbone and classifier parameters, including learned batch-normalization scales and offsets. Batch-normalization running means and variances remain fixed. | Its channel responses and pooled vector can change, as well as the classifier’s interpretation of them. |
“Frozen” means excluded from weight updates, not skipped during prediction. Both phases pass images through the entire network. An epoch is one pass through the training set. Only the 2,940 training images drive gradient updates; the same 740 held-out validation crops measure performance at saved checkpoints. The head is retained when fine-tuning begins, rather than initialized again. AdamW’s optimizer state is reset, and smaller learning rates are used for this second phase.
Why validation accuracy dips when fine-tuning begins
The dashed curve measures the fraction of all 740 validation images classified correctly. At the end of classifier-only training (step 184), accuracy is 85.3%. After 25 batches of full-network fine-tuning (step 209), it falls to 72.8%, then reaches 80.4% at step 284, 82.0% at step 368, and 86.9% at step 552.
A plausible explanation is temporary disruption of the features the head has just learned to use. Once the backbone is unfrozen, the head receives changing representations while both parts adapt to the pet task. The transition also resets the optimizer and changes learning rates. These events coincide with the dip; this single run does not isolate which caused it. The validation images and their evaluation crops do not change.
Training minimizes cross-entropy on training batches, not validation accuracy directly, so accuracy need not rise at every checkpoint. The final model classifies 12 more validation images correctly in total than the classifier-only model: a gain of about 1.6 percentage points. This is a modest observed improvement, not evidence that full-network fine-tuning reliably wins across seeds or training schedules. There is no matched longer classifier-only run to separate the effect of unfreezing from additional training. The eight plotted checkpoints are discrete measurements; connecting lines do not establish smooth recovery between them.
The procedure in pseudocode
split trainval into training and stratified validation sets
model = ImageNet-pretrained ResNet18 with a new 37-class head
save initial checkpoint and validation predictions
for phase in [classifier only: 1 epoch, full network: 2 epochs]:
choose trainable parameters and learning rates
initialize AdamW optimizer
for batch in shuffled training images:
crops = random crop, flip, normalize(batch)
scores = model(crops)
loss = cross_entropy(scores, true_breeds)
backpropagate loss; update trainable weights
at selected steps:
save model state
evaluate fixed validation crops without gradients
save scores and classification metrics
load final checkpoint in evaluation mode
for fixed validation crop:
capture layer4 output using a forward hook
vector = average each of the 512 spatial maps
save vector and classifier scores
if image belongs to the two-per-breed spatial sample:
capture outputs of layer1, layer2, layer3, layer4
render each channel as a feature-map image
fit UMAP to all 740 final vectors without breed labels
find neighbours by cosine distance in the original 512 dimensionsRecording and displaying responses
A forward hook observes a layer’s output during evaluation without changing its computation. Saved stage maps are taken after the second residual block’s addition and ReLU. The sample has two images per breed, selected before training. Maps use native spatial resolution and 8-bit color indices with a fixed per-channel scale across the 74-image final sample. This preserves spatial patterns with quantization; brightness cannot be compared directly across channels. Pooled vectors and classifier scores retain float32 precision. Display crops are compressed, but inference uses the original transformed input.
Accuracy counts correct top-ranked breed predictions divided by 740. Softmax probabilities compare the 37 scores and are not calibrated confidence. The checkpoint curve includes the initial model and seven training snapshots; its horizontal coordinate counts training batches. These are repeated measurements of the same validation set, not independent test results.
How to read UMAP
Breed labels color the points but do not enter the fit. Cosine distance compares patterns independently of their overall magnitude. UMAP can distort distances and create apparent groups. The map is descriptive and is not evidence of distinct breed clusters or a causal explanation of classification.
Resource measurement
The pilot benchmark compared batches of 8 and 16 using real images, with 10 warm-up steps and 50 measured forward/backward/optimizer steps. Those timings exclude image decoding and transfer because the repeated batch is already on the GPU. CUDA allocator peaks track tensor memory rather than total process VRAM; host process memory is sampled every 100 ms. These measurements characterize this run and hardware, not a general speed guarantee.
Scope and provenance
This is one pilot run from one seed. Near-duplicate screening flagged 25 cross-split candidates, including two training/validation pairs that remain unreviewed. Probabilities are model outputs, not calibrated certainty. Only stage outputs were recorded; intermediate convolutions and shortcut responses are not separately shown.