← Research & Papers

1 Introduction

Street images carry many cues to where they were taken: vegetation, road markings, architecture, signage, the colour of the soil. Telling countries apart from them is a hard fine-grained problem, and large geolocation systems rely on very large models and datasets (Weyand et al. 2016). This project had the opposite constraints: 10,800 labelled course images, no external data, a limit of five million parameters, and training from scratch.

Under these constraints we fixed a small backbone early and studied the training procedure instead: input resolution, augmentation, dropout, mixup, the optimiser, the length of the schedule, and how the output is formulated. After the first recipe change, every run changes one variable against a named reference, so each difference can be attributed. The aim was to learn which choices matter for a small network, and which apparent gains disappear under a fairer comparison.

2 Data and evaluation

The data are 10,800 labelled street images from 18 countries, 600 per country, each with the latitude and longitude at which it was taken. We split them 90/10, stratified by country with a fixed seed: 9,720 training and 1,080 validation images, exactly 60 per country, so per-country counts are directly comparable. A further 2,700 images form the course’s hidden test set; their labels were never available to us, so every number here is validation accuracy on the fixed split. Chance is $1/18 = 0.056$, and one validation image is worth 0.0009.

3 Model and training

Architecture. The backbone (Figure 1) follows the VGG pattern (Simonyan and Zisserman 2015): four blocks of two 3$\times$3 convolution–batch normalisation–ReLU layers (Ioffe and Szegedy 2015) and max-pooling, with widths 48, 96, 192 and 256. Global average pooling (Lin et al. 2014) replaces a flattened fully connected layer, which keeps the model small and makes its size independent of input resolution, so the 112- and 224-pixel runs compare the same network. Dropout (Srivastava et al. 2014) acts on the pooled 256-dimensional vector, after every batch-normalisation layer, so that it does not disturb the normalisation statistics. Two linear heads read this vector: 18 country logits, and a three-dimensional point normalised onto the unit sphere and regressed against the photo’s true location. The coordinate head exists in every mode, so the backbone and parameter count are identical across modes.

Training. The submitted model is trained in the cls mode at 224$\times$224 pixels (the originals are 512 pixels), with random resized crops and horizontal flips, dropout 0.5, mixup with $\alpha = 0.2$ (Zhang et al. 2018), Adam (Kingma and Ba 2015) with learning rate $10^{-3}$, weight decay $10^{-4}$ and batch size 256, an eight-epoch warm-up (Goyal et al. 2017) followed by cosine decay (Loshchilov and Hutter 2017), for 200 epochs. Colour jitter is deliberately off, since colour (red soil, vegetation, haze) is itself a geographic cue. There is no test-time augmentation or ensembling, so the reported accuracy is that of the submitted model. The labelled images are decoded once, held in GPU memory as 8-bit arrays, and augmented there in batches; an epoch takes about 14 seconds, and the final run 47.9 minutes on one NVIDIA A40. This pipeline was written by a teammate (see Contributions).

4 Experiments

We made 17 runs, all with the same seed. Figure 2 summarises the path to the submitted model; the subsections take each step in turn.

4.1 The training recipe mattered more than anything else

The first baseline was deliberately minimal: 112-pixel inputs, no augmentation, twelve epochs at a flat learning rate. Changing only the recipe (224 pixels, random resized crops and flips, 80 epochs with a five-epoch warm-up and cosine decay) raised accuracy from 0.619 to 0.777 with the architecture untouched. The gains landed where the baseline’s confusion matrix predicted: countries near chance improved most (Argentina 7 to 37 of 60, India 21 to 51, the United States 16 to 41), countries already near ceiling barely moved (Iceland 55 to 55, Canada 45 to 47), and a baseline cluster of confusions among arid countries dissolved (India predicted as Egypt: 15 to 0). We did not separate resolution, augmentation and schedule, so we cannot say how the gain divides among them.

Overfitting became the binding constraint. At 0.777 the model reached 0.996 training accuracy against 0.777 on validation (Figure 3), a gap of 22 points with augmentation already on. Capacity and resolution were no longer the bottleneck; regularisation was.

4.2 Regularisation, and how it interacts with training length

Dropout. At 80 epochs, dropout 0.2 and 0.4 performed within one image of each other (0.789 and 0.790), and dropout 0.5 fell to 0.772, below no dropout at all (Figure 4a). At 150 epochs, 0.5 was the best setting (0.806). The same rate was thus the worst setting at one schedule length and the best at another: a longer schedule gives the model more room to memorise, so it both tolerates and needs more regularisation. The failure at 0.5 over 80 epochs also shows how over-regularisation fails. Its train–validation gap (0.176) matched that of 0.4, but training accuracy fell from 0.966 to 0.949 and validation fell with it: past a point, regularisation stops trading training fit for generalisation and simply removes capacity.

Mixup. Mixup with $\alpha = 0.2$ added 14 images on top of dropout 0.5 (0.819) and 21 on top of dropout 0.4 (0.817). We had expected it to over-regularise, since 0.5 was already the peak of the dropout curve. Instead, the difference between the two dropout rates shrank from 9 images without mixup to 2 with it: mixup replaced part of what dropout supplied rather than adding to it. Regularisers that act in different places, the data distribution and the features, do not simply sum.

Optimiser. Replacing Adam with AdamW (Loshchilov and Hutter 2019), everything else fixed, gave 0.800 against 0.806, six images fewer. Best epoch (122 vs. 125) and final training accuracy (0.993 vs. 0.995) agree as well; at a weight decay of $10^{-4}$ the two optimisers behave alike. Testing this first, as the simplest single variable, would have forced a decision on a six-image difference.

Training length, and a heuristic that failed. At the final settings, 150 epochs gave 0.819, 200 gave 0.823 and 250 gave 0.809, fifteen images fewer than 200 (Figure 4b). Until then we had read a best epoch at the very end of the schedule as a sign that the schedule was too short, and extended it four times on that basis. But the best epochs of these three runs were 143/150, 196/200 and 241/250: at the end every time, including the run that is clearly too long. Under cosine decay the learning rate approaches zero at the end, so validation accuracy peaks there almost regardless of length. The heuristic is informative in one direction only: the unregularised 80-epoch run peaked at epoch 64 and then stayed flat, which is a real sign of saturation.

4.3 How the output is formulated

With backbone, initialisation, data order and schedule held fixed, we compared the three modes (Figure 5). At the 112-pixel baseline, classification reached 0.619, classification with the auxiliary coordinate loss 0.636, and pure coordinate regression 0.213.

The auxiliary target helped only an under-trained model. After the recipe was fixed, the auxiliary target no longer helped: 0.769 against 0.777 for classification alone, a nine-image difference within single-seed noise. The extra supervision had helped a model that had not yet learned enough from its labels. Had we stopped at the baseline, we would have kept the auxiliary head for the wrong reason; an ablation is only informative on top of a properly trained baseline.

Coordinate regression fails for a structural reason. Under a squared loss, a regressor predicts the average of the plausible locations. When an image is ambiguous, that average lies between candidates, often at sea or inside a neighbour, and the nearest-centroid rule then collapses whole regions onto the most central country: New Zealand was assigned to Papua New Guinea in 30 of 60 cases, Egypt to Saudi Arabia in 28, Canada to Mexico in 24. Borders are also invisible in coordinates. Measured against the spherical centroids the rule uses, the mean distance from an image to its own country’s centroid is 1,401 km for the United States and 1,529 km for Canada, while the two centroids lie 1,502 km apart: the spread within a country is as large as the separation between countries. Classification keeps its decision on a real country and can use appearance cues that coordinates cannot express, which is also why large geolocation systems treat the task as classification (Weyand et al. 2016).

4.4 Where the submitted model still fails

The submitted model classifies 889 of 1,080 validation images correctly (Figure 6). The weakest countries are Brazil (38 of 60), Mexico (42), Argentina and the United States (43 each); the strongest are Iceland and Saudi Arabia (58 each), Australia and Papua New Guinea (57 each). The largest remaining confusions are small and scattered: Argentina as Peru (6), and New Zealand, Greece and Brazil as Japan, and Canada as the United States (5 each). Several of these pair countries with similar landscapes or road environments, but we did not analyse the images behind them.

5 Discussion

Within a fixed small architecture, the training procedure moved validation accuracy by 20 points, from 0.619 to 0.823. Most of what we learned concerns comparisons rather than settings: an optimal hyperparameter under one schedule was not optimal under another; an auxiliary target’s advantage came from comparing against an under-trained baseline; two regularisers partly substituted for each other; and a reasonable-looking stopping heuristic failed exactly once, in the case that mattered.

Limitations. Every run uses a single seed, so differences of about ten images or fewer should not be read as reliable. All numbers are validation accuracies on one split, and the submitted configuration was selected on that same split, so its accuracy is an optimistic estimate of test performance; the hidden test labels were never available. The recipe step changed three factors at once. We made no new runs for this write-up.

Contributions

This was a three-person course project. A teammate designed and implemented the training pipeline, including the in-memory data handling; another compiled the results for the course report. I trained the models, ran the experiments described here, and analysed validation performance and confusion matrices. The analyses in this write-up come from the project’s committed results and experiment record; the text and figures are my own and were prepared after the course.

References