Data challenges

The 2018 MOT reform

On 20 May 2018 the MOT test changed to follow the EU Roadworthiness Directive (2014/45/EU). Defects are now graded as dangerous, major or minor, and some checks, such as diesel emissions, became stricter. Results from before and after the change aren't directly comparable, so a model trained across both would mix two different standards.

To avoid this, the failure model only uses MOTs from 20 May 2018 onwards, so every result in it was judged under the same rules. That removes the reform as a source of bias, but it doesn't make the fail rate stable: it kept falling after 2018 (see below).

Fail rates move over time

In this data the fail rate fell from over 34% in 2018 to about 28% by 2024. A model trained on earlier years therefore expects too many fails in later ones. This is why the data is split by date rather than at random, with the most recent MOTs held back for testing, and why the model's percentages are corrected using a more recent year. The model page covers this in detail.

The data can't say for certain why fail rates fell. Newer cars making up more of the fleet, garages fixing faults before the test, and changes in testing practice are all possible reasons, and the MOT records don't hold enough information to separate them.

The 2020 pandemic

In spring 2020, MOT due dates for many cars were extended by six months. This shows up in the data as a dip in tests that year and unusually long gaps between MOTs for some cars.

That matters for one of the model's features, time since its last MOT. Normally a long gap hints that a car was off the road or neglected, but in 2020 it often just meant the test was postponed. Those postponed tests fall within the training years, so the model learned the effect of a long gap partly from cars that were never neglected, which may be why it's the weakest of the features.

The model ranked cars just as well on later years without the extension, which suggests the distortion is small, but it hasn't been tested directly.

Inconsistent records

There's no standard format for how MOT testers record vehicle details, so the same car can be described in different ways. Some records give the model variant, such as "318i", while others just say "3 Series". This makes it hard to group cars reliably by model, and mapping the variations takes a lot of manual work. It's one reason the final model uses make rather than model, and uses no free text at all.

Testers also don't always check whether the odometer reads in miles or kilometres. A car can appear to jump or drop by thousands of miles between tests, which breaks the mileage history and makes clocking (mileage that's been wound back) very hard to detect reliably.

Features that looked useful but weren't

• Fuel type: petrol cars seemed to fail more often, but only because they were older on average. Once age was allowed for, fuel type added nothing.
• Mileage at the test being predicted: a strong signal, but it isn't known until the test happens, so using it would leak the future into the prediction. The model uses the mileage at the previous MOT instead.
• Advisory types: categories such as corrosion added nothing beyond the simple count of advisories in the pilot, which covered the BMW 3 Series only.

DVSA wishlist

• A test centre identifier. There's nothing in the data showing where a car was tested, so it isn't possible to see which centres struggle to record odometer units correctly, or which have suspiciously low fail rates.
• Use and keeper flags. Knowing whether a car is in personal or fleet use, and how many keepers it has had, would show whether defects rise with the number of keepers or differ between types of use.
• A reason for leaving the data. A flag for why a car stops appearing, whether scrapped, exported, declared off the road (SORN) or simply untested, would make it possible to study how long cars last (survival analysis). At the moment a car that disappears could be any of these.