Ensemble Method

An ensemble method does not tackle a task with a single computational procedure, but with many at once. Each of these procedures makes its own prediction, for example: Is this email spam or not? In the end, the individual answers are combined into a single one, for instance through a vote or an average. The trick lies in the fact that the procedures make different errors. Where one is off the mark, the others are often right, and when combined, the slip-up barely shows. The principle is the same as with a jury: a single judge can be mistaken, but the average of five verdicts is more stable.

Why the group judges more wisely than the single model

The gain in accuracy is often surprisingly large. In public data analysis competitions, ensembles almost always win, hardly ever a single procedure. The reason: a model that learns from example data always carries a certain arbitrariness. It may have picked up a few random patterns in the training data that mean nothing in reality. Averaging over many models makes these random quirks largely disappear.

Diversity is crucial here. Five models that all compute in exactly the same way bring no benefit, because they repeat the same errors. That is why the members are deliberately built to differ: different slices of the data, different starting conditions, sometimes entirely different types of procedures. The goal, as it is put, is for the errors to be as independent from one another as possible.

The price for this is computing time and traceability. Twenty models require roughly twenty times the computing power of one. And when twenty procedures vote together, it becomes harder to explain why a decision turned out the way it did. That is precisely a real problem in credit approval or medical applications.

Voting, refining, stacking on top of each other

The simplest construction is called bagging. Here, many random samples are drawn from the dataset, and a separate model is trained on each one. Afterwards, all of them vote on equal footing. The best-known representative is the random forest, a forest made up of many decision trees. A decision tree, in this context, is nothing more than a chain of yes-no questions leading to a verdict.

The second construction is called boosting, and it works not in parallel but in sequence. The first model is deliberately simple and makes many mistakes. The second is trained specifically on the cases the first one got wrong. This continues, model after model, with each one focusing on the remaining weaknesses. Procedures such as XGBoost or LightGBM work according to this pattern and are still considered the first choice for tabular data today.

A third variant is stacking. Here, an additional model is trained to learn how best to weight the predictions of the others. It thus decides whose voice counts for more in which situation. This is powerful, but also vulnerable: if this top-level model sees too little independent data, it memorizes the training data instead of learning the underlying rule.

Ensembles in weather forecasting, banking, and language models

Ensembles are most commonly found wherever data comes in tabular form. Banks use them to assess credit risks, online shops to estimate the probability of a purchase, insurers to estimate the size of claims. In weather forecasting, a model is run multiple times with slightly different starting values. This is exactly where the figure “70 percent chance of rain” comes from — it states in how many of the runs it actually rained.

The idea also lives on in modern AI systems, often under a different name. When a chatbot answers the same question multiple times and the most frequent answer is chosen, that is an ensemble over time. A related but distinct concept is Mixture of Experts: there, a routing network selects only a few specialized building blocks per query. An ensemble, by contrast, typically has all members compute and then combines their answers.