Summary Points
- Standard loss functions like MSE only predict the average outcome, ignoring uncertainty and the distribution’s width, which can hide critical information about how sure the model is.
- To capture uncertainty, models should predict both the mean and the variance (spread), turning the prediction into a probability distribution, often a Gaussian, which reveals the model’s confidence at each point.
- Training with maximum likelihood (e.g., Gaussian negative log-likelihood) encourages the model to learn both the true conditional mean and the true local variance, enabling better calibrated uncertainty estimates.
- Relying solely on deterministic predictions can lead to misleading confidence thresholds; probabilistic models provide richer, honest uncertainty estimates crucial for real-world decision-making.
Why MSE Can Be Deceiving
Mean Squared Error (MSE) is a common way to train models. It helps models learn the average outcome. However, it has a weakness. MSE only predicts a single number, the average. It ignores how uncertain we are about that prediction. For example, two different situations can have the same MSE. But, one situation might be very predictable. The other, very uncertain. Under the same MSE, both models look good. Yet, their actual risks are very different. This can lead to surprises in real use. Models trained only with MSE cannot tell you how confident they are. They provide a point estimate, not a range of possible outcomes. Therefore, relying only on MSE might give a false sense of security about a model’s knowledge.
The Hidden Assumption of Constant Uncertainty
When you train with MSE, the training process makes a hidden assumption. It assumes the uncertainty—how much the outcome varies—is the same everywhere. This is called homoscedasticity. But in reality, many signals are heteroscedastic. That means uncertainty changes depending on the situation. For instance, predicting electricity demand is easier on a normal night but harder during a heatwave. The same model cannot tell the difference if it predicts with one fixed number for uncertainty. This assumption can mislead decision-making, especially when signals are more volatile than expected. The model might be confident when it should be cautious or vice versa, leading to errors.
Moving Toward Distributions Instead of Single Numbers
The solution is to let models predict probability distributions, not just single points. Instead of only outputting a number, the model outputs the parameters of a distribution, such as the mean and standard deviation. This allows the model to express uncertainty explicitly. For example, it can say, “I expect the next value to be around 0.5, but I am 80% sure it will be between 0.4 and 0.6.” The commonly used loss for training these models is the Negative Log-Likelihood (NLL). It measures how surprised the model would be if the observed data came from its predicted distribution. NLL considers the model’s confidence and matches the predicted uncertainty to the actual variability in the data. Importantly, this approach aligns with information theory, minimizing the difference between what the model predicts and what truly happens. By predicting both the center and the width of the distribution—meaning how uncertain we are—the model provides richer, more reliable forecasts.
Continue Your Tech Journey
Learn how the Internet of Things (IoT) is transforming everyday life.
Discover archived knowledge and digital history on the Internet Archive.
AITechV1
