Fast Facts
-
Censoring hides part of a value; truncation removes the whole observation. A salary recorded as “$100k or more” is censored; a high earner omitted from the survey is truncated.
-
Ordinary least squares can learn the wrong relationship. In the example, censoring and truncation both pushed the estimated experience–salary effect below its true value.
-
Model the way data was collected. Tobit regression accounts for censored values, while truncated regression adjusts for who could enter the dataset—bringing estimates close to the truth.
-
More complex ML won’t fix flawed data. Even neural networks can predict the wrong target if outcomes are capped or observations excluded; understanding how data was generated comes first.
Start with how the data went missing
Imagine a salary survey that records every salary above $100,000 as $100,000. The survey keeps those people, but hides their exact pay. That is censoring.
Now imagine the survey excludes everyone earning more than $100,000. Those observations disappear. That is truncation. The difference matters: one hides values; the other removes records.
Use a model that matches the data
Ordinary least squares treats every recorded value as exact. It may therefore underestimate the salary–experience relationship in either case.
A Tobit model handles censored data. It uses exact salaries where available, then accounts for the chance that each capped salary meets or exceeds the limit. In Python, SciPy’s norm.logpdf can calculate likelihoods for exact values, while norm.logsf handles the censored tail.
Truncated regression handles selection into the dataset. For salaries below a limit, its likelihood adjusts each observation by the probability of earning below that limit. In Python, use norm.logpdf minus norm.logcdf, then estimate the parameters with scipy.optimize.minimize.
In a simulated example, ordinary regression estimated a slope of about 3.2 for censored data and 4.1 for truncated data. Tobit and truncated regression estimates came much closer to the true slope of 5.
Better fit depends on sound assumptions
These methods can improve estimates, but they are not automatic fixes. Standard versions assume a correctly specified relationship and normally distributed errors with steady variance. Real data may break those assumptions.
The same lesson applies to machine learning. A more complex model can still learn the wrong target if the data hides or excludes outcomes. First ask how the data was collected. Then choose a model and loss function that reflect those limits.
Expand Your Tech Knowledge
Stay informed on the revolutionary breakthroughs in Quantum Computing research.
Discover archived knowledge and digital history on the Internet Archive.
AITechV1
