Quick Takeaways
-
A new uncertainty-focused digit dataset preserves human disagreement instead of forcing every image into a single label, helping researchers study how models handle ambiguity.
-
Annotators labeled MNIST and synthetic ambiguous digits by marking each digit “yes,” “no,” or “unsure”; their responses were combined into image-level soft labels.
-
The dataset includes 3,046 MNIST images and 2,484 synthetic images. Human label variation is far more common in the synthetic set, where only about 20% of images have no disagreement.
-
Data and an optional voice-to-text extension are available on Hugging Face, including uncertainty measures, soft labels, and annotators’ spoken hesitations.
Why Human Uncertainty Matters
A digit can look like a 4 to one person and a 9 to another. Standard datasets often hide that disagreement by keeping only one label. This dataset preserves it.
Researchers re-labeled selected MNIST and synthetic ambiguous-digit images. Annotators marked each digit option “yes,” “no” or “unsure.” The team then combined responses into soft labels: probability-like scores that reflect both agreement and doubt. That distinction matters. A model’s split prediction may signal genuine ambiguity, not poor performance.
A Small, Controlled Testbed
The release includes 3,046 MNIST images and 2,484 synthetic images. Each image received input from three to 10 annotators, with about six on average. Roughly 40% of the MNIST subset showed some disagreement. In the synthetic set, only about 20% showed no variation.
Majority voting changed the original label for about 3% of MNIST images and 33% of synthetic images. Still, the annotations may contain some noise, and the team did not remove every questionable response. The dataset offers a focused way to study model behavior; it does not replace large-scale benchmarks.
Access and Adoption
Researchers can download the dataset from Hugging Face. It includes soft labels, majority-vote labels and measures of annotator uncertainty. A separate extension adds spoken responses, including hesitations, and labeling time. Because that extension was collected separately, users should treat it as a distinct resource.
The CC BY-SA 4.0 license supports reuse with attribution and share-alike terms. The data could help researchers test whether models handle uncertainty more thoughtfully. However, teams should compare different ways to turn human responses into label scores. No single method captures every kind of doubt.
Expand Your Tech Knowledge
Dive deeper into the world of Cryptocurrency and its impact on global finance.
Stay inspired by the vast knowledge available on Wikipedia.
AITechV1
