Quick Takeaways
-
Labels can miss real similarity: CLIP may rank a visually related building style as “wrong” simply because its label differs.
-
Baseline performance: On 9,315 images across 23 styles, CLIP’s top result matched the exact label for 73.5% of queries.
-
Mismatches weren’t random: The 2,465 top-result label mismatches had higher architectural similarity than a frequency-aware random baseline—0.50 vs. 0.35.
-
Evaluation should reflect relevance: Exact-label metrics are useful, but pairing them with application-specific similarity measures can better capture what users actually find relevant.
When Labels Say No, but Embeddings Say Yes
A search for a Ferrari sports car might return a Lamborghini. A strict brand-label test calls that result wrong. Yet a shopper may value its similar shape and sporty design.
That tension appears in image search, too. CLIP turns images into numerical representations, then ranks results by how close those representations sit. The system can find visual relationships across categories. But a benchmark that accepts only matching labels may miss them.
What the Architecture Test Found
Researchers tested CLIP on 9,315 images across 23 architectural styles. The system’s top result matched the query’s style label 73.5% of the time. A label-only score marked the other 26.5% as errors.
The researchers then compared styles using features such as roofs, materials, windows and symmetry. Among CLIP’s label mismatches, the average feature-based similarity score reached 0.5001. A frequency-aware random test averaged 0.3500. Across 10,000 simulated trials, none matched CLIP’s score.
That suggests many mismatches reflect real architectural similarities, not random choices. It does not prove every result helps a user.
Better Scores, With Clear Limits
The findings make a case for pairing label-based measures with measures that reflect a task’s real goals. In fashion search, shoppers may care about shape or fabric. In product search, they may care about function. In architecture, shared features can matter even when style names differ.
Still, the feature scores rely on manually defined style profiles and equal weights. They describe typical styles, not each building. CLIP can also respond to lighting, background or camera angle. Those details may distract from the architecture itself.
For teams adopting image search, the practical lesson is simple: keep exact-label scores, but do not treat them as the whole story. Add human review and task-specific checks. A result can miss the label and still meet the need—or match the label and disappoint the user.
Expand Your Tech Knowledge
Explore the future of technology with our detailed insights on Artificial Intelligence.
Explore past and present digital transformations on the Internet Archive.
AITechV1
