Quick Takeaways
- Alma Media developed an AI system to automatically label real estate photos with 23 classes, boosting search, recommendations, and internal data use.
- The team compared prompting third-party vision-language models versus training custom classifiers, highlighting cost, flexibility, and confidence estimation differences.
- Fine-tuning foundation models with LoRA improves label accuracy and reduces under-labeling, especially for challenging classes like gardens, at modest training costs.
- Using open-source models like DINOv2 or SigLIP, combined with selective fine-tuning, offers a scalable, cost-effective way to enhance image classification in business contexts.
Why Fine-Tuning SigLip Made Sense for Us
We work with thousands of real estate photos daily. Many images show multiple rooms or backgrounds. Off-the-shelf models struggle with this complexity. By fine-tuning SigLip, we tailored the model to our specific needs. It improved accuracy, especially for challenging classes like gardens. Fine-tuning also reduced the number of unlabeled images, making our system more reliable. For our scale, the investment paid off, delivering labels that support new features and better user experiences.
Understanding the Trade-Offs
Choosing between prompting an API and training your own model depends on your goals. Prompting is quick and costs less at small scales. However, at high volumes, costs escalate significantly. Training your own classifier, especially with fine-tuning, involves effort and data collection, but it offers better control and lower long-term costs. Fine-tuning with methods like LoRA keeps training manageable without sacrificing performance. Still, it can be less effective if your foundation model doesn’t match your specific challenges.
When Fine-Tuning Might Not Be the Best Choice
Not all scenarios call for fine-tuning. If your images are simple, or if high accuracy on rare classes isn’t essential, using a prompt-based model can do the job. Additionally, if your data is limited, or you want quick deployment, relying on an external API may be preferable. Sometimes, selecting a better pre-trained foundation model, like DINOv2, can deliver substantial gains without extra training. Ultimately, the decision depends on your volume, accuracy requirements, and resources.
Discover More Technology Insights
Stay informed on the revolutionary breakthroughs in Quantum Computing research.
Access comprehensive resources on technology by visiting Wikipedia.
AITechV1
