Property Finder found its AI test scores were rigged in favour of its existing AI
Why it matters
If your test answers come from the model you already use, every challenger looks worse, and you may be overpaying for it.
Ask your team
What share of our evaluation labels came from the model we use today rather than human experts?
What happened
Property Finder's engineering team found that 96.8 percent of its evaluation labels were copied from one model's predictions instead of checked by human experts. That skew made every alternative model look worse, and it surfaced while the team was migrating its vision AI workload on .
Tap an underlined word to see what it means.
The context No earlier Seventy story on this in the last 30 days.
Read the original at AWS Machine Learning Blog
Something wrong in this story? Report a mistake. We check it against the source within 24 hours.
Share with your take
Start from our line and make it yours. It stays on your device until you post it.
Ask about this story
Answered only from this story, with the line it came from.