Find XO: How to build an AI model that picks an answer from fixed choices and calibrates its confidence (nishtahir.com)

No votes yet startupxo 2h ago1 editorial / 0 comments

Writing language: KoreanRead in the original language

- In this method, instead of having the model write text one word at a time, it is made to choose one of the predefined options A through E in a single calculation. The original explains this as a way to limit the range of answers the model can produce. - The original author ran the Qwen3-1.7B model on 1,221 questions randomly drawn from the CommonsenseQA dataset and reported an accuracy of 0.5938. After a brief additional training, accuracy reportedly rose to 0.6241. - The author treated the probabilities the model outputs as confidence and reported that in the confidence range 0.9 to 1.0, the actual correct-answer rate was 0.7009, meaning the model was overconfident. After applying a temperature value of 3.797, the confidence by range and the correct-answer rate reportedly became closer.

Found on

Hacker News ↗ / 133 votes / 30 comments (checked 2026-10-11 11:48 KST)

Have you run into something like this? Tell us in a line or two what you did. You do not need to know the product.

1 comment

Editorial opinionstartupxo 2h ago

The line about the hit rate staying at only about 70 percent even in the 0.9 confidence range stands out first. If you use the score directly as a baseline, your judgment could wobble. Still, the temperature 3.797 was chosen by the author to fit the CommonsenseQA sample, so it is hard to tell from the text alone whether the same value would work on other data. If you have automated multiple-choice classification and used model scores, I am curious how you checked the accuracy by score range.

Writing language: Korean

Related reads

1 comment