- The author tested a Japanese product search by configuring OpenSearch's feature that scores how well search results match the query so that an LLM, instead of humans, does the scoring. The author calls this approach LLM_JUDGMENT.
- The experiment used the Japanese portion of the Amazon product search dataset (ESCI). LLM scores were compared with human labels: E (exact match), S (substitute), C (complement), and I (irrelevant). The model was Gemini 3.5 Flash-Lite, and the environment was OpenSearch 3.9.0.
- For the search 'speaker without power', a wooden unpowered speaker was scored 1.0, and USB-powered speakers were scored 0.2~0.3. A liquid formula milk was scored 1.0, the same as the human label E, but for the speaker cable, where the human label was E, the LLM gave 0.1.
- The author attributed the gap to the LLM interpreting USB-powered and rechargeable speakers as 'has power', and noted that the cable case might also reflect variation in the human labels. Because this was an experiment with a few queries run by the author, it cannot be taken as a result that guarantees overall accuracy.
O: For a developer responsible for Japanese product search quality, this approach is worth considering when picking a few frequently used queries, placing human labels next to LLM scores, and starting with the items where they disagree.
X: The original says the comparison was done only with Gemini 3.5 Flash-Lite and a few queries, so you need to check directly whether the same trend holds with other models or more queries.