In an experiment with a youth sports organization rule Q&A bot, splitting the rules by article raised the accuracy rate from 52% to 72%(zenn.dev)

- This is a chatbot that finds evidence in the bylaws, notices, and contact messages to answer questions such as those about fees or refunds. It splits documents into small pieces and shows the AI only the pieces closest to the question (RAG). The experiment documents are materials for a fictional club created by the author. - When the bylaws were split by article instead of cutting documents every 400 characters, the accuracy rate on 32 questions rose from an average of 52% to 72% over 3 runs. The author disclosed that grading was done by the same Claude Sonnet 4.5 that wrote the answers, so it may be lenient toward its own answers. - Running keyword matching search alongside, guaranteeing at least 2 bylaw pieces, and raising the number of pieces passed on to 8 were all within the margin of error or made results worse. Because principles and cases are written with the same words, pulling up the bylaws pushed out the needed contact messages. - When the author switched to a local model for a laptop (RTX 3050) to avoid sending data externally, the accuracy rate dropped by about 2 tenths. Replacing the search embedding reduced results by an average of 2.7 questions, and replacing the answer model with a 4 billion parameter model reduced them by about 13 more questions.