- Author Terence Eden had real users follow the install guide (README) for a program called ActivityBot. He planned this test while applying for an NLnet grant, recruited participants on Mastodon, and held a video call of 1 hours with each person.
- Participants shared their screens and spoke aloud as they explained where they got confused or stuck. The author wrote down what participants said by hand, revised the guide after each session, and then tested it again with the next person.
- The problems listed in the original include a broken link to a demo tool, people reading the guide in a terminal, confusing section order, no explanation of what the program does, and different permission settings on different web servers.
- The author said he tested the README with several paid participants, paying 25 each, and spent about 150 in total. The claim that the guide became easier to follow is the author's own judgment; the original contains no separate measurements.
I paused for a while at the part where the author takes things like sudo or a single hyphen's difference for granted. The article does not say how many people the 150 was split among, so I'm not sure how small the same approach could start. If your team revises documentation often, have you tried first deciding which situations get stuck?