When Bots Slowed a Small Site After Hacker News
Sammy Straus posted sprinklz.io, a feed reader that lets people push down topics they want to see less of, to Hacker News and picked up about a hundred upvotes. In an account published on Oct. 13, he said the site became painfully slow after the post went live, and the slowness continued and got worse several days later. Straus's write-up
Straus checked his server logs, the automatic records of every request, to explain the load. Most requests to his single-page application, a design where the browser first loads an empty shell and then JavaScript fetches the articles, never fetched articles or ran the code needed to show the page. The visitors collected the shell and stopped. They were not acting like readers.
Almost half the traffic came from one client. Its self-description said it watches for website changes to summarize them in an AI newsletter. It asked for pages many times per minute. Straus blocked it.
Other log entries listed Claude bot, OpenAI bot and Google's search indexer. The logs also showed vulnerability scanners testing paths like wp-content/phpinfo.php, even though the site does not run WordPress.
Straus added a robots.txt file, a small text file that tells compliant crawlers which parts to avoid, meant to discourage AI scanners while still allowing Google-bot and other search traffic. He also added rate limits, which cap how many requests one client can make in a short time. Straus's write-up
The broader context here is resource contention on small infrastructure. Every request has a cost even when it does no useful rendering. Handling the connection, writing a log entry and sending a reply uses CPU, memory and bandwidth before the software can tell whether the visitor is human. When automated polling reaches multiples of human volume, real page loads wait behind work that will never turn into a read. Performance falls without any increase in actual readership.
In my view, the diagnostic approach matters more than the specific block list. Straus separated first document requests from later article fetches and script runs. That split is built into single-page applications, where showing anything requires extra calls after the first HTML reply. A client that never makes those calls identifies itself as non-interactive. The method does not rely on trusting user-agent strings, which can say anything, and it avoids stronger defenses that would add friction for real users.
Looking at what this means for independent builders, the lesson is to instrument for machines from the start. Count requests by agent, path and response code, not just page views. Track the ratio of shell loads to API or article loads. Set rate budgets per client at the edge or reverse proxy, the front layer that can reject excess traffic, then use robots.txt as a statement of preference for crawlers that comply. Scanners hunting for phpinfo paths will ignore that file, but throttling by behavior still applies. The approach stays open to search while giving a solo maintainer a lever that works during a traffic spike.
Worth flagging that this outlook is optimistic rather than predictive, but the underlying exchange still favors small experiments. A feed reader that lets people push down topics they want to see less of is easy to dismiss as a minor control, yet minor controls compound into sustained attention. The fact that such a project can attract a hundred upvotes, survive contact with aggressive automation, and keep serving human readers after basic hardening is a useful reminder. Shipping on the open web still works, provided builders assume part of the audience will be software.
