I create a search engine (for fun), and it only struck me that my small project could potentially cause chaos by clicking on ads and all kinds of problems.
So what are the guidelines for a good Etiquette web browser?
Things spring is for:
- Follow Robot.txt instructions
- Limit the number of simultaneous requests in the same domain
- Not following sponsored links?
Stop the scanner from clicking on ads
- This is especially in my opinion at the moment ... how can I stop my bot from "clicking" on ads? if it goes directly to the URL in the ad, is it considered a click?
source
share