Skip to content
Menu

HentaidudeBot

HentaidudeBot is the crawler that powers Hentaidude. If you found it in your server logs: it reads the public news feeds you publish for syndication, credits you on every story and sends readers to your site for the full article.

Last updated

How to recognise it

Every request carries this User-Agent header:

HentaidudeBot/1.0 (+https://hentaidude.tv/bot)

The robots.txt product token is HentaidudeBot (case-insensitive). We never disguise the crawler as a browser and never use other User-Agents.

What it fetches

  • Your RSS or Atom feed - only for publications listed on our sources page, at most once every 30 minutes, with conditional requests (If-None-Match / If-Modified-Since) so an unchanged feed costs you a 304. Failing feeds are retried with a growing back-off.
  • The article page, once per new story (again only when the story’s feed entry changes) - to read its preview image (og:image) and to run our safety filter on the text before anything is published. We store the headline, a two-to-three sentence excerpt and a link - never the full article.
  • The story’s preview image, once - resized and served from our own server so we never hotlink your bandwidth.
  • /robots.txt - before any of the above, cached for up to 24 hours.

It does not run JavaScript, submit forms, log in, follow links around your site, or try to get past paywalls, CAPTCHAs or Cloudflare challenges (a challenged site gets its feed read only; article pages and images are skipped).

robots.txt and Crawl-delay

  • Every URL - feeds, article pages, images and every redirect hop - is checked against your robots.txt rules for HentaidudeBot (or *) before it is requested.
  • We make one request at a time per host, with at least 1 second between requests, or your Crawl-delay when it is longer (up to 60 seconds per request).
  • If your robots.txt cannot be read because of a server error or timeout, we treat everything as disallowed and try later.

How to opt out

To stop HentaidudeBot completely, add this to your robots.txt - it takes effect within 24 hours:

User-agent: HentaidudeBot
Disallow: /

You can also ask us to stop covering your publication or to remove specific stories: email contact@hentaidude.tv or use the content removal process. Removed stories return HTTP 410 and are never published again.

Contact

Crawling too often, a wrong excerpt, or anything else about the bot: contact@hentaidude.tv. Include the URL and a timestamp from your logs and we will look into it. How we choose sources and what we publish is explained in our editorial policy.