GetTheJobBot
GetTheJobBot is the crawler of bekommedenjob, a job-application tool. It reads public job ads once a night so users can find jobs that fit them — and always links to the original ad.
What it fetches
Only machine-readable, public job feeds: the interfaces of public employment services (Bundesagentur für Arbeit, job-room.ch, EURES) and the job feeds employers publish through their applicant-tracking system (such as Personio, Greenhouse, Lever, Recruitee, Ashby, softgarden, d.vinci). No pages behind a login, no application forms.
How it behaves
- It reads robots.txt and follows the rules for “GetTheJobBot” and for “*”.
- At most one request at a time per website, with a pause in between.
- A fixed daily cap on requests per website.
- When a website answers 403 or 429, it stops for that night and does not try harder.
- It asks the employment services only for what users search for, and does not re-fetch unchanged ads.
- Names, email addresses and phone numbers of contact persons are not stored.
Opting out
These lines in your website's robots.txt keep GetTheJobBot out entirely. It re-reads robots.txt at least every six hours.
User-agent: GetTheJobBot Disallow: /