Our crawler

What it is

CanadaPublicSectorJobsBot reads publicly advertised job postings from Canadian public sector employers’ careers pages and feeds, so that they can be indexed at canadapublicsectorjobs.com alongside postings from other employers. Every listing we publish links back to the posting on your site, and the apply button sends the candidate to your own application system — we never take an application ourselves.

In practice we are trying to send you qualified applicants. If that is not something you want, the robots.txt snippet above stops it within a day, and we will not argue.

How to identify it

Requests carry this User-Agent, in full:

CanadaPublicSectorJobsBot/1.0 (+https://canadapublicsectorjobs.com/bot)

The +https:// portion points back at this page, which is how you got here. We do not spoof a browser User-Agent, we do not rotate through residential proxies, and we do not use a headless browser to get around a block. If something claims to be us and does any of those things, it is not us — please tell us at sales@uniqevo.co.uk so we can say so publicly.

How fast it goes

• One request every 2 seconds per host — about 30 requests a minute, at most, no matter how many pages we still have to read.
• One connection at a time per host. We do not parallelise across your site.
• Only what changed. We re-read listing pages we already hold on a schedule rather than continuously, and we honour conditional requests where your server supports them.
• We back off. Repeated errors, a 429, or a Retry-After header pauses the source rather than retrying harder. Three consecutive failures takes it out of rotation until a person looks at it.

Across the whole index we currently read 6 active sources, hourly, covering 1,409 open postings from 8 employers. Those are counted figures, not targets.

What it respects

• robots.txt, checked before a source is set up and re-checked on every run. A Disallow that applies to us is honoured, including one aimed at User-agent: *.
• Crawl-delay, when it is longer than ours. We never go faster than you ask.
• Your terms of use. Every source is reviewed for terms before any collection strategy is written, and a source whose terms forbid automated collection is recorded as permanently excluded — it cannot be switched on later, whatever a tool can technically reach. Several well-known job boards sit in that list.
• noindex and nofollow directives on the pages we read.

We read job postings. We do not fill in forms, submit searches, log in, create accounts, or touch anything behind authentication.

How to stop it

Add this to your robots.txt:

User-agent: CanadaPublicSectorJobsBot
Disallow: /

We re-read robots.txt on every run, so the change takes effect within a day at most. To exclude only part of your site, list the paths instead of /. To slow us down rather than stop us, set Crawl-delay to whatever you prefer.

If you would rather not edit robots.txt, or you want listings that are already published removed as well, write to sales@uniqevo.co.uk or use the contact form. We action removal requests promptly and we do not require a formal notice.

Something went wrong

If our crawler caused a problem on your infrastructure — load, error volume, anything at all — tell us and we will pause the source immediately while we work out what happened. Include the timestamps and a couple of log lines if you can. That is a support request, not a complaint, and it gets answered the same way.

Feeds and APIs

If you publish a job feed, an API, or structured JobPosting data, tell us where and we will read that instead of crawling pages. It is less load on you, fresher data for us and more accurate listings for candidates. That is the outcome we would prefer with every employer.