PrulseBot

PrulseBot is the automated crawler that powers Prulse, a price-comparison service for Norwegian retail. It periodically visits product pages on partner and tracked e-commerce sites, reads publicly published structured product data (schema.org Product markup — name, price, availability, GTIN/MPN, and similar fields), and stores a price history that Prulse uses to show shoppers how prices move over time.

PrulseBot does not scrape arbitrary page content, submit forms, create accounts, or interact with a site beyond fetching pages a browser would normally load. It only ever reads publicly accessible product pages, sitemaps, and robots.txt.

User-agent policy

PrulseBot identifies itself honestly wherever it can:

PrulseBot/0.1 (+https://prulse.com/bot)

On sites that recognize and allow this identifier, every request — including the initial plain-HTTP fetch — is sent with this user-agent string.

Some sites run bot-protection services that outright challenge-block any client presenting a non-browser user-agent, regardless of intent or robots.txt rules (a "your browser looks like a bot" wall rather than a policy decision the site operator can express via robots.txt). For those sites only, PrulseBot falls back to a standard, unmodified browser user-agent (via a real, unmodified Chromium browser through Playwright) so that it can render the same public page a human visitor would see. This is a rendering fallback, not an attempt to disguise the crawler's intent or evade access control — see the compliance statement below for what stays constant either way.

robots.txt compliance

PrulseBot always fetches and honors robots.txt before crawling a site, and it always evaluates the rules under the User-agent: PrulseBot group, falling back to the wildcard User-agent: * group when a site has no PrulseBot-specific rules. This holds regardless of which fetch mode (honest or browser) is used to retrieve pages on that site — robots.txt directives targeting PrulseBot are never bypassed by rendering pages through a browser.

If robots.txt cannot be retrieved or parsed reliably (server error, network failure, or a bot-protection challenge on the robots.txt request itself), PrulseBot skips the site entirely for that run rather than guessing at what is allowed. A Disallow rule always wins; a Crawl-delay directive is honored as a floor on top of PrulseBot's own rate limits (see below).

Politeness

Independent of what any given site's robots.txt requests, PrulseBot always applies its own baseline politeness rules:

• Concurrency: at most 2 concurrent requests to a single site at a time.
• Rate limiting: at least 1.5 seconds (plus a small random jitter) between requests to the same site — longer if the site's robots.txt specifies a larger Crawl-delay.
• Backoff: on a 429/503 response, PrulseBot backs off exponentially, honoring a server-supplied Retry-After header when present (up to a bounded maximum wait).
• Circuit breaker: if a site returns a run of consecutive blocked responses (repeated challenges or 403s), PrulseBot stops crawling that site for the remainder of the run instead of continuing to retry.

Opt-out

Site operators who would prefer PrulseBot not crawl their site at all can add the following to their robots.txt:

User-agent: PrulseBot
Disallow: /

PrulseBot checks robots.txt before every crawl run, so this takes effect without any need to contact us — though we're always happy to hear from you at the address below.

Contact

Questions, concerns, or opt-out requests can be sent to [email protected].