Crawl policy

How PeptigrityBot behaves.

If you are looking at your server logs and want to know who we are, this page is the answer. Our crawler identifies itself on every request as:

PeptigrityBot/1.0 (+https://peptigrity.com/crawl-policy)

What we collect

  • Product and variant names, SKUs and product URLs
  • Listed prices, currency, and whether an item shows as in stock
  • CAS numbers and synonym lists where a product page publishes them
  • Links to certificates of analysis you publish, and the certificates themselves — see below

That is all. We do not collect customer data, order data, or anything behind a login, and we do not submit forms or add items to carts.

What we never do

  • We do not create accounts on your store, and we store no credentials.
  • We do not bypass a login, a paywall, or a cookie gate. If prices require an account, we record your shop as gated and stop.
  • We do not ignore robots.txt. If it disallows our user agent from your product paths, we do not fetch them.
  • We do not run per-shop custom scrapers. We read three standard public formats or nothing.

How often, and how gently

  • Once per day, off-peak.
  • One request at a time per domain — never parallel connections to the same store.
  • A pause between page requests, and a longer pause between shops.
  • If your server returns a rate-limit response we slow down and retry later rather than pressing on.
  • After five consecutive failed visits we stop crawling your shop and raise an alert rather than continuing to hit a site that is not responding.

How your prices are shown

Prices appear on our price comparison pages with your shop named and linked. Links to your store carry no affiliate tracking — we earn nothing from them. No shop pays for placement or position, and being listed is not an endorsement. Full details are on the methodology page.

Certificates of analysis

If your storefront publishes certificates of analysis, we record where they are and read them. That means the certificate's own contents: compound, batch or lot number, purity, the testing laboratory named on it, and the analysis date. We do not alter them, host copies as if they were ours, or strip your laboratory's name off them.

Nothing read this way is published automatically. A certificate we find becomes a proposal that a person at Peptigrity reads against the original document before anything appears on the site. If the transcription is wrong, it is corrected or discarded — it does not get published and corrected later.

When a test sourced this way is published, it is labelled shop-provided, because that is what it is: you chose which vial was sent for testing and which certificate went on your site. That label sits beside tests whose samples were bought anonymously or commissioned by us, and the distinction is visible to readers. It is not a judgement about your certificate — it is a statement about who selected the sample.

The same robots.txt rules apply here as to everything else. If your certificates live under a path you disallow, we do not fetch them.

Removal, correction, or a feed

If you want your shop excluded, tell us and we will remove it. You do not need to give a reason, and we will not ask for one. You can also block PeptigrityBot in your robots.txt, which we will honour on the next visit either way.

If a price we show is wrong, tell us and we will fix it. If you would rather send us your pricing directly than have us read your storefront, we are building support for that — get in touch and we will set it up.

All three go to the same place: contact us.

Crawling us: AI agents are welcome

We allow AI crawlers. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent are named explicitly in our robots.txt rather than left to the wildcard, so there is no ambiguity about where we stand.

The reasoning is simple: when somebody asks an assistant whether a peptide vendor is any good, we would rather the answer came from lab data than from a forum rumour. Being the cited source on purity questions is how this site gets found. There is no rate deal, no paywall, and no licence to negotiate — the data is public because it is supposed to be public.

An llms.txt at the root points at the four surfaces worth reading: the peptide forum, the compound directory, the lab-test index and the methodology. Thread pages are fully server-rendered, so a fetch with no JavaScript returns the opening post and every visible reply.

One page is marked noindex: the personalised feed at /feed. It is not blocked in robots.txt — blocking it would stop a crawler reading the noindex tag at all — but everything on it has a canonical home elsewhere, so please follow the links out rather than citing it.

If our data ever shows up somewhere misattributed or stale, tell us. We would rather fix a citation than police one. We will revisit this stance only if crawling starts costing real money.