Skip to content
Request access
Product

Gate before you collect, Trace after, Watch while you rely on a site, Matters when it counts.

  • Gate
  • Trace
  • Watch
  • Matters
  • Get the API
  • Pricing
Solutions

By what you are building or checking.

  • Training data collection
  • RAG and AI search
  • Common Crawl and existing corpora
  • Dataset building
  • Due diligence
  • Law firms
  • All use cases
India and EU

The rules each check reads, and where they are heading.

  • India
  • EU
  • Profiles
  • Where the law is heading
Resources

How it works, receipts, and who we are.

  • How it works
  • Receipts and verification
  • Why RightSignal
  • About
  • FAQ
  • Contact
Legal/Crawler and site-owner policy

Trust and transparency

Crawler and Site Owner Policy

This page is for the owners and operators of websites. It explains what RightSignal's crawler is, how to recognise it, what it reads and never does, how to tell it what you want and how to complain. The address rightsignal.legal/bot, which the crawler's User-Agent names, gives a summary of this page and links to it.

Version
1.0
Last updated
11 November 2026
Sections
6
Reading time
6 min

Contents

  1. 01The crawler
  2. 02What the crawler reads
  3. 03Rate limits and conduct
  4. 04What RightSignal does with what it reads
  5. 05Telling the crawler what you want
  6. 06Complaints
All legal documents

01The crawler

1.1

Purpose: RightSignal reads the signals that a website publishes about automated access to it and use of its content, such as robots.txt, AI and content-use files and the site's terms. It uses what it reads to give its clients a decision under the RightSignal Algorithm on whether a collection they propose should go ahead, and to keep a sealed record of what your site said at the time. RightSignal does not collect your content for its clients, does not train AI models on what it reads, and does not republish or sell it.

1.2

How to recognise it: Every request carries the User-Agent RightSignal/1.0 (+https://rightsignal.legal/bot; rights-signal observation), and the name to use in robots.txt is RightSignal. The crawler never changes its identity from site to site, never rotates identities and never uses another operator's name. RightSignal does not at present publish a fixed list of IP addresses.

1.3

Signed requests: RightSignal has built support for signing its crawler's requests under Web Bot Auth, using HTTP Message Signatures with an Ed25519 key, and it publishes the public keys at https://rightsignal.legal/.well-known/http-message-signatures-directory. Where a request carries such a signature, you can verify it against that directory. Not every request is signed at present, so use the User-Agent to recognise the crawler, and note that RightSignal never treats being verified as permission.

02What the crawler reads

2.1

Pages read: In one check of a site the crawler may read the site's signal files, the page that a client asked about, the home page and up to 10 policy pages. Signal files are robots.txt, ai.txt, llms.txt and files under /.well-known/, and policy pages are terms of use, legal notices, privacy policies, AI or content-use policies and copyright or licensing pages. The crawler does not read through a site's articles, listings, media or other content.

2.2

robots.txt: In each check the crawler fetches robots.txt first, unless it holds a copy less than 24 hours old. It obeys the rules for RightSignal and for * for the page that a client asked about, the home page and every other content page. If your server answers a request for robots.txt with a server error or does not answer, the crawler treats the whole site as disallowed. The crawler always requests the signal files themselves, because they exist to be read by crawlers.

2.3

Policy pages: The crawler reads your own policy pages even where robots.txt disallows them, because your terms are where you say what you permit and a decision made without them would know less about your wishes. It reads at most 10 such pages in one check, under its own User-Agent and within every rate limit, and it never passes a barrier to reach one. The sealed record marks each such page as read despite a robots.txt disallow, records the disallow, and uses the text only to work out what your site permits.

2.4

Pages drawn by scripts and archived copies: RightSignal may render a site's home page once in a browser when its links are drawn by scripts, to find its policy pages. That browser sends RightSignal's own User-Agent, keeps to robots.txt for every host the page loads from, loads no analytics or advertising, clicks, types, accepts and solves nothing, and stops at a 429, a challenge or a block. It loads at most one page and 60 resources within 90 seconds, and where a site's terms still cannot be found an assistant may look for them in a read-only browser under the same identity, rules and stop conditions. Where a policy page cannot be read, RightSignal may look up a copy in a public web archive, which makes no further request to your site and serves as context only.

03Rate limits and conduct

3.1

Rate limits: After robots.txt, the crawler makes at most 10 requests at a time to one host and no more than 200 requests to one site in one check. If your robots.txt sets a Crawl-delay for RightSignal or for *, the crawler makes one request at a time and waits that long between requests, up to 30 seconds. If your server answers with 429 Too Many Requests or a Retry-After header, the crawler makes no further request to that host in that check.

3.2

What the crawler never does: The crawler never passes a login page, a paywall, a bot challenge or any other access barrier, and it never solves a CAPTCHA. It never uses a password, an account, a cookie or any client's credentials, and it never submits a form or sends a request that would change something on your site. It never tries again after a bot challenge, and if your site answers with a barrier or a refusal, RightSignal records that answer as what it is.

04What RightSignal does with what it reads

4.1

Reading and decisions: RightSignal reads the pages with its own software and, in live use, may pass the text of policy pages to an AI model that helps to find and read them. The signed rules of the RightSignal Algorithm reach every decision, and a model never does. RightSignal then gives its client one of the five outcomes that the page How the RightSignal Algorithm Works explains.

4.2

Effect of your signals: An outcome of "Collect" means only that RightSignal found nothing on your site that refuses or conditions that collection under the profile the client chose. It is not a licence or permission from you, and RightSignal never tells a client that it is. The Algorithm treats an access barrier, a robots.txt disallow addressed to the crawler in question and a clear written refusal on your site as withdrawing your permission for the purpose concerned, and it then returns "Do not collect".

4.3

The record and the client's crawler: RightSignal seals a record of what your site said at the time, gives it to the client and keeps it for seven years by default, so the client has notice of your terms and refusals. RightSignal does not operate its clients' crawlers. Each client runs its own crawler and decides for itself what to collect, so the operator of any crawler other than RightSignal is responsible for it.

05Telling the crawler what you want

5.1

Limiting or slowing the crawler: To keep the crawler away from your content pages, add a robots.txt group for User-agent: RightSignal with the rule Disallow: /. The crawler will then read only your signal files and, within the limits in clause 2.3, your policy pages. To slow the crawler down, set a Crawl-delay or answer with 429 or Retry-After.

5.2

Refusing the crawler altogether: You can answer the crawler's requests with 401 or 403, which it never passes and records as a refusal. You can also write to crawler@rightsignal.legal from an address at your site's domain, and we will stop the crawler's requests to your site within 7 days of confirming that the request comes from the site. Where RightSignal cannot read a site's signals, the Algorithm is designed to give a hold or a stop and not a collect.

5.3

How your wishes reach clients: Every decision is made from what your site publishes at the time, whether as machine-readable signals, as human-readable statements in your terms or as access barriers. A clear statement published where a crawler can read it is recorded and reaches every client that asks about your site. This clause describes what RightSignal reads and is not advice on what you should publish.

06Complaints

6.1

How to complain: Write to crawler@rightsignal.legal about the crawler's behaviour, to privacy@rightsignal.legal about personal data on a page that we captured, and to legal@rightsignal.legal with a legal notice or claim. Please give us your site's address, what happened and when, and what you would like us to do. We will acknowledge your message within 3 working days, check our sealed records of the requests that we made to your site, tell you what we find and put right anything that we got wrong.

6.2

What we cannot do: We do not tell anyone which of our clients asked about a site unless a court or the law requires it, and we cannot stop a client's own crawler. We do not alter or delete a sealed record, because a sealed record of what your site said is evidence that may help you as much as anyone. If our reading of your site is wrong, tell us what your site said and when, and we will check the capture and work to correct the reading for future decisions.

6.3

Changes and provider: RightSignal will update this page before it changes how the crawler behaves in any way that this page describes. Manraj Singh Chandpuri, a Canadian citizen and a law student in India, trading as RightSignal, of Chandigarh, India, provides RightSignal, and the Privacy Policy explains how we handle personal data in the pages that we read.

Read with this

  • Privacy and dataPrivacy PolicyVersion 1.0 · 13 min
  • Trust and transparencyHow the RightSignal Algorithm WorksVersion 1.0 · 6 min
  • Trust and transparencySecurityVersion 1.0 · 5 min
rightsignal
RightSignal Legal

Know what a website allows before you collect, and keep the proof.

Product

  • Gate
  • Trace
  • Watch
  • Matters
  • Pricing

Solutions

  • Training data collection
  • RAG and AI search
  • Common Crawl and existing corpora
  • Dataset building
  • Due diligence
  • Law firms
  • Scraping and data services
  • Model builders

Law

  • India
  • EU
  • Profiles
  • Where the law is heading
  • How the algorithm works
  • Not legal advice

Get the API

  • Developer overview
  • Documentation
  • API terms
  • Security

Resources

  • How it works
  • Receipts and verification
  • About
  • Why RightSignal
  • FAQ
  • Contact
  • Request access

Legal

  • Terms of service
  • IP and confidentiality
  • Acceptable use
  • Privacy policy
  • Data processing agreement
  • Subprocessors
  • Cookie notice

For website owners

  • Crawler and site-owner policy
Not legal adviceRightSignal Algorithm 1.3.6

Not legal advice: RightSignal provides evidence and decisions under the RightSignal Algorithm. It does not provide legal advice or a legal opinion, and nothing it produces states that any collection or use is lawful. A decision may not be correct, complete or current, because websites change and a reading can miss wording. You operate your own crawlers and you alone decide what to collect and how to use it. Take advice from your own advocate. The full notice is at rightsignal.legal/legal/not-legal-advice and the terms are at rightsignal.legal/legal/terms.

© 2026 Manraj Singh Chandpuri, trading as RightSignal, Chandigarh, India, who conceived, designed and built RightSignal. RightSignal is for businesses and professionals only and is not for anyone under 18.