The comparison at a glance
| Question | A general AI assistant | A robots.txt checker | RightSignal |
|---|---|---|---|
| What does it read? | Whatever it recalls or happens to fetch. It may not open the site at all. | Robots.txt only. | Robots.txt, reservation files, headers, page tags, AI files, terms, policies, licences and barriers. |
| Does it quote the site's exact words? | It may paraphrase, and it may invent a quotation. | It shows the robots.txt rule. | It quotes the lines relied on, and code checks each quotation against the page. |
| Is the answer the same tomorrow? | It can change with the wording of your question or the model version. | It is the same if the file is the same. | A recorded decision replays to the same result under the same signed version. |
| Is the time of reading recorded? | It is not. | Usually it is not. | Every source carries the exact time it was read. |
| Does it separate training from answers and search? | Only if you ask, and with no consistency. | It cannot, beyond the crawler names in the file. | Five purposes are decided separately. |
| Does it apply Indian and EU law in a set way? | It gives a general impression of the law. | It applies no law. | It decides under a named profile for India, the EU or both. |
| What can you keep? | A chat transcript. | A log line. | A sealed receipt that anyone can verify. |
| What does it do at a barrier? | It applies no fixed rule. | It does not see barriers. | It records the barrier, stops and never tries to pass it. |
| What happens when it cannot read something? | It may guess. | It may treat a missing file as permission. | It names the gap and returns a hold or a stop. |
Why not ask Claude or ChatGPT whether you may scrape a site?
A general AI assistant is a good way to learn about the subject. It is a poor way to make and record a decision, for six reasons.
- Its answer is not evidence. An assistant's reply shows what a model said. It does not show what a website said. Nobody can take the reply to a counterparty or a court as proof of the site's position.
- It is not repeatable or versioned. The same question can produce a different answer on another day or from another model. You cannot say which rules were applied, because no fixed rules were.
- It is not sealed with the wording and the time. A decision about a site is only as good as the record of what the site said and when. A chat does not capture the page, and it does not fix the moment.
- It does not fetch and check every source. A reliable answer needs robots.txt read for your crawler, the reservation file, the headers and every policy page. An assistant may read one of these, or none, and will not tell you which it missed.
- It may guess. A language model can produce a confident summary of terms it never opened and can attribute to a site words the site does not use.
- It leaves no record of good faith. If you are asked later how you decided, a transcript shows that you asked a chatbot. A receipt shows that you read the source and followed it.
RightSignal uses AI too, in a narrow way. An AI reader helps to read long policy pages. It must quote the sentence it relies on, RightSignal's own code checks that quotation against the page word for word, and the signed Algorithm makes the decision. You get the reach of a language model without resting anything on its word.
Why is a robots.txt check not enough?
Robots.txt is the oldest signal and one of the most important. It is also only one of the places where a site speaks.
- It tells a crawler which paths to fetch. It rarely says anything about what the content may be used for.
- It says nothing of terms. A refusal of AI training, a licence requirement or a reference to Section 43 of the Information Technology Act sits in the terms of use, where a robots.txt checker never looks.
- It must be read for the right crawler. A Dutch court found in 2024 that rules naming particular AI crawlers did not reserve rights against a different crawler. A checker that reads the file for a generic agent can give you the wrong answer for yours.
- It can be missing or unreadable. A file that returns an error is not a file that permits. Many simple checkers treat the two alike.
Why not have a lawyer read each site's terms?
A lawyer's reading of a site's terms is the most careful reading there is, and for your most important sources you may still want one. It does not scale to a crawl. A training corpus draws on thousands of sites, their terms change without notice, and a memo written in March says nothing about what a site said in June. RightSignal does the reading at the scale of a crawl, quotes the clauses that matter and holds the difficult cases for a person. Your lawyers spend their time on the holds, with the evidence already in front of them.
What RightSignal does not claim
RightSignal does not tell you that a collection is permitted in law, and it does not give legal advice. It will hold or stop on some sites where fuller facts might have allowed collection, because it is designed to prefer a hold to a wrong collect. It cannot read what a site has closed. The How it works page sets out its limits.