Common Crawl and existing corpora

Trace is available by request

Using Common Crawl moves the crawling to someone else. It does not move the question of what each site said.

Many models are trained on Common Crawl or on datasets filtered from it.

Teams often assume that a public archive settles the question of permission. It does not. Each page in the archive came from a website with its own robots.txt, its own terms and its own view of AI training, and that view may have changed since the crawl. Trace reads what each site in your corpus said at crawl time and what it says today, and it gives you a sealed report and a list of what to leave out.

Ask about a Trace assessmentRead about Trace

Why "we only use Common Crawl" is not an answer

  • The archive records robots.txt, not terms. Common Crawl's crawler follows robots.txt rules addressed to it. It does not read a site's terms of use, and it keeps no dated copy of them.
  • Sites have changed since your crawl. A corpus built from an older crawl can hold pages from sites that refuse the crawler today.
  • Your use is not the archive's use. The archive collects pages. You train a model on them. A site's terms may refuse AI training even where its robots.txt admitted the crawler.
  • Model providers are already asked about it. The EU template for the public summary of training content asks providers to describe the measures taken to respect reservations, including by third parties from which datasets were obtained. One provider's published summary, last updated on 23 July 2026, states that its training data includes text from Common Crawl and that it applies opt-out signals to the domains in public web datasets.

How Trace reads a Common Crawl corpus

You name the crawl by its identifier and give the list of domains you took from it.

Trace reads Common Crawl's index for those domains and counts the pages each site contributed.

Trace retrieves the robots.txt record the archive kept for each site and decides the crawl-time answer for Common Crawl's crawler as of the archive's date.

Trace reads each site today, under RightSignal's own name, and decides today's answer from everything the site now publishes.

Trace compares the two and tells you where a refusal existed at crawl time, where one has been added since and where one has been removed.

Trace never needs your dataset. It works from the crawl identifier and the domain names, and it never downloads the crawled pages.

Click the scene for the next step

What crawl-time evidence can show

The archived robots.txt can show that a site refused the crawler at crawl time. It can never show a Collect for crawl time, because the site's terms from that date were not kept. RightSignal states this in every Trace report. Today's reading is complete and can reach any outcome, and Trace never presents today's evidence as what the site said at crawl time.

What you receive

  • You receive a sealed report with a row for each site, purpose and profile.
  • You receive the crawl-time answer with the archive's date, and today's answer with the capture date.
  • You receive an exclusion list that your pipeline can apply before the next training run.
  • You receive a review queue that puts the most significant rows first.

What you gain

Tell us which crawl you used, and we will tell you what Trace can show.