Traceafter you collectAvailable by request

Find out what the sites in your dataset said, then and now.

Trace looks back at a dataset you already hold.

Ask about a Trace assessmentSee how it works

For each website in it, Trace finds dated evidence of what the site's signals said when the content was collected, reads what the site says today and applies the same signed RightSignal Algorithm to both. You receive a sealed report, a row for every site and a list of what to leave out of your next training run. Trace lets you answer a question about old data with evidence, and it tells you where no evidence exists so that nobody has to guess.

Audit the data you already hold.

Why teams ask for Trace

Investors, enterprise customers, regulators and courts ask the same question in different words. They want to know what a model was trained on and what each source said about that use. For most datasets nobody checked at the time. Trace gives you a dated and quotable answer for each site, and it marks the sources that deserve a closer look before you train on them again.

Two ways to start

  • A Common Crawl crawl and a list of domains

    You name the crawl by its identifier and give the domains you took from it, or a rule such as every domain under a country suffix. Trace reads Common Crawl's own index to count the pages for each site and to find the robots.txt file that Common Crawl's crawler fetched during that crawl.

  • A list of addresses with dates

    You give Trace a file of page addresses and, where you know them, the dates you collected them. Trace reads the file on your side and works out the sites, the page counts and the date range.

Trace never needs the content of your dataset. It works from addresses and dates. For a Common Crawl study it reads the index and the archived robots.txt records only, and it never downloads the crawled pages.

How Trace works with Common Crawl

Trace

  1. Trace takes the crawl identifier and your domains and counts the pages each site has in that crawl.

  2. For each site it retrieves the robots.txt record that Common Crawl archived, with the archive's own date.

  3. It decides the crawl-time answer for Common Crawl's crawler as of that date, using that record as the only evidence from the time.

  4. It reads the site today with RightSignal's own crawler, under its own name, and decides today's answer from everything the site now publishes.

  5. It compares the two answers and gives each site one of five change classes.

Change classWhat it means
Refusal at both timesThe site refused the crawler at crawl time and refuses today.
Refusal added since crawl timeThe archived robots.txt did not refuse, and the site refuses today.
Refusal removed since crawl timeThe site refused at crawl time and does not refuse today.
No refusal at either timeNeither reading shows a refusal.
Not comparableOne side is missing or could not be determined.

RightSignal makes every request under its own declared name. It never sends another crawler's name, and the name of Common Crawl's crawler is used only as the subject of the decision.

What crawl-time evidence can and cannot show

This is the most important thing to understand about Trace, and RightSignal states it in every report.

Common Crawl keeps the robots.txt files its crawler fetched. It does not keep a dated copy of each site's terms. Evidence from crawl time can therefore show that a site refused the crawler at the time. It can never show a Collect, because the site's terms from that time are not available to read. For a site whose archived robots.txt did not refuse, the best crawl-time answer is Information needed.

Today's reading has no such limit. RightSignal reads the site's robots.txt, signal files, terms and policies as Gate does, so today's answer can be any of the five outcomes.

Trace keeps the two moments apart. It never presents what a site says today as what the site said when you collected. Each row carries one of three labels, which are evidence from crawl time, evidence from today only, and no dated evidence. The archived record is also Common Crawl's record of what its crawler received on its date. RightSignal seals the copy it retrieved and labels it as archive evidence.

If you collected with your own crawler and kept no record from the time, Trace can tell you what each site says today and will mark the collection-time answer as not known. This is the strongest practical reason to start using Gate now. From the day your pipeline stores Gate receipts, you hold collection-time evidence that no later audit can recreate.

What you get

  • You get a sealed report that anyone can verify, with a row for each site, purpose and profile.

  • Each row gives the crawl-time answer with its archive date, today's answer with its capture date, the change class and the lines the answers rest on.

  • Each row carries a disposition for the data you hold. The dispositions are Restrict for the stated use, Review required and Unassessable.

  • You get an exclusion list in a machine-readable file that names the sites and paths marked Restrict, so your pipeline can drop them before the next run.

  • You get a review queue that groups the rows a person should look at, with the most significant first.

A reviewer can add a note, confirm a row or record your organisation's own decision. A review never turns a refusal or a missing check into permission.

Where Trace fits

Trace suits three situations. A model team uses it before retraining, to remove sources that refused at the time or have refused since. A company uses it before a funding round or a large sale, to answer due diligence with a document. A general-purpose model provider uses it to describe, in its public summary of training content, what was done about reservations in a dataset that came from a third party.

Who Trace is for

Trace is for teams that already hold web data and did not check each source at the time. That includes teams that trained on Common Crawl or its derivatives, dataset companies with older collections and the lawyers and investors who review them.

Tell us which crawl or dataset you hold, and we will tell you what Trace can show.