Training data collection

Choose your training sources before you spend compute on them.

A training crawl touches thousands of websites, and each one may say something different about AI training.

Finding a refusal after the model is trained is the expensive way to learn of it. Gate lets you find it before the crawl, when leaving a source out costs nothing. Every source you keep carries a sealed receipt that shows what the site said on the day you collected.

Request accessRead about Gate

What you will be asked later

A model team is asked about its data by several people over the life of a model. An enterprise customer asks in a procurement form. An investor asks during diligence. A rightsholder asks in a letter. If you serve the EU, the public summary of training content asks you to describe how you respected reservations of rights. Each of them wants the same thing, which is a record of what each source said and what you did about it.

How Gate fits your crawl

Before the crawl

Paste or upload your list of candidate sites in the app. Gate reads each site for AI training under your profile and shows the answer for each one.

Sort the list

Remove the sites that return Do not collect. Send the holds to a reviewer with the lines Gate quoted. Keep the sites that return Collect or Collect on conditions, and note the conditions.

During the crawl

Your crawler asks the Gate API for each site's current answer and stores the receipt number beside every item it collects.

At release

Your dataset has a receipt for every source. You can count sources by outcome, list the conditions you accepted and show the evidence for any single site.

On the next refresh

Decisions expire, so your crawler asks again and collects only from sites whose answer still stands.

Click the scene for the next step

What to expect

Expect many stops. Many sites have already said no, in robots.txt, in their terms or with a barrier, and a team that has not looked does not know which ones.

What you gain

  • You avoid fetching, storing and training on content that you would later have to remove.
  • You hold a record for each source that answers a customer, an investor or a rightsholder in the site's own words.
  • You carry each condition with the data it applies to, so it is not lost between the crawl and the training run.
  • You can show, source by source, whether the facts the Delhi High Court relied on in its interim ruling in ANI Media v OpenAI, which is under appeal, were present, or that an EU reservation was looked for and not found.

Which profile to use

Use the India profile if you train and deploy in India only. Use the EU profile if you place a model on the EU market. Use the India and EU Bridge if you are in India now and plan to serve Europe, so that you collect once under the stricter reading of both.

Bring your seed list, and see which sources remain.