For scraping and data services

Give your clients more than a promise that you follow robots.txt.

Many scraping services make one of two statements to their clients.

Some say that they follow robots.txt. Others say that responsibility for how the data is used rests with the client who asked for it. Both statements leave the client with nothing to show when someone asks whether a source allowed the collection. With Gate in your pipeline you can offer something better. Before you collect, each target site's robots.txt, signal files, terms and barriers are read for your client's stated purpose, and your client receives a sealed receipt of what was found.

Request accessRead the developer overview

What your clients are being asked

The companies that buy scraping are AI teams, data businesses and enterprises, and they face new questions. Their customers ask for data provenance. Their investors ask for it in diligence. If they serve the EU, the AI Act asks general-purpose model providers to describe the measures that third parties took when supplying data. In India, site owners cite Section 43 of the Information Technology Act in terms that a robots.txt check never reads. A client who cannot answer these questions will look for a supplier who helps them answer.

What changes when Gate is in your pipeline

Compare
Without GateWith Gate
You tell clients that you follow robots.txt.You tell clients that each site's robots.txt, signal files, terms, licences and barriers were read before collection.
You tell clients that use of the data is their responsibility.You give clients a receipt for each source, decided for their own purpose and profile, which they can keep and verify.
A site owner's complaint reaches you with no record on either side.You can show the site owner what its own site said when you checked, and that you stopped where it said stop.
You compete on price and speed.You compete on evidence as well, which is what enterprise buyers pay for.

What you can say to your clients

Once Gate is in your pipeline, you can make statements like these and support each one.

  • Every target site is checked before collection for the purpose you name, under the Indian or EU profile you choose.
  • The check reads the site's terms and AI policies as well as its robots.txt.
  • We do not collect from a site that returns Do not collect.
  • You receive a sealed receipt for each source, and you can verify it without us.
  • The checking service never passes a login, a paywall or a bot challenge.

These statements describe conduct and evidence. They do not say that a collection is permitted in law, and neither you nor RightSignal should say that. Responsibility for the use of data stays where your contract with your client places it. What changes is that both of you hold a record.

How it fits your service

Skip this walkthrough
  • Collect fetch
  • Collect on conditions fetch, and keep the conditions
  • Hold for review flag for a person
  • Do not collect skip
Illustration. The mix of tiles is a fixed pattern, not a measured result.
Fig. 1. G04, One answer for each site, before the crawler fetches. Illustration.
  1. Your client names the target sites and the purpose.

  2. Your pipeline asks Gate for each site's answer through the API.

  3. You collect from the sites that return a collect, and you apply any conditions.

  4. You skip the sites that return a stop and report the holds to your client with the quoted lines.

  5. You deliver the data with the receipt numbers attached.

Gate works beside your crawler and does not replace it. Your crawler keeps its own identity and makes its own requests. Gate contains nothing that helps a crawler get past a site's controls, and RightSignal's acceptable use policy does not allow a client to disguise its crawler or rotate its identity.

What to know before you start

In early access

Gate decides for a named crawler. In early access, Gate accepts the well-known crawler names in its registry, and registration of your own crawler's name is planned. Gate offers no plugin for a particular scraping framework yet. You integrate through the API. The ready-made policies today cover AI training and, under the EU profile, text and data mining.

Tell us how your pipeline works, and we will show you where the check belongs.