Content Signals: check the deployed robots.txt policy

Cloudflare Content Signals let a site express preferences for how its content may be used for search, AI answers, and model training.

They do not technically block a crawler.

For an agency, the practical job is to check whether the policy deployed in the client’s robots.txt matches what the client actually intends.

Cloudflare defines three signals:

• search for building a search index and returning search results
• ai-input for using content as input to AI systems, including retrieval and grounding
• ai-train for training or fine-tuning AI models

Read the deployed policy first

A Content Signal declaration can look like this:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Read each field independently.

A yes expresses permission for that use through the Content Signals policy. A no expresses a restriction. If a field is omitted, the site has not used Content Signals to grant or restrict that use.

Do not treat the presence of a Content-Signal line as a problem by itself.

First ask what the client wants the policy to say.

Check the live robots.txt response

Inspect the deployed URL:

https://client.example/robots.txt

Do not rely only on a repository file, CMS setting, or origin configuration.

Cloudflare can manage robots.txt and prepend managed content to the origin response, so the live response is the version readers and crawlers actually receive.

Check:

1. the exact Content-Signal line
2. the surrounding User-agent group
3. whether the values match the client’s intended policy

If no policy has been agreed yet, record that as the next decision instead of guessing what the values should be.

Report the result plainly

A useful client finding does not need a long explanation.

For example:

Finding: The deployed robots.txt contains `Content-Signal: search=yes, ai-train=no`.

Client policy: Search use is allowed and AI training is restricted.

Result: The deployed Content Signal matches the approved policy as of 13 August 2026.

If the deployed line does not match the intended policy, identify which layer controls the response before changing anything.

After the change, fetch the live robots.txt again and confirm the deployed value.

Keep Content Signals separate from crawler blocking

Content Signals describe permitted uses after content is accessed. They are not a technical mechanism for preventing access.

If a client needs to block a specific crawler or restrict access to part of a site, review the crawler and bot controls available in the client’s hosting or security stack separately.

Do not report ai-train=no as proof that training crawlers are technically blocked.

For a broader explanation of crawler access, see the AI crawler glossary.

Know what each signal means

Signal	What it covers
search	Building a search index and providing search results
ai-input	Using content as input for AI inference, grounding, RAG, or generative search answers
ai-train	Training or fine-tuning AI models

A missing field is neutral within the Content Signals policy. Do not convert an omission into yes or no in a client report.

Cloudflare is also testing an optional content-use extension for expressing how accessed content may be stored or reused. Treat that as a separate policy choice and check Cloudflare’s current documentation before including it in a client recommendation.

Frequently asked questions

Does ai-train=no technically stop AI training?

No. Cloudflare describes Content Signals as preferences rather than technical countermeasures. If a client needs technical blocking, review crawler-access controls separately.

Is a missing Content-Signal field the same as yes or no?

No. An omitted field means the site has not used Content Signals to grant or restrict permission for that use.

Why can the deployed robots.txt differ from the origin file?

Cloudflare’s managed robots.txt feature can add content to the live response. Check the deployed URL rather than assuming the origin file is exactly what crawlers receive.

Should an agency recommend Content Signals to every client?

Not automatically. First confirm the client’s policy for search, AI input, and training. Then decide whether Content Signals are an appropriate way to express those preferences.

Check the deployed policy, then recheck after changes

Start with the live robots.txt, compare the Content Signal values with the client’s intended policy, and report the result in plain language.

If the policy changes, fetch the deployed file again and confirm that the live response now matches the approved decision.

Maverank for agencies can surface Content Signal findings alongside the rest of a client’s AI-readiness review.