We are in the middle of a full revamp for Berkley Security, a trusted security solutions provider that has been protecting homes and businesses across Mississippi and Louisiana since 1978. While working through the redesign, we stumbled onto something that completely changed how we think about building websites going forward.
Someone shared a tool called isitagentready.com with us. It scans your website and checks how ready it is for AI agents. Not just traditional bots or search crawlers, but the actual AI agents that are actively being built into search, assistants, and automated workflows right now. It checks things like robots.txt, markdown negotiation, MCP, OAuth, Agent Skills, and agentic commerce standards.
We punched in berkleysecurity.com and the result was not pretty.
Starting From Zero
The initial scan came back with a score of 25 out of 100. Level 0. "Not Ready." The only thing it passed was Bot Access Control, and that was because Cloudflare Pages already handles robots.txt by default. Everything else was red.
No sitemap. No Link headers. No markdown support for agents. Zero on content accessibility. Zero on API and skill discovery. Basically invisible to any AI system trying to understand what the site actually is.
That is a problem. If you are running a business website in 2026 and AI agents cannot read your content, you are going to start losing ground fast. These systems are already being used to answer questions, compare services, and make recommendations. If your site cannot be parsed by them, you simply do not exist in that world.
The First Round of Fixes
We started with the basics. The things that any website should have but that most people overlook because they have no visible impact on the front end.
Sitemap and robots.txt
We generated a proper sitemap.xml listing every canonical URL on the site. All the
service pages, city landing pages, and blog posts. Then we updated robots.txt to
reference it with a Sitemap: directive so crawlers and agents can find it immediately
without guessing.
Link Response Headers
Per RFC
8288, we added Link response headers to the homepage. These point agents to the sitemap and
robots.txt using the rel="describedby" relation type. It is basically a
machine-readable way of saying "here is where you can learn about this site." We set this up through
a Cloudflare Pages Function middleware that injects the headers on every HTML response
automatically.
After deploying those changes, we rescanned. Score jumped to 33. Discoverability went from 1/3 to 2/3.
Adding Markdown for Agents
This was the interesting one. Cloudflare has a feature called Markdown for Agents that converts HTML to markdown at
the edge when an AI agent sends an Accept: text/markdown header. The idea is simple.
HTML is heavy and wasteful for AI systems. Markdown is clean, structured, and uses way fewer tokens.
The catch is that the built-in Cloudflare feature requires a Pro plan or higher. We are running this site on Cloudflare Pages, so we needed another approach.
We found an open source Cloudflare Pages Function by BWENGR that handles this at the middleware level. It
intercepts requests, checks for the Accept: text/markdown header, and if present,
converts the HTML response to clean markdown before sending it back. It also includes the
x-markdown-tokens header with an estimated token count and a
Content-Signal header for content usage preferences.
The same middleware also handles injecting the Link headers on every HTML response, so we consolidated both features into a single function. No Pro plan needed. It just deploys with your Pages project.
After that deploy, the score moved to 42. Content went from 0/1 to 1/1. Agent Skills discovery went to 1/6 after we added the skills index.
Agent Skills Discovery
The Agent Skills Discovery
RFC (v0.2.0) is a new standard for letting AI agents know what your site or service can do.
You publish a JSON index at /.well-known/agent-skills/index.json that includes a schema
reference, and a skills array where each entry describes a capability with a name, type,
description, URL to a SKILL.md file, and a SHA-256 digest for integrity.
We created a SKILL.md for Berkley Security describing their security services, coverage areas, and how to get in touch. Then we computed the hash and wired it into the index.json. Straightforward once you understand the format.
The Final Score: 58
After all the changes, the final scan came back at 58. Level 4: "Agent-Integrated." Here is the breakdown:
- Discoverability: 100 (3/3) - robots.txt, sitemap.xml, and Link headers all passing
- Content: 100 (1/1) - Markdown negotiation fully working
- Bot Access Control: 100 (2/2) - Already passing from the start
- API, Auth, MCP & Skill Discovery: 17 (1/6) - Agent skills index found
The API/Auth category sitting at 17 is expected and honestly correct. The scanner checks for things like an API catalog per RFC 9727, OAuth endpoints, and MCP server discovery. None of those apply to this site. Berkley Security is a static marketing website with a single Cloudflare Pages Function that handles their contact form through Mailgun. That is not a public API. There is no OpenAPI spec, no REST endpoints, and no health check to advertise. Listing a form handler in an API catalog would be misleading.
So 58 is effectively the ceiling for this type of website, and we hit it.
What This Actually Means
Here is the thing. Most business websites right now score somewhere between 0 and 25 on this test. They have a robots.txt because their hosting platform generates one, and that is it. No sitemap referenced in it. No markdown support. No structured discovery for agents.
That was fine two years ago. It is not fine now. AI agents from OpenAI, Google, Anthropic, and a dozen other companies are actively crawling the web, reading content, and using it to power search results, recommendations, and automated workflows. If your site cannot serve them clean, structured data, you are at a disadvantage.
The changes we made took less than a day to implement. Most of them are just files you drop into your project. A sitemap.xml. A robots.txt with a Sitemap directive. A middleware function. A .well-known directory with a JSON index. None of this requires redesigning your front end or changing your content. It is all infrastructure that lives behind the scenes.
What We Used
- isitagentready.com - Free scanner that checks your site against emerging agent standards
- BWENGR's Cloudflare middleware - Open source Pages Function for markdown negotiation without a Pro plan
- Cloudflare Markdown for Agents docs - The official documentation on the feature
- Agent Skills Discovery - The RFC and specification for agent-readable skill indexes
If you are running a business site and want to get ahead of this, reach out. We are already implementing these standards across all of our client projects. Subscribe to Surefire Studios and let us get your site ready for the next wave of how people find businesses online.
Frequently Asked Questions
An agent-ready website can be discovered, read, and understood by AI systems like ChatGPT, Google Gemini, Claude, and other automated agents. This means having a proper sitemap, supporting markdown content negotiation so agents can request clean text instead of raw HTML, including Link headers for machine discovery, and publishing an agent skills index so AI systems know what your site offers.
It is a free tool built by Cloudflare that scans your website against emerging AI agent standards. It checks your robots.txt, sitemap, markdown negotiation support, Link headers, API catalog, MCP discovery, OAuth endpoints, Agent Skills, and more. It gives you a score from 0 to 100 and tells you exactly what to fix.
Markdown for Agents is a content negotiation feature where your website returns a
markdown version of its HTML content when an AI agent requests it using the
Accept: text/markdown header. This saves tokens, reduces noise from HTML
tags, and gives agents cleaner data to work with. Cloudflare offers this natively on Pro
plans, but you can also implement it yourself using a Pages Function middleware.
No. While Cloudflare's built-in Markdown for Agents feature requires a Pro, Business, or Enterprise plan, you can achieve the same result using a Cloudflare Pages Function middleware that handles the HTML-to-markdown conversion at the edge. The open source middleware by BWENGR does exactly this and deploys automatically with your Pages project at no additional cost.
Link response headers, defined in RFC 8288, are HTTP headers that point to related resources. For AI agent discovery, they tell agents where to find your sitemap, API documentation, or other machine-readable resources without needing to parse your HTML. Adding them to your homepage gives agents a fast path to understand your site structure and capabilities.
It is a JSON file published at /.well-known/agent-skills/index.json that
follows the Agent Skills Discovery RFC v0.2.0. It tells AI agents what your site or
service does by listing skills with a name, description, type, URL to a SKILL.md file,
and a SHA-256 digest for verification. Think of it as a machine-readable summary of your
business capabilities.
The API catalog check per RFC 9727 is designed for organizations that publish APIs for external consumption, like SaaS platforms, cloud providers, and developer tools. A typical business website with a contact form and marketing pages does not have a public API to advertise. Listing an internal form handler in an API catalog would be misleading. The 58 score represents the realistic ceiling for a standard marketing website.
For a typical Cloudflare Pages or static site, the entire process takes less than a day. You are creating a sitemap.xml, updating robots.txt, writing a middleware function for markdown and Link headers, and setting up a .well-known directory with your agent skills index. None of it requires redesigning your front end or changing your existing content.