Skip to main content
Agent Readability makes the content behind your Pull Zone easier for AI agents to consume. It can convert HTML pages into clean, token-efficient Markdown on demand and generate an llms.txt file that describes your website and points agents to its most useful pages. Human visitors continue to receive your regular website. Markdown is returned only when explicitly requested, either through the Accept header or a Markdown representation URL.

​
How Agent Readability works

Agent Readability provides two related capabilities:
  • Markdown for AI agents converts an HTML page into a content-focused Markdown representation. It removes presentation markup and common boilerplate while preserving useful content such as headings, paragraphs, lists, links, images, code blocks, and tables.
  • Generated llms.txt creates a Markdown index containing a short description of your website and links to important pages, grouped into relevant sections. You choose whether Agent Readability discovers those pages automatically, uses a list you provide, or selects them from a sitemap.
Generating and serving llms.txt requires Markdown for AI agents to be enabled. Both features require Bunny Optimizer to be enabled on the Pull Zone.

​
Enable Markdown for AI agents

  1. Open your Pull Zone in the bunny.net dashboard.
  2. Select Optimizer
  3. Enable Bunny Optimizer if it is not already enabled.
  4. Select Optimizer → Agent Readability.
  5. Turn on Enable Markdown for AI agents.
Enable Markdown for AI agents
The same page can now be requested as either HTML or Markdown. To request Markdown, send text/markdown in the Accept header:
A request without the Markdown Accept header continues to receive the original HTML page. Browsers do not request text/markdown by default.

​
Markdown representation URLs

Alternatively, you can request a Markdown representation directly through a URL instead of using the Accept header. The URL format depends on whether the original path ends with a slash: The representation keeps the original page URL as its base, so relative links and images become absolute URLs that continue to point through your Pull Zone.

​
What gets converted

Agent Readability converts HTML documents and keeps their readable content. During conversion, it:
  • Preserves headings, paragraphs, lists, links, images, code blocks, blockquotes, task lists, strikethrough text, and tables
  • Removes navigation, footers, sidebars, forms, dialogs, search controls, scripts, styles, and embedded media
  • Removes hidden and aria-hidden content
  • Resolves relative links and image sources against the public Pull Zone URL
If a response cannot be converted, for example, because it is not HTML, is too large, or conversion fails, Agent Readability returns the original response instead of replacing it with incomplete Markdown.

​
Generate an llms.txt file

An llms.txt file is a Markdown index at the root of your website. It gives AI agents a concise overview of the site and links them to representative content without requiring them to discover every page themselves. To generate it:
  1. Open Optimizer → Agent Readability and enable Markdown for AI agents.
  2. Click Generate llms.txt.
  3. Choose a generation mode and provide any requested source information.
  4. Click Generate llms.txt in the dialog.
  5. Wait for the status to change from Processing to Generated. Click Refresh to check the latest status.
  6. Turn on Enable llms.txt serving.
The generated file is served from the root of the Pull Zone hostname shown in the dashboard:
Generating the file does not publish it automatically. It becomes publicly available only after generation succeeds and Enable llms.txt serving is turned on.

​
Generation modes

Agent Readability can select up to 20 source pages for one llms.txt generation.
Agent Readability discovers representative pages for you. It considers pages that have already been successfully converted to Markdown and looks for a sitemap declared in robots.txt. If robots.txt does not declare one, it tries /sitemap.xml.This mode works best after Markdown representations of some pages have already been requested or when your site publishes a sitemap.
For all modes, the selected pages must be reachable publicly and must return usable Markdown when requested with Accept: text/markdown. Pages that cannot be fetched or converted are skipped. If no valid pages are provided, the generation fails.

​
Regenerate llms.txt

Regenerate the file if generation fails or when the structure or primary content of your website changes:
  1. Open Optimizer → Agent Readability.
  2. Click Regenerate llms.txt.
  3. Choose a generation mode and start generation.
  4. Wait for the status to return to Generated.
Generating llms.txt requires significant processing, so regeneration is rate limited. The waiting period is determined by the system and can range from 5 minutes to 24 hours. The dashboard shows when regeneration becomes available again.

​
Troubleshooting

​
A Markdown request returns HTML

Check that:
  • Bunny Optimizer and Markdown for AI agents are enabled.
  • The request method is GET or HEAD.
  • The Accept header contains text/markdown, or you are using a valid Markdown representation URL.
  • The requested resource returns HTML. Static assets and formats such as images, PDFs, JSON, XML, and existing Markdown files are not converted.
Agent Readability also falls back to the original response when the HTML response is larger than 4 MB or cannot be converted. If HTML Prerender is enabled, it takes precedence when a recognized crawler requests the original page URL. Use the Markdown representation URL when that crawler specifically needs Markdown.

​
llms.txt generation fails

Confirm that the source URLs:
  • Are absolute http:// or https:// URLs
  • Are publicly reachable without credentials
  • Return successful HTML responses through the Pull Zone
  • Can be returned as text/markdown when requested
For Automatic mode, make sure the Pull Zone has previously served some Markdown pages or exposes a sitemap. For Use sitemap, verify that the sitemap URL and any child sitemaps are publicly reachable. If generation fails, wait until the retry time shown in the dashboard and try again.

​
llms.txt is not available at the site root

The file is served only when its status is Generated and Enable llms.txt serving is on. Verify both settings, then request /llms.txt from the Pull Zone hostname shown in the dashboard.
Last modified on September 24, 2026