Designing Content for Machine Readers
Content Modelling for AI Search, Agents and Retrieval
Designing Content for Machine Readers
Download the full white paper as a PDF.
Author: Andy Blyth PgDip BSc(Hons) FdSc, Technical Architect
Site: technicaldogsbody.com
Published: 18 November 2026
Status: Published
Version: 1.0
Table of Contents
- Executive Summary
- Section 0: Why This Is the Architect's Problem
- Section 1: How Machines Read Content
- Section 2: Measuring Machine Readability
- Section 3: Principles for Machine-Readable Content Models
- Section 4: Implementation Patterns
- Section 5: Case Study
- Section 6: Governance, Risk and Adoption
- Conclusion
- Appendix A: Field Classification
Executive Summary
AI search, assistants, agents and retrieval systems read content without seeing the page it was designed for. They work from text, structure and labels, so they understand what the content model makes explicit and miss what the template only implies. When a fragment of content loses its subject, a machine reader cannot use it. Anthropic found that adding that context to each chunk before indexing cut retrieval failures by 35% with embeddings alone, and by 49% with keyword search added.1
The content model is the part an architect controls, and this paper tests its effect directly. The same facts were held in a generic content model and a typed one, and read through the same pipeline by three sizes of model. The test uses 19 questions on synthetic data, and the generic model reproduces the failures measured in Section 2. The typed model answered all 19 questions correctly with every model, under a strict and a permissive prompt; the generic one answered between 14 and 17.
Within each model, both content models answered at about the same speed. The gain is in which model you need: the smallest model, reading typed content, answered every question in a median of 0.27 seconds. The largest, reading the generic content, took 3.50 seconds and still missed two.2
This paper gives architects and technical leads on Optimizely, Contentful and similar platforms a practical framework for designing content that machines can read. It covers how to measure a content model, five principles to design it by, patterns for both platforms, a reproducible test and the governance to keep the model in shape.
The paper has seven sections:
- Section 0 (Why This Is the Architect's Problem) explains why the content model is an architecture decision, made early, that outlasts every redesign.
- Section 1 (How Machines Read Content) describes the four machine readers: AI search, third-party assistants, autonomous agents and retrieval systems you own.
- Section 2 (Measuring Machine Readability) sets out five measures of a content model and a maturity model, applied to technicaldogsbody.com.
- Section 3 (Principles for Machine-Readable Content Models) gives five principles, each with one test question.
- Section 4 (Implementation Patterns) turns the principles into content types for Optimizely CMS 13 and Contentful, and compares how each platform exposes content to machines.
- Section 5 (Case Study) tests a generic model against a typed one with three sizes of local model, with every answer published.
- Section 6 (Governance, Risk and Adoption) covers the editor experience, review and ownership, AI-generated content, the risks, and an audit for any content model.
Section 0: Why This Is the Architect's Problem
Your content now has readers that never see the page. A search engine summarises it, and an assistant quotes it. An agent acts on it. A retrieval system splits it into pieces and keeps only the pieces that match. None of them sees your design, your layout or the heading above the paragraph. They see what the content model gives them.
These readers already stand between your content and the people it was written for. In the UK, Ofcom reports that about 30% of searches now show an AI overview.3 In the US, Pew Research Center found that when a Google search showed an AI summary, people clicked a traditional result on 8% of visits, against 15% when no summary appeared.4 Section 1 (How Machines Read Content) describes how each reader works.
The Decision Is Made Early
The content model is usually settled in discovery, in the first weeks of a project, by architects and technical leads. It decides which types exist, what each field means, which entries reference which, and where each fact lives. Those choices then outlast the work built on them. A site can be redesigned, rebuilt in a new front-end framework or moved to a new host while the content model stays the same, because changing it means migrating every entry that uses it.
That makes the content model an architecture decision. It is cheap to get right at the start and expensive to change later. Section 6 (Governance, Risk and Adoption) covers the cost of remodelling an existing site.
Why It Is Not Someone Else's Problem
Content designers, editors and search specialists all improve how machines read a site, but they work inside the content model. An editor can write a clearer testimonial. An editor cannot add a field that says who the testimonial is about. A search specialist can add structured data to a page, but it can only describe the facts the model already holds. The shape of the model sits with the people who design it.
The case study in Section 5 (Case Study) shows how much that shape matters. The same facts were held in two content models and read through the same pipeline by three sizes of model. The original model answered between 14 and 17 of the 19 questions, and the remodelled one answered all 19.2
What This Paper Covers, and What Will Date
This paper is not a playbook for generative engine optimisation (GEO) or answer engine optimisation (AEO). It does not cover tactics for ranking in AI answers. It is not a vendor comparison either: Optimizely CMS and Contentful are the worked examples, and the principles apply to any CMS with a structured content model.
Parts of it will date. The platform features in Section 4 (Implementation Patterns) and the crawler controls in Section 6 will change. So will agent protocols such as the Model Context Protocol (MCP), and the models tested in Section 5. The measures in Section 2 (Measuring Machine Readability) and the principles in Section 3 (Principles for Machine-Readable Content Models) describe how machines read content, and should last longer than the tools that read it.
This paper gives architects and technical leads a way to measure their content model and five principles to design it by. It adds patterns for Optimizely CMS 13 and Contentful, a reproducible test of the effect, and an audit to run on the models they already have.
Section 1: How Machines Read Content
Your content now has four kinds of machine reader, and none of them sees the page the way a person does. Each one reads text, structure and labels. The design, the layout and the template that gave the content its meaning are mostly invisible to them.
The scale is no longer small. In the UK, Ofcom reports that about 30% of searches now show an AI overview, and 53% of adults say they see these summaries often.3 ChatGPT had 1.8 billion UK visits in the first eight months of 2025, almost five times the 368 million in the same period of 2024.3 Cloudflare reported in July 2026 that more than half of the traffic on the internet is now non-human.5
This section describes each of the four readers, how it reaches your content, and what it needs from you. Parts of it will date quickly. The products and protocols named here change month by month; the way each reader consumes content changes much more slowly, and that is what the rest of the paper builds on.
1.1 The Four Readers
| Reader | How it reaches your content | What it needs from you | How much control you have |
|---|---|---|---|
| AI search | Rendered pages, retrieved from a search index | Crawlable pages, clear headings, accurate structured data | High, and well documented |
| Third-party assistants and their crawlers | Crawled or fetched pages, summarised alongside other sources | Clear, self-contained statements | Low |
| Autonomous agents | Screenshots, HTML, the accessibility tree, APIs and MCP servers | Semantic markup, a stable layout, explicit data and actions | Medium, if you offer a proper interface |
| Retrieval systems you own | Delivery APIs, exports and vector indexes | Entries that stand alone, metadata and permissions | High, because it is your own estate |
1.2 AI Search
AI search reads your pages through the same index as traditional search. Google says its generative AI features, including AI Overviews and AI Mode, are rooted in its core ranking and quality systems.6 They use two techniques. Retrieval-augmented generation (RAG) fetches relevant pages from the index and grounds the answer in them. Query fan-out runs several related searches at once to gather more material for the same answer.6
The reader still sees your content. The person often does not. Pew Research Center tracked the browsing of 900 US adults in March 2025. When a Google search showed an AI summary, users clicked a traditional result on 8% of visits, against 15% when no summary appeared.4 They clicked a link inside the summary itself on about 1% of visits.4 Pew found that 18% of searches produced a summary; Ofcom's later UK figure is about 30%, although the two studies measured different countries, periods and methods.4
This is the reader you have the most influence over, and Google is clear about how to use it. Its guide says there are no additional requirements to appear in AI features, and that you do not need special files, special markup or content written specially for AI.6 Good search practice still applies. The difference is that a summary now stands between your content and the person, so your content has to survive being summarised.
1.3 Third-Party Assistants and Their Crawlers
Assistants such as ChatGPT, Claude, Copilot and Perplexity read the web in two ways. Their crawlers collect pages, mostly to train models, and their search tools fetch pages to answer a question as it is asked. In both cases your content is read, compressed and blended with other sources before anyone sees it.
Cloudflare measures this with a crawl-to-refer ratio: the number of pages a platform's crawlers request for each visit that platform sends back.7 In July 2025, Google crawled about 5 pages for each referral. OpenAI crawled about 1,091, and Anthropic about 38,066.7 Cloudflare notes that the Anthropic figure may be overstated, because visits from the Claude apps do not always carry a referrer.8 In the same month, 79% of AI crawling was for training, up from 72% a year earlier.7
You have little control over this reader. You can block its crawlers, and Section 6 (Governance, Risk and Adoption) covers that choice. You cannot choose how it splits, summarises or attributes your content. The only lever you hold is the content itself: whether each statement still makes sense once it has been lifted out of the page.
1.4 Autonomous Agents
Agents act for a person rather than answering a question. They book, compare, fill in forms and complete tasks. Google's web.dev guidance describes three ways an agent reads a website: a screenshot passed to a vision model, the raw HTML, and the browser's accessibility tree.9 Most agents combine all three, using the HTML and accessibility tree for structure and the screenshot for layout.9
Each view loses something. A screenshot is slow and expensive to analyse. The HTML shows nesting but not intent: a <div> styled as a button is not a button to an agent. The accessibility tree is the cleanest map of the page, but only if the markup is semantic.9 The guidance puts the problem plainly: designs built for humans, with complex hover states and shifting layouts, are "functionally broken for agents".9
The better route skips the page entirely. A Model Context Protocol (MCP) server lets an agent work with CMS content and capabilities directly, with no page in between. Contentful already offers an MCP server, and I used it to research this paper. Connected to my own Contentful space, an AI assistant read the content model directly: content types, field names, validations and entries. It never saw technicaldogsbody.com. Every judgement it made about what an entry was came from the model alone, and Section 3 (Principles for Machine-Readable Content Models) shows how often that model fell short.
1.5 Retrieval Systems You Own
The fourth reader is the one you build. Internal copilots, support assistants and site search with generated answers all follow the same pattern. Content is split into pieces, each piece is indexed, and the best matches are passed to a language model with the question.
This reader exposes weak content models fastest, because it reads the entries themselves, not the pages. If an entry relies on the page around it for its subject, its date or its relationships, that context is lost the moment the entry is indexed. Anthropic measured the cost of that loss, and Section 3.2 (Name the Subject in Every Entry) sets out the figures.1
Retrieval systems do not rely on embeddings alone. Many vector search services can also filter on metadata, such as a date, an author or a content type, to narrow the entries a search considers. Azure AI Search, for example, filters on the non-vector fields in the same index, and by default applies the filter during the vector search itself.10 A typed content model supplies those fields directly. A white paper with a publication date and an author reference can be filtered on both; a generic block with a field called date, or a date in its body text, cannot.
It is also the reader you control completely. You choose what is indexed, how it is split, which metadata travels with it and who can see it. That makes it the right place to start, and Section 5 (Case Study) uses it for that reason.
1.6 What the Four Readers Share
The four readers differ in how they reach your content and how much influence you have over them. They share one limitation: none of them sees the design. Each reads text, structure and labels, and web.dev's summary applies to all four: the quality of that machine-readable representation decides how well the reader performs.9
That representation is set by the content model long before any page is designed. Section 2 (Measuring Machine Readability) gives you a way to measure it.
Section 2: Measuring Machine Readability
You can measure how readable your content model is to machines before you change anything. The measurement needs read access to the content model and a way to count. It does not need an AI system.
Measure the model, not the pages. Section 1 (How Machines Read Content) showed that every machine reader works from text, structure and labels. The content model sets the structure and the labels for every entry of every type at once. A page audit finds symptoms one page at a time; a model audit finds the cause, and fixing one content type improves every entry of that type.
2.1 Five Measures
Each measure tests one of the principles set out in Section 3 (Principles for Machine-Readable Content Models).
| Measure | What to count | What good looks like |
|---|---|---|
| 1. Field naming (3.1) | Every field, classed as meaning, generic or presentation | Presentation fields only in types that exist to control layout |
| 2. Self-describing types (3.2) | Content types whose meaning is set by a template or by the page around them | None |
| 3. Relationship references (3.3) | Entry references that describe meaning, against those that describe layout; links held as text | Every relationship is a reference, and some describe meaning |
| 4. Duplicated facts (3.4) | Facts held in more than one entry, in a sample of your most important content | None |
| 5. Time and ownership (3.5) | Content types with a review date, an owner and a validity range | Every type whose content can go out of date |
The first measure needs a definition, because it does the most work. Classify every field by one question: does its name say what the content is, without the page or the template?
- Meaning. The name states what the content is. Examples:
slug,publishedDate,metaDescription. - Generic. The name states a data type or a role, but not what the content is. Examples:
title,content,date. - Presentation. The name states where or how the content is shown. Examples:
ctaText,showAssetLink,colourWay.
A generic field is not a fault on its own. A field called title is clear inside a content type called Testimonial. The same field inside a type called Content Block tells a machine nothing.
2.2 Worked Example: technicaldogsbody.com
The worked example applies the five measures to my own site's Contentful model, read through the Content Management API on 5 November 2026.11
Content model: technicaldogsbody.com, Contentful
Content types 18
Fields 86
1. Field naming
Meaning 26 / 86 = 30.2%
Generic 34 / 86 = 39.5%
Presentation 26 / 86 = 30.2%
Types with a presentation field 9 / 18 = 50.0%
2. Self-describing types
Content Block entries 139
Content Block fields named for meaning 0 / 15
Template values that set its meaning 9
(3 on the block, 6 on the collections that hold it)
3. Relationship references
Entry references 15
Describing layout or navigation 15 / 15 = 100%
Describing what an entry is about 0 / 15 = 0%
Links held as text (sampled) at least 1
4. Duplicated facts (sample: one white paper)
Entries holding the paper 3
Entries repeating the title 3
Entries linking the PDF 2
Publication dates 2, which disagree
5. Time and ownership
Types with a review date 0 / 18
Types with an owner 0 / 18
Types with a validity range 0 / 18
Types with an author 0 / 18
Date fields 2
Three results stand out.
The busiest type says the least. Content Block holds 139 entries: testimonials, employment history, awards, projects, white paper listings and more. None of its 15 fields is named for what it holds. Nine template values, set on the block or on the collection around it, decide what each entry is. A machine reading the API sees 139 entries of one generic type.
Every reference describes layout. All 15 entry references in the model say where something appears or what a menu links to. None says what an entry is about, who wrote it or what it relates to. The model can tell a machine that a testimonial sits inside a collection called Kudos. It cannot tell it whom the testimonial is about.
The parts built for search engines score best. The Page type has 8 of its 11 fields named for meaning, and Page Meta has 3 of 4. Both were designed around search engine needs: slugs, canonical URLs, meta descriptions and publication dates. The model is most machine-readable where it was already serving a machine.
The breakdown by content type shows where the presentation fields sit.
| Content type | Fields | Meaning | Generic | Presentation |
|---|---|---|---|---|
| Photo Page | 7 | 0 | 1 | 6 |
| Blog Snippets | 4 | 0 | 1 | 3 |
| Content Block Collection | 4 | 0 | 2 | 2 |
| Menu Item | 4 | 0 | 2 | 2 |
| Search | 2 | 0 | 1 | 1 |
| Content Block | 15 | 0 | 9 | 6 |
| Header | 5 | 2 | 1 | 2 |
| Banner | 6 | 2 | 2 | 2 |
| Page | 11 | 8 | 1 | 2 |
| Amazon Recommendation | 5 | 2 | 3 | 0 |
| Amazon Recommendations | 4 | 2 | 2 | 0 |
| Code Kata | 4 | 4 | 0 | 0 |
| Footer | 2 | 1 | 1 | 0 |
| Gallery | 2 | 0 | 2 | 0 |
| Labels | 2 | 0 | 2 | 0 |
| Page Meta | 4 | 3 | 1 | 0 |
| Social Link | 3 | 1 | 2 | 0 |
| TagGallery | 2 | 1 | 1 | 0 |
Some presentation fields belong where they are. Blog Snippets and Search configure components, and a field such as pageSize is right for that job. The problem is presentation fields in types that hold content, such as Content Block and Photo Page.
2.3 A Maturity Model
The five measures place a content model on one of five levels.
| Level | Name | Signs in the five measures |
|---|---|---|
| 1 | Page-shaped | Few content types; most content sits in rich text on pages |
| 2 | Componentised | Reusable components exist, but templates decide what they are; references describe layout only |
| 3 | Structured | Types are named for what they hold; presentation fields stay in layout types; some references describe meaning |
| 4 | Governed | Types that can go out of date carry an owner, a review date and a validity range; each fact has one canonical entry |
| 5 | Machine-ready | Content is exposed deliberately to chosen machine readers, and their use of it is measured |
My site sits at Level 2. It has reusable components, but its busiest content type takes its meaning from templates, and none of its references describes meaning. Two types come close to Level 3: Code Kata names all four of its fields for what they hold, and Page names 8 of its 11. That is a common pattern: a model rarely sits at one level throughout, and the parts that score well show the team what the rest could look like.
2.4 Limits of the Measure
The measures are a proxy, and they have three limits.
Field names are not content. A well-named field can still hold vague or out-of-date content. The measures show what the model allows, not what editors write. Section 6 (Governance, Risk and Adoption) covers the editorial side.
The classification is a judgement. Each of the 86 fields needed a call, and a second reviewer would move a few. colourWay is the clearest borderline case. On my site, Ember means work and Teal means personal, so the field holds real meaning; it is classed as presentation because its name describes a colour, not a category. That is the point of the test. Appendix A (Field Classification) lists every field and its class, so you can check or change any of them.
It measures the model, not the effect. A better score should mean better answers from machine readers, but this section does not prove it. Section 5 (Case Study) tests that claim directly, with the same questions asked of the current model and a remodelled one.
The method scales. An enterprise model has more types, but the counting can be scripted against the Contentful Content Management API or Optimizely content type definitions, and Section 4 (Implementation Patterns) shows how. Section 3 sets out what the measures are testing for.
Anthropic. "Introducing Contextual Retrieval." Anthropic, September 2024. https://www.anthropic.com/research/contextual-retrieval↩↩
Andy Blyth. "The Assistant Test." Source code, synthetic data and results, runs on 5 November 2026. https://github.com/technicaldogsbody/the-assistant-test Chat models
gemma4:e2b(digestb37049369adf…),gemma4:e4b(dc35e8d9c606…) andgemma4:12b(6114515d63c1…); embedding modelembeddinggemma-2:270m(c21dd30c1e38…).↩↩Ofcom. "From apps to AI search: how the UK goes online in 2025." Ofcom, 10 December 2025. https://www.ofcom.org.uk/media-use-and-attitudes/online-habits/from-apps-to-ai-search-how-the-uk-goes-online-in-2025 Growth: 1,800 million ÷ 368 million = 4.9.↩↩↩
Athena Chapekis and Anna Lieb. "Google users are less likely to click on links when an AI summary appears in the results." Pew Research Center, 22 July 2025. Based on 68,879 Google searches by 900 US adults in March 2025, 12,593 of which showed an AI summary. https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/↩↩↩↩
Arielle Weiss, Zach Albertson and Emily Lanfear. "Content Independence Day, one year on: building the business model for the agentic Internet." Cloudflare, 1 July 2026. https://blog.cloudflare.com/agentic-internet-bot-report↩
Google Search Central. "Optimizing your website for generative AI features on Google Search." Google, published 15 May 2026, last updated 10 July 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide↩↩↩
João Tomé. "The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals." Cloudflare, 29 August 2025. Ratios are crawls per referred visit for July 2025, rounded to the nearest whole number. https://blog.cloudflare.com/crawlers-click-ai-bots-training/↩↩↩
David Belson and Sam Rhea. "The crawl before the fall… of referrals: understanding AI's impact on content providers." Cloudflare, 1 July 2025. https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/↩
Kasper Kulikowski and Omkar More. "Build agent-friendly websites." web.dev, Google, last updated 1 April 2026. https://web.dev/articles/ai-agent-site-ux↩↩↩↩↩
Microsoft. "Filters in vector queries." Azure AI Search documentation, last updated 27 April 2026. https://learn.microsoft.com/en-us/azure/search/vector-search-filters↩
Content model of technicaldogsbody.com, Contentful, read through the Content Management API on 5 November 2026. The classification of each field is listed in Appendix A (Field Classification).↩
Found this useful?
Share it or download a copy to keep.
Andy Blyth
Andy Blyth, an Optimizely MVP (OMVP) and Technical Architect at MSQ DX with a keen interest in martial arts, occasionally ventures into blogging when memory serves.



