Field notes on the agentic web.
Software agents now read pages, call tools and, in early pilots, pay. This is our running account of what has changed, which standards matter, and what an ordinary business should do about it. We update it as the standards move, and every figure links to its source.
The visitors changed before the websites did.
Over halfof Internet traffic in 2026 was not human, for the first time.Cloudflare, The Internet has a second audience, 30 Sep 2026
8%of Google results-page visits that showed an AI summary ended in a click on a regular result, against 15% without one (US adults, March 2025).Pew Research Center, 22 Jul 2025
40%of respondents at organisations with revenue above $1 billion report scaling AI agents, up from 27% a year earlier.McKinsey, The state of AI in 2026, 25 Aug 2026
12% to 66%is how far agents’ task success on real computer tasks rose on the OSWorld benchmark, from about 12% to about 66%.Stanford HAI, AI Index 2026, 13 Apr 2026
95%of 200 Indian enterprises surveyed have agentic AI in some form, and most require human approval before any agent acts.NASSCOM Insights, Agentic AI adoption in Indian enterprises, Oct 2026
75%of B2B purchase decisions will be influenced by AI agents in 2027.Forrester, 2027 B2B go-to-market predictions, 21 Oct 2026
The grammar agents speak.
Much of the plumbing now sits with neutral bodies: the Linux Foundation, the IETF, the FIDO Alliance. Here is where each piece stood in November 2026, with a link to each one.
| Standard | What it does | Where it stands |
|---|---|---|
| Model Context Protocol (MCP) | Connects agents to tools and data through servers anyone can run. | Stewarded by the Linux Foundation’s Agentic AI Foundation since December 2025; the official MCP Registry is still in preview. |
| Agent2Agent (A2A) | Lets agents from different vendors describe themselves and work together. | Version 1.0 is stable; agents publish a card at /.well-known/agent-card.json. Part of the Agentic AI Foundation. |
| Agent Skills | Packaged procedures an agent loads when a task needs them. | Donated to the Agentic AI Foundation in October 2026. |
| Agentic Resource Discovery | A single file that lists a domain’s MCP servers, agents and skills. | A proposal (version 0.91, August 2026), already checked by Lighthouse when present. |
| WebMCP | Lets a web page offer tools that an in-browser agent can call. | A community draft in an origin trial; expect it to change. |
| Content Signals | A robots.txt line that says whether content may be used for search, AI answers or training. | Published in 2025; still declared by a small minority of large sites. |
| Web Bot Auth | Agents sign their requests so a site can tell a real assistant from an impostor. | An IETF working-group draft; major AI companies already sign some of their agents’ traffic. |
| Agent payments | Lets an agent pay or check out on someone’s behalf, with a record of what was approved. | Several competing efforts (AP2 at the FIDO Alliance, ACP, UCP, x402, card-network agent tokens); live volumes are still small. |
| llms.txt | A plain index of a site written for language models. | Widely published and cheap to keep. Google has said its search does not use it. |
What being agent-ready asks of you.
Google’s own advice is that no special files are needed to appear in its AI answers. Being useful to agents is a different question, and these are the habits that answer it.
Be readable
Clean, semantic pages with stable layouts. Agents read the accessibility tree as well as the text, so accessible sites are easier for agents to use.
Say what you allow
A robots.txt that separates search, AI answers and training, so your choices travel with your content.
Be callable
Where customers or partners need to act, offer a proper interface: an API, an MCP server, forms that work without guesswork.
Keep one source of truth
Prices, policies and rules in one maintained place, so every agent quoting you quotes the current version.
Decide the autonomy
For every agent you run, write down what it may observe, draft, do with approval and do alone.
Measure it
Lighthouse now scores agentic browsing, and readiness checkers score discovery files. Track both like page speed.
The forecasts expect many projects to fail.
The same analysts who expect agents everywhere expect many projects to be cancelled, demoted or rebuilt. The reasons repeat: context that cannot be trusted, autonomy without governance, and systems the client cannot run without the vendor.
Over 40%of agentic AI projects will be cancelled by the end of 2027, for cost, unclear value or weak risk controls.Gartner prediction, 25 Jun 2025
One in fourenterprise context-layer efforts will be able to support agentic AI at production scale.Forrester, Predictions 2027, 21 Oct 2026
20%of agentic workflows will be reverted to deterministic processes to save money.Forrester, Predictions 2027 for automation, 28 Oct 2026
Trust is the scarce part.
Within a few years, ordinary businesses will rebuild their front doors and back offices around agents. The ones that do it well will have agents that read the right facts, act within limits someone chose, and leave a record anyone can check.
That is why we build deterministic context with Spiderbrain, publish an open format and an open benchmark, and design every operation with its autonomy written down. It is also why this site is kept to the same agent-readiness standard we hold our clients’ sites to.
Find out how your business reads to an agent.
We will run the readiness checks on your site and one of your operations, and show you what an agent sees today.