The way content is accessed online is changing fast. Alongside regular readers, websites are increasingly visited by AI systems and automated crawlers, and most publishers haven't yet developed a clear strategy for how to handle them. IAB Tech Lab has released new guidance on bot and crawler management strategies, now available for public comment until June 25, 2026. It's designed to help content owners think through their options in a structured way.
What does the guidance cover?
The guidance outlines the range of available approaches and the advantages and trade-offs of each strategy, giving stakeholders a clearer understanding of how to manage access from all non-human traffic, including AI systems, in a way that aligns with their business goals. Importantly, it's written for business leaders, not just technical teams. It gives non-technical leadership the questions they need to ask to understand the value and hard costs associated with each decision they make.
Why does this matter now?
This guidance supports the broader rollout of the CoMP API - an industry initiative aimed at creating a more structured relationship between AI companies and the publishers whose content they use. Many content owners and publishers have not yet developed formal strategies around their approach to bots and crawlers, and this guidance is designed to address that gap.
How to get involved
The public comment period is open until June 25, 2026. You can read the full guidance at https://iabtechlab.com/botmanagement/ and share your feedback before it's finalised.
