5 Decisions Before Setting Up robots.txt for AI Agents
Before allowing AI agents on your site, decide: why you’re doing it, what content to show, who to let in, what limits to set, and how to measure the results. Simple, no coding required.
Key takeaways
- Robots.txt is a guideline for bots, not a lock. Set rules and combine them with other security measures.
- Define your business goal: sales, FAQ responses, or leads—this helps you choose content and measurement.
- Build an allowlist of known AI agents and set entry limits (speed, hours, budget).
- Measure results: use UTM links, dedicated pages/FAQs, discount codes, and goals in analytics.
Traffic from AI responses is increasing, and AI agents are reading websites on behalf of users more frequently. Meanwhile, some networks are planning to block certain bots by default without your consent. This is a good time to consciously set up your robots.txt file—simply, without coding.
What’s Changing and Why It Matters
An AI agent crawler is a bot (an automated program) that reads your website in real-time to provide answers to users. Think of it as an assistant that checks your site instead of a customer.
Robots.txt is a file that contains rules for bots. It’s like a sign on a door: ‘this area is open, this area is closed.’ It’s a guideline, not a lock—well-behaved bots follow it, but some misbehaving ones might ignore it.
Why now? According to announcements from July 1, Cloudflare plans to block certain AI crawlers by default starting September 15, unless the website owner gives permission. Check with your provider for the current status, but the trend is clear: consent first, then access. The takeaway: it’s better to define your rules consciously today.
5 Decisions Before Setting Up robots.txt
Instead of focusing on technical details, approach the topic from a business perspective. Here are five simple decisions to organize your policy regarding AI agents.
- Purpose: What should happen after the bot visits? Sales, quick FAQ responses, or lead generation (contact for sales)? A clear goal helps determine which content to share and how to measure the outcome.
- Content Scope: What will you show and what won’t you? No personal data, confidential pricing, customer panels, or internal documents. Allow access to FAQs, product pages, warranty policies, terms of service, and public价格
- Who to Allow: Create an allowlist (a list of who is allowed in) of known AI agents. Start with major players and those from whom you see traffic or sales. Block or limit the rest.
- Limits and Security: Set entry speed (to prevent the bot from slowing down your site), hours (e.g., nighttime windows), and a ‘budget’—how many pages the bot can visit daily. Add extra security measures: IP blocks for un
- known bots.
- Measuring Results: Tag links with UTM (a suffix added to links like ?utm_source=agent), prepare dedicated FAQs/landing pages with clear calls to action, and use discount codes specifically for agents. Set goals in your
Setting up robots.txt for AI agents is now a business decision: who can enter, what content they can access, what limits to set, and what results to expect. Organize these five steps, and you’ll gain traffic from AI responses without risking your data. Want to get this done in 60 minutes? Schedule a brief consultation—I’ll help you set up your policy and measurement.
Frequently asked questions
Does robots.txt really protect content?
Robots.txt is just a guideline for bots. Good bots follow it, but malicious ones may ignore it. Therefore, combine robots.txt with other methods: IP blocks, rules in your application firewall (WAF), and protection for logged-in areas.
Where can I get a list of agents for the allowlist?
Check the documentation from providers and server/Cloudflare logs. Look for ‘User-Agent’ names in traffic and compare them with official descriptions (like help pages from manufacturers). Only allow verified agents; set stricter rules for the rest.
Will blocking AI agents harm SEO?
Not necessarily. Traditional SEO mainly involves Googlebot/Bingbot. You can have different rules for AI agents than for regular search engines. Test on selected subpages and monitor traffic and conversions before making a global decision.
How can I measure revenue from AI agent traffic?
Add UTM to links, use dedicated discount codes, and set goals in analytics (lead, cart, purchase). Compare conversion rates and cart values from the ‘AI/agents’ channel to other sources. This gives a clear picture of ROI.
What are AEO and GEO in simple terms?
AEO (Answer Engine Optimization) is preparing content so that it gets picked up by AI responses in search engines. GEO is a similar approach for ‘generative engines.’ In practice, it’s about traffic and sales from AI responses, not just links.