Guide · 4 Mar 2025 · 5 min read
What an AI crawler can and cannot read on your homepage
Start with robots, sitemaps and structured data. A walk through the checks BrandPilot runs.
Start with access
Before any system can recommend your business, it has to be allowed to read your pages. Open yourdomain.com/robots.txt. If it contains a blanket Disallow for all user agents, or blocks the crawlers of the assistants you care about, nothing else on this list matters. A BrandPilot audit reports the robots.txt status code and whether the homepage path is allowed.
Give crawlers a map
A sitemap.xml lists the pages you want discovered. It should be referenced from robots.txt, return a 200, and contain canonical URLs only. Sites with a missing or empty sitemap force crawlers to guess, and the pages they guess are rarely your service pages.
Say what you are in structured data
JSON-LD types such as Organization, LocalBusiness, Service and FAQPage turn marketing prose into facts a machine can quote: legal name, address, opening hours, services offered. Add the types that are true for your business, keep them in sync with visible text, and avoid decorative markup you cannot back up.
Make the page legible without JavaScript
Many crawlers read the initial HTML only. Check that your title, meta description, one clear H1 and the main body text are present in the server response, and that the canonical tag points to the version of the URL you want cited.
Declare your language
Set the lang attribute on the html element, and use hreflang if you publish Arabic and English versions. A page that states its language and points to its translation is far easier to match to a question asked in that language.