Wiki
Getting Started
What is CrawlEZ?
CrawlEZ is an AI-visibility platform for WordPress. It does two things: (1) it generates and maintains the files AI systems read about your site — llms.txt and schema.org structured data — and (2) it fixes the on-page SEO, GEO (AI-readiness) and performance issues that hold your site's visibility score down. You install a lightweight WordPress plugin, connect it to your CrawlEZ account with a GUID, and it keeps everything in sync automatically.
Does CrawlEZ fix the issues the AI-Visibility Check reports?
It fixes the ones a WordPress plugin genuinely can — and it won't pretend to fix the rest. The Check scores three areas: SEO, GEO (AI readiness) and Performance. The AI-readiness work is automatic the moment the plugin is active: it generates and maintains your llms.txt and schema.org data and fills the on-page SEO gaps (meta description, Open Graph/Twitter, canonical, viewport, heading structure, FAQ structured data, robots.txt AI-crawler access). For Performance, it fixes your compression and browser-caching issues for you — in one click via "Scan & fix automatically", on Apache hosting. What no plugin can fix, and we don't claim to: your server's response time (TTFB), a heavy page-builder theme, your image file sizes, real page content, internal linking, and your HTTPS certificate. In short — installing CrawlEZ handles your llms.txt and AI-readiness outright and fixes compression and caching for you; it does not, and cannot, make every score perfect on its own.
Run the AI-Visibility Check →How does CrawlEZ work?
When you publish or update content on your WordPress site, the CrawlEZ plugin detects the change and sends a notification to our platform. Our AI agent receives the signal, calls back to your plugin, and instructs it to rebuild your llms.txt and schema.org files with the full, up-to-date content of your site. The loop is: plugin detects change → notifies agent → agent triggers regeneration → plugin rebuilds files.
Is the WordPress plugin free?
Yes. The CrawlEZ WordPress plugin is free to download and install. You can get it from the plugin page and it requires no payment to run on your WordPress site.
Download the plugin →What WordPress version do I need?
CrawlEZ requires WordPress 6.0 or higher and PHP 8.0 or higher. It has no other dependencies — no page builder, no WooCommerce, no specific theme. It works on any standard WordPress installation.
The Plugin
How do I install the CrawlEZ plugin?
Sign up for a CrawlEZ account and we'll take you straight to your onboarding page. Click the Download CrawlEZ plugin button there to get your personalized ZIP — your Site GUID is already baked in, so there's no copy-and-paste step. In your WordPress admin go to Plugins → Add New → Upload Plugin, choose the ZIP, click Install Now, then Activate. The moment activation finishes the plugin connects to CrawlEZ automatically and your dashboard opens.
Go to onboarding →Where do I find my Site GUID?
You don't need to copy it anywhere — the personalized plugin you downloaded already has your GUID baked in, and the connection happens automatically the moment you activate the plugin. If you ever need to see or share it (for support or debugging), go to Settings → CrawlEZ in your WordPress admin, where the GUID is displayed prominently at the top with a Copy button.
What happens the moment I activate the plugin?
Three things happen automatically on activation: (1) a unique GUID is generated and stored for your site, (2) your llms.txt and schema.org files are generated from all current published content, and (3) WordPress rewrite rules are registered so your files are served at /llms.txt and /crawlez/schema.json. You're fully set up before you even leave the activation screen.
Can I regenerate the files manually?
Yes. In Settings → CrawlEZ, there is a Regenerate Now button. Click it to force a full rebuild of both files on demand. This is useful after bulk imports, theme changes, or any time you want to ensure the files are current without waiting for a publish event.
Does the plugin slow down my site?
No. The plugin operates entirely in the background. File generation writes to the WordPress options table (not disk), and the webhook notification to the agent uses a non-blocking HTTP call — WordPress fires it and moves on without waiting for a response. Your publish action completes at normal speed.
Does the plugin update itself?
Yes, on a paid plan. CrawlEZ ships a private update channel: while your subscription is active the plugin checks in with the platform and applies new versions automatically through WordPress's normal background updater — no re-downloading or re-installing. Before every scheduled content refresh it also runs an update check first, so your AI files are never rebuilt on an out-of-date version. Sites without an active subscription keep running their current version and simply don't receive the automatic updates.
What CrawlEZ Optimizes
What does the plugin fix once it's activated?
On activation the plugin starts improving how AI engines and search engines see your homepage. SEO: it fills a missing meta description, Open Graph and Twitter Card tags, adds a canonical URL and a mobile viewport tag when your theme omits them, and tidies your heading structure (one H1, no skipped levels) in your content. GEO (AI readiness): it serves llms.txt, injects Organization, WebSite and FAQPage structured data, and keeps AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) allowed in robots.txt. These run at page-render time and never overwrite content you've already set — they only fill gaps.
Is there a one-click way to fix what the AI-Visibility Check found?
Yes. On Settings → CrawlEZ → SEO there is a "Scan & fix automatically" button. It scans your live homepage, turns on exactly the fixers for the problems it finds, and then shows you two lists: what it fixed automatically, and what still needs you (with a plain-English reason for each — things like installing an SSL certificate, writing more page content, or upgrading your hosting). It only enables fixes for problems that were actually reported, and it never overwrites settings you already chose.
How do I add FAQs that AI engines can quote?
Go to Settings → CrawlEZ → Organization and use the FAQs editor to add question-and-answer pairs. CrawlEZ publishes them as FAQPage structured data on your homepage, which AI assistants quote directly and which improves your GEO score. Add as many as you like; incomplete pairs (a question with no answer) are simply skipped.
Can CrawlEZ improve my performance score?
Yes for the parts a plugin can fix, and it's clear about the parts it can't. The fixes it owns are gzip compression and browser caching — turn them on with one click via "Scan & fix automatically", or from Settings → CrawlEZ → Speed (these write to .htaccess, so they apply on Apache hosting). The Speed tab also lets you defer JavaScript, remove emoji scripts, and tune the WordPress heartbeat. Note these performance fixes are opt-in, not automatic on activation — unlike the llms.txt and SEO/AI-readiness work, which is automatic. What a plugin cannot fix is raw server response time (TTFB), a heavy page-builder theme, or your image file sizes — those depend on your hosting and theme, and CrawlEZ tells you that plainly rather than claiming a perfect score.
My site is hidden from search engines — can the plugin fix that?
Yes, with one click. If WordPress is set to "Discourage search engines" (Settings → Reading), your visibility is capped no matter what else you do. CrawlEZ detects this and shows an "Allow search engines to index this site" button on the SEO page. It never flips this for you automatically — you stay in control — but the fix is a single click when you're ready.
Will CrawlEZ conflict with Yoast, Rank Math or another SEO plugin?
No. CrawlEZ detects popular SEO plugins (Yoast, Rank Math, SEOPress, The SEO Framework) and, by default, steps aside so it never emits duplicate tags. If you'd rather CrawlEZ handle a given area, you can switch the relevant setting from "Auto" to "Always on" in its admin card.
Managing Your Websites
How do I add another website?
From your CrawlEZ dashboard you can add another website at any time, as long as you're within your plan's site limit. Each website gets its own connection and its own personalized plugin download — install and activate that plugin on the new WordPress site exactly as you did the first, and it connects automatically. Adding a site uses one slot from your plan; if you're at your limit, upgrade your plan or free a slot by removing a site you no longer track.
How do I remove a website?
Open the website in your dashboard and choose Delete. This removes the website from your account and frees up the slot so you can add a different site within your limits. Deleting a website here is about your CrawlEZ account — to also stop the plugin running on that WordPress site, deactivate and delete the CrawlEZ plugin in that site's WordPress admin.
What happens if I delete a website by mistake?
Deleting a website is permanent — its connection and tracked data are removed. There is no undo. If you want to track that site again you'll need to add it back as a new website and reconnect it: download the fresh personalized plugin for the re-added site and activate it on WordPress (a new Site GUID is issued). In short: you can freely add and remove sites within your plan limits, but a deleted site must be re-added and reconnected from scratch — it does not come back on its own.
llms.txt & schema.org
What is llms.txt and why does it matter?
llms.txt is a plain-text file served at the root of your site (e.g. yoursite.com/llms.txt) that gives AI language models a structured, crawlable summary of your entire site. It follows the llmstxt.org specification and includes your site name, tagline, a linked index of all your pages and posts, and the full text content of your key pages. AI assistants, search agents, and LLM-powered tools read this file to understand your site without having to parse HTML.
What is schema.org / schema.json?
schema.org is a standardised vocabulary for describing content on the web. CrawlEZ generates a JSON-LD file at /crawlez/schema.json that describes your site as a WebSite, your organisation, and every published page and post as a WebPage or BlogPosting — including publication dates, authors, categories, and tags. Search engines and AI crawlers use this structured data to understand your content at a semantic level.
Where are my generated files served?
Your llms.txt is served at yoursite.com/llms.txt and your schema.org file at yoursite.com/crawlez/schema.json. Both are virtual files — they are served dynamically by WordPress via rewrite rules, not as files on disk. If your permalinks are set to Plain, you may need to go to Settings → Permalinks and click Save Changes to flush the rewrite rules after activating the plugin.
How current is the content in my files?
The files are regenerated automatically every time you publish, update, unpublish, trash, or permanently delete a post or page. They always reflect the complete set of currently published content on your site. The files are also regenerated on demand via the Regenerate Now button in settings, or when the CrawlEZ agent receives a content-change notification from your plugin.
What is llms-full.txt?
llms-full.txt is the expanded companion to llms.txt, served at yoursite.com/llms-full.txt. Where llms.txt is a linked index, llms-full.txt contains the complete text of every published page and post compiled into a single file — so an AI system can read your whole site in one fetch without crawling each URL. CrawlEZ builds and refreshes it automatically alongside your other files; there is nothing to configure.
Can I keep certain pages out of the AI files?
Yes. On Settings → CrawlEZ → SEO there is an "Exclude content from AI files" panel listing your published pages and posts. Tick anything you don't want shared with AI systems — legal boilerplate, thin or internal pages, and so on — and save. The excluded items are dropped from both llms.txt and llms-full.txt, and the files are rebuilt immediately. You can re-include them any time by un-ticking them.
The Agent
What does the CrawlEZ agent do?
The CrawlEZ agent is the orchestration layer between your WordPress site and the platform. When your plugin notifies us of a content change, the agent receives the instruction, calls your plugin's secure REST endpoint with your GUID, and triggers a full content regeneration. It confirms success and reports back the URLs of the regenerated files.
How does the agent authenticate with my plugin?
The agent authenticates using your Site GUID as a secret key. It sends the GUID in the X-CrawlEZ-GUID HTTP header when calling your plugin's REST endpoint. Your plugin validates this with a constant-time comparison (hash_equals) so timing attacks are not possible. The GUID never appears in public and is only shared between your WordPress settings and your CrawlEZ account.
What endpoint does the agent call on my WordPress site?
The agent calls POST /wp-json/crawlez/v1/regenerate on your WordPress site. This endpoint is registered by the CrawlEZ plugin and is only accessible with a valid GUID. It is hidden from the public REST API index so it does not appear in /wp-json/ discovery.
Troubleshooting
My llms.txt or schema.json returns a 404. What do I do?
This almost always means WordPress's rewrite rules need to be flushed. Go to Settings → Permalinks in your WordPress admin and click Save Changes — you don't need to change anything, just click the button. This re-registers the rewrite rules and your files will be available immediately. If the 404 persists, deactivate and reactivate the CrawlEZ plugin.
My files exist but the content looks outdated. What do I do?
Click Regenerate Now in Settings → CrawlEZ. This forces a full rebuild from all current published content. If you're seeing cached content in your browser, try opening the file URL in a private/incognito window. If the problem persists, check that the CrawlEZ plugin is the active version and that no caching plugin is serving a stale copy of the URL.
The agent isn't being notified when I publish. What do I do?
Check that your site can make outbound HTTP requests — some hosting providers block them by default. Go to Settings → CrawlEZ and confirm the Webhook endpoint shown there is reachable from your server. You can also verify by clicking Regenerate Now, which triggers regeneration locally without the agent, to confirm the plugin itself is working.
I lost or need to reset my Site GUID. What do I do?
Your GUID is stored in the WordPress options table under the key crawlez_guid and is always visible in Settings → CrawlEZ. It does not expire or rotate automatically. If you believe your GUID has been compromised, deactivate and delete the plugin, reinstall it — a new GUID will be generated on activation. You will then need to update the GUID in your CrawlEZ account.
I connected my GUID during onboarding but the agent isn't responding. What do I do?
Make sure your WordPress site is publicly reachable (not behind a local dev environment, VPN, or firewall). The agent calls your plugin's REST endpoint from our servers, so it must be accessible via the public internet. You can test by visiting yoursite.com/wp-json/crawlez/v1/status in a browser — if you see a JSON response, the endpoint is live.
Still have questions?
Reach us at support@crawlez.net