This guide shows how to allow the crawler safely, using Cloudflare’s recommended approach.
Before You Start
You must have access to your site’s Cloudflare dashboard.
You should be comfortable making WAF/Security changes.
If your site is protected by Cloudflare Bot features (Bot Fight Mode / Super Bot Fight Mode), you may need to allow/skip those checks as well.
Recommended: Cloudflare WAF Custom Rule (Security Rules)
Cloudflare recommends using WAF Custom Rules (a.k.a. Security Rules) instead of broad IP Access Rules because you can be more precise.
Steps
Log in to Cloudflare and select your domain.
Go to Security → WAF → Custom Rules (or Security Rules, depending on your Cloudflare plan/UI).
Click Create rule.
Name it something like:
Allow Rankley OnPage Audit.Add an expression that matches the crawler.
Match by User‑Agent (most common)
Use a User‑Agent condition. Example:
(http.user_agent contains "RSiteAuditor")
Notes:
RSiteAuditoris a common default User‑Agent used by the Rankley OnPage crawler.If your organization has configured a custom User‑Agent for auditing, use that value instead.
Optional: Restrict by Path
If you only want to allow auditing for your public pages (and avoid admin areas), add exclusions. Example:
(http.user_agent contains "RSiteAuditor") and not (http.request.uri.path contains "/wp-admin") and not (http.request.uri.path contains "/admin")
(Adjust exclusions to match your site.)
Choose the Action
Cloudflare offers a few actions. The safest starting point is usually:
Skip and select only the features you want to bypass (e.g., Managed WAF rules / Rate Limiting / Bot checks), instead of allowing everything.
If you still see blocks after “Skip”, you may need to broaden what is skipped.
Save the rule.
Alternative: IP Access Rules (Simpler, Use Carefully)
You can allow by IP, IP range (CIDR), or ASN using Security → Tools → IP Access Rules (naming may vary).
⚠️ Important: Cloudflare warns that an Allow IP Access Rule can bypass other security checks (WAF managed rules, rate limiting, etc.). Use it only if you fully trust the traffic source and you’re confident you’re allowing the correct IP range.
When to use this
You have an explicit list of crawler IPs/ranges from the crawler provider.
You need a fast fix and can accept the security tradeoff.
Steps
Cloudflare dashboard → select your domain.
Go to Security → Tools → IP Access Rules.
Click Add.
Enter the IP / CIDR range / ASN.
Action: Allow.
Scope: this zone (recommended unless you truly want all zones).
Add a description and save.
After Making Changes: Verify
Re-run the On‑Page audit in Rankley.
If it still shows 0 pages crawled:
Check Cloudflare Security Events for blocks.
If the event shows Bot/WAF blocks, adjust the WAF Custom Rule action to skip the specific blocking feature.
Still Blocked?
Some sites block crawlers based on additional signals (JavaScript challenges, bot scoring, country/ASN, or strict rate limits).
What you can try:
Enable Switch proxy pool in Rankley’s On‑Page crawl options (helps if blocks are IP/ASN based).
Enable Support cookies and Disable cookie popup in Rankley’s On‑Page crawl options (helps with cookie banners/consent flows).
Share a Cloudflare Security Event screenshot (redact sensitive details) with support so we can identify what is being blocked.
Security Notes
Prefer WAF Custom Rules over global allowlists.
Prefer Skip specific features over broad “Allow” when possible.
Avoid allowlisting admin/login paths.