Tools and providers
Technical SEO survey tools compared by crawl limit: Sitebulb per audit, JetOctopus per month and the rules both obey
Compare Sitebulb and JetOctopus for technical SEO site surveys by crawl limit, seats and published price, plus the RFC 9309 and RFC 9110 rules crawlers follow.
What to take away
- Technical SEO survey tools crawl a site and report status codes, redirects and robots rules. It is a different product from a questionnaire platform.
- Sitebulb caps crawls per audit10,000 URLs on Lite and 500,000 on Pro. JetOctopus caps crawled URLs per month: 500K, 2M or 5M.
- Seat rules differ. Sitebulb charges from £7 a month for extra users; JetOctopus lists unlimited seats on every plan.
- Every crawler should follow the Robots Exclusion Protocol in RFC 9309 and report HTTP status codes as RFC 9110 defines them.
Two ways to limit a crawl
A crawler needs a limit, because a site can have millions of URLs. Desktop software caps each audit and leaves the machine to cope. Cloud software sells a monthly quota and runs on its own servers. That one difference decides which plan fits a client site.
Sitebulb is desktop software for Windows or Mac. Its Lite plan allows 10,000 URLs per audit for one user. Its Pro plan allows 500,000 URLs per audit, and the page says extra users start from £7 a month. Yearly plans save 15 per cent.
Sitebulb: the desktop crawler
Sitebulb Pro runs on your own computer, so the machine limits the size of site you can crawl. Most normal work computers can handle sites up to 500,000 URLs, the default limit. You can raise it to 2 million URLs, at the risk of stretching the computer.
A 14-day free trial is available. For a small agency the useful questions are the audit cap, the number of users and whether the client site needs JavaScript rendering. Check these against the quality checklist before you buy.
JetOctopus: the cloud crawler
JetOctopus prices by monthly crawled URLs. The Growth plan covers 500K crawled URLs a month, Pro covers 2M and Ultra covers 5M, with a custom option above that. Seats are unlimited on every plan, and data is kept for 12 months on the three standard plans.
Listed prices are in euros or dollars, with a 25 per cent saving when billed annually. On the annual toggle Growth shows €299 a month, Pro €699 and Ultra €1,359. On quarterly billing the same plans show €399, €949 and €1,799. Ask for a sterling quote, with VAT, before comparing.
Side-by-side comparison
| Point | Sitebulb | JetOctopus |
|---|---|---|
| Where it runs | Your Windows or Mac machine | Vendor cloud |
| Crawl limit | 10,000 URLs per audit (Lite), 500,000 (Pro) | 500K, 2M or 5M crawled URLs a month |
| Seats | One user; extra users from £7 a month | Unlimited on every plan |
| Published price | Plan prices not shown in page text; extra users from £7 a month | €299, €699 or €1,359 a month, billed annually |
| Free trial | 14 days | Ask the vendor |
Illustrative arithmetic: a four-person team on Sitebulb Pro pays for one user plus three extra users, so the extras add 3 × £7 = £21 a month. A site of 120,000 URLs crawled weekly uses about 4 × 120,000 = 480,000 crawled URLs a month, which fits inside the 500K Growth quota only just.
The rules every crawler follows
A survey tool is a crawler, and the Robots Exclusion Protocol sets its manners. Under RFC 9309, the rules must be accessible in a file named /robots.txt in the top-level path of the service. The /robots.txt URI is implicitly allowed, so a crawler can always fetch it.
The protocol also requires that, if no matching group exists, crawlers obey the group with a user-agent line with the "*" value, if present. Check that your survey tool identifies itself with a product token, so the right group is applied.
Status codes in the report
A survey report is mostly a list of HTTP status codes. Under RFC 9110, the 301 status code indicates that the target resource has been assigned a new permanent URI. The 404 status code indicates that the origin server did not find a current representation for the target resource.
Use those definitions when you brief a client. A 301 is a planned move, so check the destination. A 404 is a missing page, so check whether it should exist. A tool that reports both under one heading hides the difference.
What to check before you buy
- Count the URLs on the largest client site, including parameters.
- Decide between a per-audit cap and a monthly quota.
- Count the users who need an account, and the price of each extra seat.
- Ask for a sterling price with VAT and the billing period.
- Confirm that the crawler obeys robots.txt and identifies itself.
- Run a trial crawl and compare the status-code counts with the server logs.
See the best providers shortlist, the pricing models comparison and the vendor checks before you sign. For free alternatives for site owners, read Search Console alternatives. Use the selection checklist for the final decision.
Common questions
Is a survey tool the same as a crawler?
In technical SEO, yes: it surveys a site by following links and recording what each URL returns. A questionnaire platform is a different product.
Which limit suits a large site?
A monthly quota suits a site you recrawl often. A per-audit cap suits occasional deep audits. Work out the number of URLs and the crawl frequency first.
Does a crawler have to obey robots.txt?
The Robots Exclusion Protocol sets the rules that crawlers are requested to honour. Configure the tool to follow them unless you own the site and have a reason to override them.
