How many pages can I crawl with the Optimizer?

The Optimizer Module will crawl the site until all linked pages are found. You may, however, add additional link sources for us to crawl which might expose content that is not internally linked.

In total, across all projects in the Optimizer, you can crawl up to 500,000 URLs. The maximum for one domain is 250,000 but if you want to manually set limits for each project, you can do so through the management interface at the bottom of the Optimizer home page.

Project management interface

If you need to crawl more than 250,000 URLS per site, please contact support.