The Crawler Restriction option enables you to control how the FusionBot crawler handles links to content throughout your site. Depending on how you have designed your site will determine which option is optimal for indexing as many pages as possible on your site. Following are the three options available and how FusionBot will navigate your site and create your searchable content as a result of your selection:

Option 1 (Default) - Directory Level Indexing:  Selecting this option will cause the FusionBot crawler to only follow links on your site that are contained directly within or below the DIRECTORY path provided in the Domain Path / URL field. This allows you to create an index that restricts the searchable content of your site to just a specific portion of your site.

For example, entering as your Domain Path:



Option 2 - Server Level Indexing:  Selecting this option will cause the FusionBot crawler to index ALL pages it finds on your site regardless of their location, whether or not you have included multiple directories as part of your Domain Path / URL. This is useful for those who wish to provide a starting point for FusionBot to begin crawling their site that is multiple directories deep, however, they still wish for their entire site to be crawled. For example, entering as your Domain Path:



Option 3 - Domain Level Indexing:  Selecting this option will cause the FusionBot crawler to index ALL pages it finds on your site regardless of the directory in which they are located AND regardless of the server / hostname in which these files are hosted. This is the most liberal option, enabling FusionBot to follow as many links as possible on your site, as long as they match the DOMAIN portion of your Domain Path / URL. For example, if you have subscribed the following URL: