Crawl a website directory

From the Dashboard or Projects page,

1. Select the Create New Robot button.
2. Select Crawler.
3. Enter the required information.

4. Under the Settings tab, make any desired configuration changes.

• If you wish to use input data to provide the crawler with URLs to visit, activate the Dynamic URL? checkbox.
• To follow the rules of any robots.txt files on the site (recommended), activate the Respect robots.txt? checkbox.

5. Under the Output tab, create any output fields your project requires. These fields will store the output data generated by the crawler.

6. Under the Page Processors tab, configure any page processors required by your project. See Page processors for details.

7. When all necessary page processors are configured, select the blue Save button in the top-right of the page to save the crawler.

On the Projects page,
8. Select the crawler.
9. Select the Create Run button near the top-right of the page.

10. Enter a name for the run.

11. Select the New Run.
12. Select Open in the slide-in panel.
13. Under the Configuration tab, change settings as needed.

14. Under the Integrations tab, configure any required integrations.

15. Under the Executions tab, you may launch the execution when ready, or view existing execution information.