AutoBots

What are AutoBots?

AutoBots are, in addition to being our allies in the struggle against the Decepticons, a way of organizing many extractor robots that should extract the same type of data given the same type of input.

An example would be that you'd want to extract product details from a wide range of sites using the product URLs on the individual sites. AutoBots will map these URLs to the proper Extract robots and provide a single endpoint for your integrations to get any product extracted.

Additionally, AutoBots allow you to query for data from URLs that are not yet created for the AutoBot, which will automatically create a new entry, allowing you to create the proper Extractor at a later point.

Input
The first thing you'll need to determine is what input your AutoBot, and thus all Extractor robots within, will require. The url field is required since that is what the AutoBot uses for mapping URLs to robots, but you can add additional fields like query if your AutoBot should search on various pages for a search string.

You can also use a data type for input, as long as it contains a url field as well.

All extractor robots within the AutoBot will have the exact same input fields as the AutoBot itself.

Output
Next, you'll determine what output fields the robot should have. The same applies here as with input fields: all robots within this AutoBot will have the exact same output fields as the AutoBot itself.

Unlike the input fields, output fields do not require any specific field to be available.

Sites
The sites section shows you the mapping between URLs (domains) and robots. Any request to the AutoBot for a specific domain will go to the same Extractor robot. If an entry in the mapping is not marked as Ready, the execution will fail.

As you execute the AutoBot with various URLs, it will automatically populate the Sites mapping with new empty entries for unknown entries. It will also automatically add more examples to the existing domains, allowing you to test your robot with more URLs and when a site changes.

You can also add URLs manually to the AutoBot through the Add urls button which opens a dialog that allows you to paste in several urls, one per line.