Pipes

A Pipes robot connects other robots and data together and can apply a transformation to that data. Pipes can also interact with third-party services like API's, send HTTP requests, and more. We often refer to Pipes as a super robot because of its ability to control and govern other robots.

A few examples of uses for Pipes are:

• Execute multiple extractors and/or crawlers, connecting outputs to inputs
• Transform text and images
• Perform calculations
• Join separate datasets
• Parse JSON and XML
• Send HTTP requests
• Send data to an SQL database
• Read data from datasets
• Do lookups in a dictionary
• Interact with third-party applications like Google Geocoding or Foursquare

In real-world use, this means easier price comparisons, simple image resizing, unifying datasets from multiple websites, and normal Extract-Transform-Load (ETL) operations.

Adding nodes

A Pipes robot contains nodes, each of which performs a specific action. To add nodes to your pipe simply drag and drop them from the right-hand list of actions onto the pipes grid:

Debugging

You can also debug pipes robots – both while they're running and once they've finished. You do this by opening the execution and selecting the green bug icon found next to each result.

Read more about debugging

Limitations

Because of the extremely flexible and technical nature of Pipes robots, there are some limitations on their design and the amount of data they support.

Multiple inputs to the same connector
A node in a Pipes robot needs values for all connectors that have inputs before it can run. However, if a single input connector receives values from multiple nodes, it can lead to unpredictable results:

In the example above, the title connector has multiple input values.

Dexi will try to figure out how to use the inputs but this might not always produce the result that you expect. As an example, it can lead to nodes, following the node with multiple input values for the same connector, not being executed.

We advise against using this design.

Unlimited number of rows not supported
Pipes robots are fully streaming — they process rows on a true row-by-row basis but they might not always support very large datasets.

Limits depend on the size of each row as well as the design of the Pipes robot.

We recommend creating mulitple Pipes robots as needed.

Collect action
Specifically, the Collect (and Collect Bulk) action can cause issues. The action awaitsall input data from all previous (ancestor) nodes before it will run.

An execution of a Pipes robot with this issue can fail with an empty result log because the process running the Pipes robot runs out of memory and crashes.