Pipes Robot Settings

Pipes robots are the Swiss-army knives of Dexi: they can do pretty much anything. However, with great flexibility comes the potential for great confusion.

For an introduction to Pipes robots, head on over to Pipes.

Executing other robots (Default configurations)

To execute another robot in a Pipes robot, use the Execute robot action:

When the Pipes robot is then executed (or more precisely, when a configuration of the Pipes robot is executed), a Default configuration of the other robot is automatically created by the platform.

The Default configuration is identical to any other standard robot configuration. However, to avoid confusing and unpredictable behavior, the fields of Default configurations cannot be changed except for the Proxies field.

This enables the user to control which proxies are being used for the executions of the Default configuration (the proxy specified on the Pipes configuration is not propagated to the Default configuration). See What should I know about proxy servers and anonymity? for more information about proxies.

Two other important settings, the maximum number of retries and maximum concurrency, should be controlled from the Execute robot action as shown in the screenshot above.

Max retries

Maximum number of retries has a couple of different meanings and can be controlled from a couple of different locations:

• Maximum attempts on a Pipes configuration: how many times to retry execution of each input/result of the configuration. Note! Pipes by design rarely fail so it rarely makes sense to change this option.
• Max retries on an Execute robot action: how many times to retry executions of the Default configuration.
• Maximum attempts on a Default configuration: how many times to retry execution of each input/result of the configuration. Has no effect and cannot be changed. Use Max retries on Execute robot action instead.

To read more about retries, see section About Retries on How can I create a run?

Max concurrency

Similarly, maximum concurrency has a couple of different meanings and can be controlled from a couple of different locations:

• Concurrent inputs/results on a Pipes configuration: the number of inputs/results on the Pipes configuration that can run concurrently
• Max concurrency on an Execute robot action: the number of executions of the Default configuration that can run concurrently. See details in "Calculating Worker Usage" section below.
• Concurrent inputs/results on Extractor Default configuration: the number of inputs/results on the configuration that can run concurrently. Has no effect and cannot be changed. Use Max concurrency on Execute robot action instead.

Calculating Worker Usage

When a Pipes robot executes another robot (via the "Execute robot" action), a child execution (of the Default configuration of that other robot) starts running (if rate limiting allows it). Each such child execution consumes a single worker (if the child execution is of a Pipes robot, the child executions of that robot will also consume a single worker - and so on for as many levels as exist).

Generally, the number of Pipes child executions is controlled by the "Max concurrency" value of its predecessor "Execute robot" action - OR - in the case of top-level "Execute robot" actions, by "Concurrent inputs" on Pipes config.

The sub sections below present a few examples of how to calculate the number of workers used by Pipes executions. Each sub section describes the worker usage of the corresponding "level" in the Pipes robot, depending on the design of your Pipes robot.

Pipes configuration level

A single worker is used for each concurrent input defined on its configuration, e.g. 20 concurrent inputs uses 20 workers.

So if your Pipes robot does not have any "Execute robot" actions, only 20 workers are used.

Top-level

A top-level action in a Pipes robot is an action that does not have any predecessor action.

As mentioned above, "Concurrent inputs" on the Pipes configuration controls the worker usage of any top level "Execute robot" action, ie. no matter what value of "Max concurrency" is specified on any such action, it is overwritten by the value of "Concurrent inputs" on the Pipes configuration. A single worker is used for each top-level child execution.

Continuing the example above, if the Pipes robot has a single top-level "Execute robot" action, 20 workers are used for the concurrent inputs on the Pipes configuration and 20 workers are used for the top-level child executions , ie. 20 + 20 = 40 workers are used in total.

Nested levels

The total worker usage of a Pipes execution that has "nested" "Execute robot" actions is calculated as follows:

1. Workers used by top-level, level_1: concurrency_pipes_config + concurrency_top_level - where concurrency_pipes_config == concurrency_top_level, as mentioned above.
2. Workers used by nested level l, level_l: concurrency_level - 1 * concurrency_level_l
3. Maximum worker usage: level_1 + sum(level_2, ..., level_l, ... level_n)

Continuing the example above, if the top-level "Execute robot" action has a "Max concurrency" value of 30 and that action has a single, nested "Execute robot" action with a "Max concurrency" value of 5, the maximum number of workers that can be used by the Pipes execution is:

level_1: 40 workers (from section "Top-level" above)
level_2: 20 * 5 = 100 workers (as mentioned above, the "Concurrent inputs" on the Pipes configuration overwrites the "Max concurrency" on the top-level "Execute robot" action, so the calculation above uses 20, not 30)

Maximum worker usage: 40 + 100 = 140 workers

If the Pipes robot has another nested level that contains an "Execute robot" action, say, with a "Max concurrency" value of 7, the maximum number of workers that can be used by the Pipes execution is:

level_1: 40 workers (from section "Top-level" above)
level_2: 20 * 5 = 100 workers
level_3: 5 * 7 = 35 workers

Maximum worker usage: 40 + 100 + 35 = 175 workers

More Examples

Multiple top levels

The screenshot below shows an example of a Pipes robot that has two top-level "Execute robot" actions:

The maximum worker usage can be calculated as follows:

"Concurrent inputs" on the Pipes configuration: 11

level_1_1: 11 + 11 = 22 workers
level_2_1: 11 * 5 = 55 workers
level_3_1: 5 * 7 = 35 workers

Worker usage for graph 1: 22 + 55 + 35 = 112 workers.

level_1_2: 11 + 11 = 22 workers
level_2_2: 11 * 4 = 44 workers

Worker usage for graph 2: 22 + 44 = 66 workers.

Maximum worker usage: 112 + 66 = 178 workers

Pipes robot within a Pipes robot

The screenshots below an outer Pipes robot executing an inner Pipes robot.

Outer Pipes Robot	Inner Pipes Robot

The maximum worker usage does not follow the same formulae as above but can instead be calculated as follows:

"Concurrent inputs" on the Pipes configuration: 3

Outer Pipes robot:

level_1: 3 + 3 = 6 workers
level_2: 3 * 1 = 3 workers
level_3_1: 1 * 1 = 1 worker
level_3_2: value of "Max Concurrency" is disregarded

Worker usage for outer robot: 6 + 3 + 1 = 10 workers.

Inner Pipes robot:
The inner Pipes robot uses the value of "Concurrent Inputs" set on the configuration of the outer Pipes robot:

level_1: 3 * 3 = 9 workers
level_2: 3 * 3 = 9 workers

Worker usage for inner robot: 9 + 9 = 18 workers.

Maximum worker usage: 10 + 18 = 28 workers

Worker usage limits

From the Robot Settings tab of your account settings, you can control how many workers can be used for each robot type.