Batch execute tasks
Task in alpha phase
This Data Factory task is in phase. You can contact the Product-Live team at contact@product-live.com if you want more details and get an early access.
This task executes a list of Data Factory tasks as a single batch: each entry of the list is one execution of an allowed Data Factory task (see Allowed tasks) with its own input parameters. The entries are distributed over 1 to 5 parallel slots (concurrency) and run sequentially inside each slot; the per-entry results are aggregated into a single results list, always returned in input order, delivered under result.<outputKey> inline and/or as JSON/XML files (one key per outputs[] entry), together with a summary (total, success and error counts, duration). Use it to run N independent executions of the same task (or of different tasks) from a single job step - for example N HTTP calls, N scripts or N item lookups - instead of hand-writing a loop with per-iteration wiring. The batch never stops on a failing entry: remaining entries still run and failures are reported per entry and in summary.errorCount.
Task name:
Examples
json
{
"name": "task-batch-execute",
"taskReferenceName": "run_batch",
"description": "Run two independent script executions as one batch",
"type": "SUB_WORKFLOW",
"optional": false,
"inputParameters": {
"taskName": "code-execute-javascript",
"request": {
"method": "INLINE",
"contentType": "application/json",
"json": {
"tasks": [
{
"key": "calc-1",
"inputs": {
"expression": "return { value: $.data.x * 2 };",
"data": { "x": 21 }
}
},
{
"key": "calc-2",
"inputs": {
"expression": "return { value: $.data.a + $.data.b };",
"data": { "a": 10, "b": 32 }
}
}
]
}
}
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
json
{
"name": "task-batch-execute",
"taskReferenceName": "run_http_batch",
"description": "Perform independent HTTP calls, 4 at a time",
"type": "SUB_WORKFLOW",
"optional": false,
"inputParameters": {
"taskName": "protocol-http-perform",
"concurrency": 4,
"request": {
"method": "INLINE",
"contentType": "application/json",
"json": {
"tasks": [
{
"key": "get-product-1",
"inputs": {
"domain": "api.example.com",
"path": "/v1/products/1",
"method": "GET",
"responses": ["JSON"]
}
},
{
"key": "get-product-2",
"inputs": {
"domain": "api.example.com",
"path": "/v1/products/2",
"method": "GET",
"responses": ["JSON"]
}
},
{
"key": "post-event",
"inputs": {
"domain": "api.example.com",
"path": "/v1/events",
"method": "POST",
"body": { "type": "JSON", "json": { "hello": "world" } },
"responses": ["JSON"]
}
}
]
}
},
"outputs": [
{ "outputKey": "json", "outputMode": "INLINE", "mediaType": "application/json" },
{ "outputKey": "xmlFile", "outputMode": "ATTACHMENT", "mediaType": "application/xml" }
]
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
json
{
"name": "task-batch-execute",
"taskReferenceName": "run_batch_from_file",
"description": "Run the tasks described in a JSON file produced by a previous task",
"type": "SUB_WORKFLOW",
"optional": false,
"inputParameters": {
"request": {
"method": "ATTACHMENT",
"contentType": "application/json",
"file": "${build_tasks_file.output.result}"
},
"concurrency": 2,
"outputs": [
{ "outputKey": "jsonFile", "outputMode": "ATTACHMENT", "mediaType": "application/json" }
]
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Definition
| Property | Type | Required | Description |
|---|---|---|---|
| The name (or type) of the task | |||
| The unique name of the task in your job. It is used to reference this task in the workflow. | |||
| The functional description of this task in your job. | |||
The type of the task. It must be SUB_WORKFLOW. | |||
| : | true: the job continues if there is an error on this task. false: the job fails. | ||
| Input parameters of the task. See below |
Inputs Parameters
| Property | Type | Required | Description |
|---|---|---|---|
The tasks list to execute. Rejected with request is required when absent. | |||
| : | INLINE: the tasks list is provided inline in request.json.ATTACHMENT: the tasks list is loaded from request.file.Any other value is rejected: request.method must be INLINE or ATTACHMENT. | ||
| : | The content type of the tasks list. application/xml is only meaningful with method: ATTACHMENT (an XML tasks file). Any other value is rejected: request.contentType must be application/json or application/xml. | ||
The inline tasks list wrapper. request.json.tasks[] must be an array, otherwise the task fails with request.json.tasks[] is required when method is INLINE. | |||
One entry per task execution. An empty array is valid: the batch completes immediately with empty results and a zeroed summary. | |||
A caller-chosen identifier echoed in the matching result entry. Recommended for readability of results and logs. When omitted, a default task_<n> key is generated from the entry's 1-based position in the input list; default keys are globally unique, whatever concurrency is (see Result ordering). | |||
The Data Factory task to execute for this entry. Overrides the top-level taskName. Must belong to the allowed tasks list, otherwise the batch fails with task[<i>].taskName "<name>" is not an allowed task. | |||
The input parameters passed to the executed task, exactly as you would set them in the task's own inputParameters. Defaults to {}. | |||
A file (an object with a non-empty url) containing the tasks list, in JSON or XML depending on request.contentType. See The tasks list file. Rejected with request.file.url is required when method is ATTACHMENT when absent. | |||
The default Data Factory task to execute, used for every entry that does not set its own taskName. Must belong to the allowed tasks list, otherwise the batch fails with taskName "<name>" is not an allowed task. Every entry must end up with a task name, from this default or from its own taskName. | |||
Default: 1. Min: 1. Max: 5.The number of parallel slots the entries are distributed over (round-robin). Inside a slot, entries run sequentially. 0, negative or non-numeric values are rejected with concurrency must be a number >= 1; values above 5 are rejected with concurrency must be <= 5. A concurrency higher than the number of entries is silently clamped down to it. | |||
Default: [{"outputKey": "json", "outputMode": "INLINE", "mediaType": "application/json"}]The requested result deliveries; several can be combined, and each one is exposed as result.<outputKey>. Supported combinations:INLINE + application/json: the per-entry results array is returned inline in result.<outputKey>.ATTACHMENT + application/json: the results are delivered as a JSON file in result.<outputKey>.ATTACHMENT + application/xml: the results are delivered as an XML file in result.<outputKey>. | |||
The key of the output in the result object. Must be a non-empty string, unique across the outputs entries. A missing/empty key is rejected with "outputs[<i>].outputKey" input parameter is required and must be a non-empty string; a duplicate with "outputs[<i>].outputKey" input parameter must be unique, "<key>" is already used by another output. | |||
| : | The delivery mode of this output. | ||
| : | The media type of this output. application/xml is only available as ATTACHMENT: INLINE + application/xml is rejected with "outputs[<i>].outputMode" input parameter "INLINE" is only supported for "mediaType" "application/json". | ||
| : | Default: warningControls the job logs written by the batch: a start message, one progress log per entry, a completion summary ( Batch completed (...): N tasks in Xs (Y tasks/min) - S success, E errors) and a "top 10 slowest tasks" breakdown. Most of these are info-level; the summary is logged as warning when at least one entry failed.This value is also propagated to every executed task, so a single logLevel governs the verbosity of the whole batch. An entry that sets its own logLevel in inputs overrides it. |
The tasks list file
With request.method: ATTACHMENT, the file must contain the same tasks[] entries as the inline form (key, optional taskName, inputs), either as JSON or as a generic XML document:
json
{
"tasks": [
{
"key": "get-product-1",
"taskName": "protocol-http-perform",
"inputs": {
"domain": "api.example.com",
"path": "/v1/products/1",
"method": "GET",
"responses": ["JSON"]
}
},
{
"key": "get-product-2",
"taskName": "protocol-http-perform",
"inputs": {
"domain": "api.example.com",
"path": "/v1/products/2",
"method": "GET",
"responses": ["JSON"]
}
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
xml
<map xmlns="http://www.w3.org/2005/xpath-functions">
<array key="tasks">
<map>
<string key="key">get-product-1</string>
<string key="taskName">protocol-http-perform</string>
<map key="inputs">
<string key="domain">api.example.com</string>
<string key="path">/v1/products/1</string>
<string key="method">GET</string>
<array key="responses">
<string>JSON</string>
</array>
</map>
</map>
</array>
</map>1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
The XML form uses the generic map / array / string / number / boolean representation (each element carrying a key attribute), so any JSON inputs structure can be expressed.
Allowed tasks
Only a fixed whitelist of Data Factory tasks can be executed in a batch. The task names (the top-level taskName and every entry's taskName) are checked against this list before anything runs; an unknown name fails the batch immediately with taskName "<name>" is not an allowed task (or task[<i>].taskName "<name>" is not an allowed task for a list entry).
The list covers, by family:
- Product-Live data (
data-*) - all the find / create / patch / delete wrappers (accounts, items, jobs, job executions, publications, suggestions, projects, user groups, variables...). - Job context -
context-account-get,context-user-get. - Code execution and transformation -
code-execute-javascript,json-transform-jq. - HTTP -
protocol-http-perform. - Files -
file-transformation-xslt,file-transformation-merge-xml,file-validation-xsd. - Tables -
table-import-items,table-import-schema,table-refresh-items. - Rules engine - the four
rules-engine-*tasks. - Exchange -
exchange-apply-suggestions,exchange-create-publications. - Orchestration -
task-batch-executeitself (batches can be nested) and a few internal building blocks.
Outputs
| Property | Type | Description |
|---|---|---|
An object containing the result of the task. Each output key is the key of the output defined in the outputs[].outputKey input parameter. | ||
The batch summary: total (number of entries), successCount, errorCount, taskName (the default task name) and durationMs. The counters are always computed from the internally merged results, whatever outputs requests. |
JSON inline output - outputs[].outputMode=INLINE + outputs[].mediaType=application/json
result.<outputKey>is the per-entry results array (see below).jsonis the output key used by the defaultoutputs.
JSON file output - outputs[].outputMode=ATTACHMENT + outputs[].mediaType=application/json
result.<outputKey>is a JSON file containing{ "results": [...] }with the same per-entry results.
XML file output - outputs[].outputMode=ATTACHMENT + outputs[].mediaType=application/xml
result.<outputKey>is an XML file with the same per-entry results.
Each entry of the results array has the following shape:
json
{
"key": "calc-1",
"index": 0,
"status": "COMPLETED",
"success": true,
"output": {
"result": { "value": 42 }
}
}1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
| JSON path | Description | Occurs |
|---|---|---|
[].key | The entry's key (or the generated task_<n> default, <n> being the entry's 1-based position in the input list). | 1..1 |
[].index | The 0-based position of the matching entry in the input tasks[] list. The results list is already sorted by it; use it to correlate a result with its input entry. | 1..1 |
[].status | The terminal status of the executed task (COMPLETED, COMPLETED_WITH_ERRORS...). | 1..1 |
[].success | true when the executed task completed successfully; false when it failed. | 1..1 |
[].output | The full output of the executed task (for example output.result for code-execute-javascript, or output.statusCode for protocol-http-perform). On a failed entry it may still carry the sub-workflow information needed to investigate. | 0..1 |
[].error | On a failed entry, the error details. | 0..1 |
Result ordering
The results list is always returned in input order, regardless of concurrency. Entries are distributed round-robin over the parallel slots (with concurrency: 2, slot 1 gets entries 1, 3, 5... and slot 2 gets entries 2, 4, 6...) and each slot runs its share sequentially, but the merged results are re-ordered by input position before delivery - for every requested output (INLINE array, JSON file and XML file alike):
- Each result entry carries
index, the 0-based position of the matching entry in the inputtasks[]list; the results list is sorted by it. - Default keys are
task_<n>with<n>the entry's 1-based input position, so they are globally unique whateverconcurrencyis; explicitkeys are still recommended for readability.
Error handling
- A failed entry does not stop the batch: the remaining entries still run, the batch itself ends
COMPLETED, the failed entry is reported withsuccess: false(and nooutput.result), andsummary.errorCountis incremented. If the whole batch must fail on any entry error, checksummary.errorCountafterwards (for example with aswitch+terminate). - The batch is task-agnostic: it only sees the terminal status of each executed task. For
protocol-http-perform, an HTTP error response (4xx/5xx) is still a successful task execution and counts as a success. To count HTTP errors as entry failures, make the entry itself fail on them withassertions+failOnError: truein the entry'sinputs. - Invalid inputs (missing
request, invalidconcurrency, task name not in the allowed list...) fail the batch immediately, before any entry runs, with the quoted validation message.
Limits and additional notes
concurrencyis capped at 5 parallel slots (values above are rejected) and is automatically clamped down to the number of entries.- The whole batch execution - and each parallel slot - is subject to a 7200 seconds (2 hours) timeout. Size batches so that each slot (roughly
total / concurrencysequential executions, plus per-entry orchestration overhead) fits well within it; split very large lists into several batch executions. - Only tasks of the allowed list can be batched.
- Any rate limit of the executed task still applies to each entry (for example the Product-Live public API limits); raising
concurrencydoes not raise those limits. - An empty
tasks[]list is not an error: the batch completes with a zeroedsummary(total: 0), an empty array under eachINLINEoutput key and an empty object under eachATTACHMENToutput key (no file is produced). - The
summary.successCount/summary.errorCountcounters are always computed from the internally merged per-entry results, whateveroutputsrequests - anATTACHMENT-only request reports the same counters as anINLINEone. - Batches can be nested (
task-batch-executeis itself an allowed task), but nested batches share the same timeout and concurrency constraints.
Related tasks
code-execute-javascript- a frequent batch target; also useful to build a tasks list file programmatically.json-transform-jq- build thetasks[]array from the output of a previous task.protocol-http-perform- batch HTTP calls; combine with itsassertionsto make HTTP errors count as entry failures.loop- sequential iteration inside a job, when each iteration depends on the previous one.data-job-execution-create- start other jobs instead of tasks.