Learning Center

Data / Await job executions ​

Task in alpha phase

This Data Factory task is in phase. You can contact the Product-Live team at contact@product-live.com if you want more details and get an early access.

This task waits until one or more job executions reach a terminal status, then returns them — including their output, where a job exposes its results (e.g. produced files).

It is the standalone counterpart of the wait option of the Create job executions task: use it to wait for job executions created earlier in the job, by another task, or through the API.

Task name:

Examples ​

json
{
    "name": "data-job-execution-await",
    "taskReferenceName": "await_job_executions",
    "description": "Wait for the job executions to complete",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "jobExecutionIds": ["6a74a85718d742f7721bb267"],
        "pollingIntervalSeconds": 15,
        "timeoutSeconds": 1800
    }
}
json
{
    "name": "data-job-execution-await",
    "taskReferenceName": "await_job_executions",
    "description": "Wait for the job executions to complete",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "jobExecutionIds": ["6a74a85718d742f7721bb267"],
        "outputs": [
            {
                "mediaType": "application/json",
                "outputMode": "ATTACHMENT",
                "outputKey": "json"
            }
        ]
    }
}

Definition ​

PropertyTypeRequiredDescription
The name (or type) of the task
The unique name of the task in your job. It is used to reference this task in the workflow.
The functional description of this task in your job.
The type of the task. It must be SUB_WORKFLOW.
:

true: the job continues if there is an error on this task.
false: the job fails.
Input parameters of the task. See below

Inputs Parameters ​

PropertyTypeRequiredDescription
The Product-Live API key.
If not provided, an automatically generated API key will be used to perform the request on behalf of the current user. If provided, the given API key will be used to perform the request.
A job execution is only visible to the account that created it: to await an execution created with another account's API key, provide the same key here.
The ids of the job executions to wait for (a single id string is also accepted). Maximum: 100 ids, duplicates are ignored.
Default: 15, minimum: 5. Delay between two status checks.
Default: 3600, maximum: 86400. Overall wait budget: the task fails when it is exceeded. timeoutSeconds / pollingIntervalSeconds must not exceed 720.
Default: true. If true, the task fails when an awaited job execution ends on a terminal status other than COMPLETED. If false, the task completes and the returned executions carry their terminal status.
The outputs of the request.
Default: [{"mediaType": "application/json", "outputMode": "INLINE", "outputKey": "json"}].
The media type of the output. Today, application/json and application/xml are supported.
INLINE: The output is provided inline in the output object. ATTACHMENT: The output is provided as an attachment in the output object. INLINE is only supported for mediaType=application/json
The key of the output.
Either json or xml

Outputs ​

PropertyTypeDescription
The number of job executions returned.
An object containing the terminal job executions. Each output key is the key defined in the outputs[].outputKey input parameter.
A map of jobExecutionId to terminal status.
A human readable summary.

Json output - outputs[].outputMode=INLINE and outputs[].mediaType=application/json ​

  • result.json is an array of objects as defined in the job executions API, in their terminal state: the output property carries what each job exposed, e.g. produced files as { "url": "..." } objects.
  • json is the default output key but may be changed by setting the outputKey property.

Failure behavior ​

The task fails (with the last observed statuses and an explanatory message in its output) when:

  • an awaited job execution ends on a terminal status other than COMPLETED and failOnError is true;
  • timeoutSeconds is exceeded before every job execution reaches a terminal status;
  • some ids are still unknown after 3 consecutive checks (wrong id, execution created by another account than the API key's one, or deleted execution).

A transient error during a status check is not a failure: the check is retried at the next polling interval.

Limits and additional notes ​

  • Terminal statuses are COMPLETED, COMPLETED_WITH_ERRORS, FAILED, REMOVED, TERMINATED, TIMED_OUT and USER_ERROR.
  • A job produces files by exposing them in its own output: map the produced file in the job's outputParameters (or, if the job ends with a TERMINATE task, in that task's outputParameters, which win). There is no implicit "files of an execution" listing.
  • A pipeline runs a single job at a time: a job waiting for an execution scheduled on its own pipeline blocks it until the wait times out.

Additional examples ​

Create a job execution and await it in two steps ​

job.json

json
{
    "name": "data-job-execution-create",
    "taskReferenceName": "create_job_execution",
    "description": "Create a job execution",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "request": {
            "method": "INLINE",
            "contentType": "application/json",
            "model": "default",
            "json": {
                "operations": [
                    {
                        "key": "a",
                        "element": {
                            "jobId": "65819ba43ad5b963658ab35d",
                            "info": {
                                "title": "Export triggered by the parent job"
                            },
                            "input": {}
                        }
                    }
                ]
            }
        }
    }
}
json
{
    "name": "data-job-execution-await",
    "taskReferenceName": "await_job_execution",
    "description": "Wait for the created job execution",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "jobExecutionIds": ["${create_job_execution.output.result.json[0].response.id}"],
        "timeoutSeconds": 1800
    }
}

Await a job execution of another account ​

Provide the API key of the account that created the job execution (typically stored in a secret variable).

json
{
    "name": "data-job-execution-await",
    "taskReferenceName": "await_partner_job_execution",
    "description": "Wait for the partner job execution",
    "type": "SUB_WORKFLOW",
    "optional": false,
    "inputParameters": {
        "apiKey": "${workflow.globalVariables.partner_api_key}",
        "jobExecutionIds": ["${create_job_execution.output.result.json[0].response.id}"],
        "timeoutSeconds": 1800
    }
}