Learning Center

Handling Failures ​

Data Factory provides several mechanisms to manage errors that may occur during job execution. You can configure when a job should time out and how it should react to task‑level failures.

Job‑level timeout ​

You can configure a timeout at the job level to prevent long‑running executions from blocking a pipeline.

The timeoutSeconds property in job.json controls the maximum runtime of a job:

  • If the job runs longer than timeoutSeconds, it is marked as TIMEDOUT.
  • If timeoutSeconds = 0, no timeout is applied.

This setting is particularly useful to avoid blocking a pipeline when an external system is slow or unresponsive.

Example:

json
{
  "schema": "1.0",
  "key": "notify-http-example",
  "title": "Notify HTTP Example",
  "timeoutSeconds": 3600,
  "tasks": [
    {
      "name": "protocol-http-perform",
      "taskReferenceName": "protocol_http_perform",
      "description": "",
      "optional": false,
      "type": "SUB_WORKFLOW",
      "inputParameters": {
        "domain": "my-api.example.com",
        "headers": {
          "Content-Type": "application/json"
        },
        "body": {
          "message": "Hello, world!"
        }
      }
    }
  ]
}

Handling task failures ​

In some scenarios, you want a job to continue even if a particular task fails, but still react to that failure (for example, by sending a notification). To achieve this, you can:

  1. Mark the task as optional.
  2. Inspect the task status in a subsequent SWITCH task.

Example:

json
{
  "schema": "1.0",
  "key": "notify-http-example",
  "title": "Notify HTTP Example",
  "timeoutSeconds": 0,
  "tasks": [
    {
      "name": "protocol-http-perform",
      "taskReferenceName": "protocol_http_perform",
      "description": "",
      "optional": true,
      "type": "SUB_WORKFLOW",
      "inputParameters": {
        "domain": "my-api.example.com",
        "headers": {
          "Content-Type": "application/json"
        },
        "body": {
          "message": "Hello, world!"
        }
      }
    },
    {
      "name": "switch",
      "taskReferenceName": "switch",
      "description": "",
      "optional": false,
      "type": "SWITCH",
      "inputParameters": {
        "status": "protocol_http_perform.status"
      },
      "evaluatorType": "value-param",
      "expression": "status",
      "decisionCases": {
        "COMPLETED_WITH_ERRORS": [
          {
            "name": "notification-send-email",
            "taskReferenceName": "notification_send_email",
            "description": "",
            "optional": false,
            "type": "SUB_WORKFLOW",
            "inputParameters": {
              "mode": "FILE",
              "file": "file://assets/error-email.html"
            }
          }
        ]
      }
    }
  ]
}