Pipelines

A pipeline is an execution stack in which jobs run. It isolates job execution and controls scheduling within your account.

Rules

Rule	Description
Single execution	Only one job can run on a pipeline at a time.
FIFO scheduling	Jobs are scheduled First In, First Out: the first job submitted is the first executed.
Cooperative scheduling	No preemption. A running job must complete before the next one can start, no matter how long it takes.
Cancellation	Cancelling a running job releases the pipeline immediately. The next queued job starts.

Association with jobs

A job must be associated with one and only one pipeline. A job that is not associated with a pipeline cannot be executed.

Waiting queue

When a job is launched on a busy pipeline, it is placed in a waiting queue. Jobs in the queue are executed in submission order as the pipeline becomes available.

flowchart LR
    subgraph pipeline ["Pipeline"]
        direction TB
        running["Job A — Running"]
        w1["Job B — Waiting (1st)"]
        w2["Job C — Waiting (2nd)"]
        w3["Job D — Waiting (3rd)"]
        running ~~~ w1 ~~~ w2 ~~~ w3
    end

When Job A completes (or is cancelled), Job B starts automatically, then Job C, and so on.

Job execution lifecycle

When a job is launched, the pipeline determines whether to execute it immediately or queue it:

flowchart TD
    launch((Job is launched))
    available{Is the pipeline<br/>available?}
    execute[Execute job immediately]
    queue[Place job in the<br/>waiting queue]
    done(End)
    launch --> available
    available -- Yes --> execute
    available -- No --> queue
    execute --> done
    queue --> done
    classDef green fill:#9f6,stroke:#333,stroke-width:2px;
    class launch green

When a job execution ends, the pipeline processes the next queued job:

flowchart TD
    finished((Job execution ends))
    empty{Is the waiting<br/>queue empty?}
    select[Select the oldest<br/>job in the queue]
    start[Launch the selected job]
    done(End)
    finished --> empty
    empty -- Yes --> done
    empty -- No --> select
    select --> start
    start --> done
    classDef green fill:#9f6,stroke:#333,stroke-width:2px;
    class finished green

For more details on job execution statuses (PENDING, RUNNING, COMPLETED, etc.), see Job executions.

Setting up pipelines

Pipelines are configured by our Professional Services team. Contact your Product-Live representative to set up or add pipelines on your account.

TIP

If you need more than one simultaneous execution on a pipeline, please contact our support team.

Best practices

For guidance on when and how to use multiple pipelines (isolating critical jobs, separating workloads), see the Data Factory Pipelines guide.