Data Factory Task API
Introduction
In addition to the Data Factory Task defined by the Product-Live platform, you may defined your Data Factory custom Task. A sample implementation using the NestJs framework and Typescript language is available here (Product-Live/data-factory-task-example).
Custom tasks are not executed by the Data Fatory plateform, they are executed inside your own infrastructure. In their execution they need to poll Data Factory to know when there is data to process, and then update its execution to Data Factory with the generated output.
The Data Factory Tasks API Swagger definition is available on our Product-Live API portal here.
General information
- Rate limit: 50 requests per minute
Create a Custom Task
Request URL
Path parameters
None
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| Content-Type | string | application/json |
| X-Api-Key | string | Your account API key |
Request Body
/!\ A mettre à jour avec un exemple concret et illustratif /!\
json
{
"key": "test",
"description": "test",
"retryCount": 0,
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Request Example
bash
curl -X 'POST' \
'https://api.product-live.com/v1/data_factory/tasks' \
-H 'accept: */*' \
-H 'X-Api-Key: <REDACTED>' \
-H 'Content-Type: application/json' \
-d '{
"key": "test",
"description": "test",
"retryCount": 0,
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}'1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
ts
import { createConfiguration, ServerConfiguration, TaskApi, VariableApi } from '@product-live/api-sdk';
export async function main(): Promise<void> {
const configuration = createConfiguration({
baseServer: new ServerConfiguration(process.env.API_BASE_PATH || '', {}),
authMethods: {
ApiKeyAuthHeader: process.env.API_ACCESS_TOKEN
}
});
const taskApi = new TaskApi(configuration);
const tasks = await taskApi.getTasks();
}1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
Response Example
json
{
"object": "task",
"id": "6584633de0d9331b1fe07784",
"createdAt": "2023-12-21T16:09:33.057Z",
"retryCount": 0,
"key": "test",
"description": "test",
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Get a Custom Task by Id
Request URL
Path parameters
| Parameter | Type | Description |
|---|---|---|
| taskId | string | A Task id |
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
None
Request Example
bash
curl -X 'GET' \
'https://api.product-live.com/v1/data_factory/tasks/6584633de0d9331b1fe07784' \
-H 'accept: application/json' \
-H 'X-Api-Key: <REDACTED>'1
2
3
4
2
3
4
ts
import { createConfiguration, ServerConfiguration, TaskApi } from '@product-live/api-sdk';
export async function main(): Promise<void> {
const configuration = createConfiguration({
baseServer: new ServerConfiguration(process.env.API_BASE_PATH || '', {}),
authMethods: {
ApiKeyAuthHeader: process.env.API_ACCESS_TOKEN
}
});
const taskApi = new TaskApi(configuration);
const task = await taskApi.getTaskById('task id');
}1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
Response Example
json
{
"object": "task",
"id": "6584633de0d9331b1fe07784",
"createdAt": "2023-12-21T16:09:33.057Z",
"retryCount": 0,
"key": "test",
"description": "test",
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
List Custom Tasks
Request URL
Path parameters
None
Request parameters
| Parameter | Type | Description |
|---|---|---|
| page | integer | Optional The page number to retrieve. Default value if missing : 0 |
| size | integer | Optional The number of items per page. Default value if missing : 10 |
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
None
Request Example
bash
curl -X 'GET' \
'https://api.product-live.com/v1/data_factory/tasks?size=10&page=0' \
-H 'accept: */*' \
-H 'X-Api-Key: <REDACTED>'1
2
3
4
2
3
4
ts
import { createConfiguration, ServerConfiguration, TaskApi, VariableApi } from '@product-live/api-sdk';
export async function main(): Promise<void> {
const configuration = createConfiguration({
baseServer: new ServerConfiguration(process.env.API_BASE_PATH || '', {}),
authMethods: {
ApiKeyAuthHeader: process.env.API_ACCESS_TOKEN
}
});
const taskApi = new TaskApi(configuration);
const tasks = await taskApi.getTasks();
}1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
Response Example
json
{
"object": "list",
"data": [
{
"object": "task",
"id": "6584633de0d9331b1fe07784",
"createdAt": "2023-12-21T16:09:33.057Z",
"retryCount": 0,
"key": "test",
"description": "test",
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}
],
"totalElements": 1
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Update a Custom Task
Request URL
Path parameters
| Parameter | Type | Description |
|---|---|---|
| taskId | string | A Task id |
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
json
{
"key": "test",
"description": "test 2",
"retryCount": 0,
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Request Example
bash
curl -X 'PUT' \
'https://api.product-live.com/v1/data_factory/tasks/6584633de0d9331b1fe07784' \
-H 'accept: application/json' \
-H 'X-Api-Key: <REDACTED>' \
-H 'Content-Type: application/json' \
-d '{
"key": "test",
"description": "test 2",
"retryCount": 0,
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}'1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
ts
import { createConfiguration, ServerConfiguration, TaskApi } from '@product-live/api-sdk';
export async function main(): Promise<void> {
const configuration = createConfiguration({
baseServer: new ServerConfiguration(process.env.API_BASE_PATH || '', {}),
authMethods: {
ApiKeyAuthHeader: process.env.API_ACCESS_TOKEN
}
});
const taskApi = new TaskApi(configuration);
const task = await taskApi.updateTask('task id',{
key: '<REDACTED>',
description: '<REDACTED>',
retryCount: 0,
inputKeys: [],
outputKeys: [],
id: '<REDACTED>'
});
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Response Example
json
{
"object": "task",
"id": "6584633de0d9331b1fe07784",
"createdAt": "2023-12-21T16:09:33.057Z",
"updatedAt": "2023-12-22T10:21:17.330Z",
"retryCount": 0,
"key": "test",
"description": "test 2",
"inputKeys": [
{
"key": "string",
"description": "string"
}
],
"outputKeys": [
{
"key": "string",
"description": "string"
}
]
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Delete a Custom Task
Request URL
Path parameters
| Parameter | Type | Description |
|---|---|---|
| taskId | string | A Task id |
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
None
Request Example
bash
curl -X 'DELETE' \
'https://api.product-live.com/v1/data_factory/tasks/6584633de0d9331b1fe07784' \
-H 'accept: application/json' \
-H 'X-Api-Key: [REDACTED]'1
2
3
4
2
3
4
ts
import { createConfiguration, ServerConfiguration, TaskApi } from '@product-live/api-sdk';
export async function main(): Promise<void> {
const configuration = createConfiguration({
baseServer: new ServerConfiguration(process.env.API_BASE_PATH || '', {}),
authMethods: {
ApiKeyAuthHeader: process.env.API_ACCESS_TOKEN
}
});
const taskApi = new TaskApi(configuration);
const task = await taskApi.deleteTask('task id');
}1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
Response Example
json
1
json
{
"date": "Thu Dec 21 2023 16:33:19 GMT+0000 (Coordinated Universal Time)",
"code": "NOT_FOUND",
"message": "nothing found",
"httpStatusCode": 404
}1
2
3
4
5
6
2
3
4
5
6
Poll Data Factory to start Custom Task execution
TIP
It is up to your custom Task to regularly call Data Factory to check if there is an execution pending for that custom Task. When there is processing to do, the Task execution id is provided in response to the call.
Request URL
Path parameters
| Parameter | Type | Description |
|---|---|---|
| taskId | string | A Task id |
Request parameters
None
Request Headers
| Header | Type | Description |
|---|---|---|
| accept | string | */* |
| X-Api-Key | string | Your account API key |
Request Body
None
Request Example
bash
curl -X 'GET' \
'https://api.product-live.com/v1/data_factory/tasks/6584633de0d9331b1fe07784/poll' \
-H 'accept: */*' \
-H 'X-Api-Key: <REDACTED>'1
2
3
4
2
3
4
ts
import { createConfiguration, ServerConfiguration, TaskApi } from '@product-live/api-sdk';
export async function main(): Promise<void> {
const configuration = createConfiguration({
baseServer: new ServerConfiguration(process.env.API_BASE_PATH || '', {}),
authMethods: {
ApiKeyAuthHeader: process.env.API_ACCESS_TOKEN
}
});
const taskApi = new TaskApi(configuration);
const taskExecution = await taskApi.pollTaskExecution('task id');
}1
2
3
4
5
6
7
8
9
10
11
12
13
2
3
4
5
6
7
8
9
10
11
12
13
Response Example
json
{
"date": "Thu Jul 01 2020 12:00:00 GMT+0000 (Coordinated Universal Time)",
"code": "NOT_FOUND",
"message": "Task not found",
"httpStatusCode": 404
}1
2
3
4
5
6
2
3
4
5
6
json
{
"object": "task_execution",
"id": "658468663ad5b9123d8ab732",
"taskId": "6584633de0d9331b1fe07784",
"createdAt": "2020-01-01T12:00:00.000Z",
"updatedAt": "2020-01-01T12:00:00.000Z",
"status": "IN_PROGRESS",
"input": {
"string": "string"
},
"output": {
"string": "string"
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
2
3
4
5
6
7
8
9
10
11
12
13
14