Create a Pipeline

post

https://us.hevodata.com/api/v2/pipelines

Creates a Pipeline with the specified Source, Destination, and Pipeline settings.

You can create a Pipeline for any Source if you can send its credentials in the request. The credentials are of three types:

Static token: The Source authenticates with a credential that does not expire, such as a Stripe API key or a HubSpot private app token. Send it in config.
Your own OAuth app: The Source authenticates with OAuth, and you supply the credentials of an OAuth app that you own. This works for the OAuth connectors that Hevo can authorize through the API. A request that sends auth for a connector that does not authorize with OAuth, or for one that Hevo cannot authorize this way, returns a 400.
Database credentials: The Source is a database, such as MySQL or PostgreSQL. Send its host, port, user, password and database name in config.

To use your own OAuth app, send its credentials in the auth object. Refer to Authenticating SaaS Sources for more information about the credentials that each Source accepts.

You cannot create a Pipeline for a Source that requires you to sign in through a browser. Such a request returns a 400.

Replication does not start by default. The create_disabled value defaults to true, so Hevo creates the Pipeline in a disabled state. Enable it with the Enable a Pipeline endpoint once you are satisfied with it.

Hevo validates the whole request before it writes anything. If you send auth, Hevo first obtains a token. It then tests the Source and Destination connections in parallel, and waits up to five minutes for each. If a test fails, the request returns a 400 with the connector’s reason, and nothing is saved. The 200 response returns once the Pipeline is configured, before Hevo reads the Source schema. Until Hevo has read the Source schema, the object list can be empty and source_schema_status can be incomplete. Before you select objects by name or enable the Pipeline, call Get Details of a Pipeline or List All Pipeline Objects and wait until the schema is ready.

Refer to Creating a Pipeline with Hevo API for more information about the steps for creating a Pipeline.

Recent Requests

Time	Status	User Agent
Retrieving recent requests…

Loading…

Body Params

name

string

required

Unique name for your Pipeline, not exceeding 255 characters. The name must start with a letter or a digit, and can contain only letters, digits, spaces, hyphens (-) and underscores (_). Hevo removes any leading and trailing spaces.

source

object

required

Select the Source you want to configure in your Pipeline.

destination_id

int64

required

≥ 1

Unique identifier of the Destination where you want to load data. This field is required when you do not send the destination object. Send this field or the destination object, not both.

destination

object

A new Destination to create along with the Pipeline. Send this instead of destination_id. Creating a Destination this way also needs the permission to create a Destination, which Hevo checks before it validates the body. Hevo tests the new Destination in parallel with the Source and saves it only after both tests pass. It is not removed again if a later step of the Pipeline creation fails, so delete it yourself in that case.

destination_prefix

string

required

Prefix that you want Hevo to add to the Destination table names. Hevo removes any leading and trailing spaces, and checks the format against the rules of the Destination type. The prefix must also be unique among your team’s Pipelines that load into the same existing Destination.

You cannot change the prefix after the Pipeline is created.

options

object

Settings that control how the Pipeline starts. Send create_disabled and include_all_objects inside this object. Sent at the top level of the body, they are ignored and their defaults apply.

replication_type

string

enum

required

Type of data that you want Hevo to replicate from your Source objects.

The value you send must also be one that your Source connector offers. If it is not, the request returns a 400.

Call the Get Configuration Details of a Source Connector endpoint to see what the connector supports.

Show Details

HISTORICAL_AND_INCREMENTAL	Hevo first replicates all existing data from the selected Source objects and then captures only data changes.
INCREMENTAL_ONLY	Hevo replicates only new and changed data from your Source object after the Pipeline is created.

Allowed:

load_mode

string

enum

required

Mode in which you want to load data from the Source objects into the Destination tables.

This field accepts only APPEND and MERGE, and only the ones that both your Source connector and your Destination support. Any other value, including HISTORY_TRACKING and OVERWRITE, returns a 400.

HISTORY_TRACKING is an object-level load mode. Set it on an object through the Manage Objects in a Pipeline endpoint. Read History Mode for more information.

Show Details

MERGE	Existing records are updated based on matching criteria, such as primary keys, and new records are added to your Destination table if no match is found.
APPEND	New records are added at the end of your Destination table without modifying or deleting any existing data.

Allowed:

schema_evolution

string

enum

required

Policy that you want Hevo to apply when the schema of your Source changes.

Show Details

ALLOW_ALL	Hevo replicates all Source schema changes, such as new or updated columns and tables, to the Destination.
BLOCK_ALL	Hevo does not replicate any Source schema changes to the Destination.
ALLOW_COLUMN_LEVEL	Hevo replicates only new columns from the Source to the Destination. New schemas and tables are not synced.

Allowed:

schedule

object

required

Interval at which you want the Pipeline to run jobs to process data.

failure_handling_policy

object

required

Specifies how you want Hevo to handle failures for Source objects.

latency_alert

object

Configuration for the alerts that you want Hevo to send when data loading is delayed.

Responses

400

The Pipeline could not be created. This occurs when:

The Source requires you to sign in through a browser.
A credential is rejected by the Source.
A required field for the selected authentication method is missing.
The include_all_objects value is false while create_disabled is false.
The Pipeline name is already in use.
The request has no body.
A required field or block is missing, or a value fails validation.

403

You do not have the permission to create a Pipeline. Creating a Destination along with the Pipeline also needs the permission to create a Destination, which Hevo checks before it validates the body.

404

The Destination could not be found. This occurs when:

No Destination exists with the specified destination_id.
The Destination is in a different region from the base URL that you call.

500

Hevo could not complete the request. This occurs when:

A connection test is still running after five minutes.
You send auth and the OAuth provider is unavailable.

A request that the OAuth provider rate-limits returns a 429.

200Pipeline created successfully.