A Pipeline replicates data from a Source to a Destination. You create and manage a Pipeline, along with its Source, with the hevo_pipeline resource. Hevo creates the Source as part of the Pipeline, so you specify the Source settings in the Pipeline resource. The Destination can be a hevo_destination resource in your configuration, or an existing Destination that you read with the hevo_destination data source.
Two other resources complete a Pipeline:
-
hevo_pipeline_objectsselects the objects that the Pipeline replicates. -
hevo_pipeline_stateenables or disables the Pipeline.
Prerequisites
-
An active Hevo account. You can start with a 14-day full-feature, free trial account.
-
You are assigned the required role for the actions that you want to perform.
-
You have generated your API credentials. Read Authentication for the steps to generate them.
-
The machine or CI/CD runner that runs Terraform uses Terraform version 1.5 or higher.
-
You have configured the Hevo provider. Read Configuring the Hevo Terraform Provider for the steps.
-
An existing Destination, or a
hevo_destinationresource in your configuration. Read Managing Destinations with Terraform for the steps to create one. -
You have the connection details and credentials of your Source.
Creating a Pipeline
To create a Pipeline, specify its Source, its Destination, and its settings:
resource "hevo_pipeline" "mysql" {
name = "MySQL to Redshift"
source_name = "MySQL Production"
source_connector_id = "SOURCE_CONNECTOR_MYSQL"
source_type = "AWS_RDS_MYSQL"
source_config = {
database_host = var.mysql_host
database_port = 3306
database_user = var.mysql_user
database_password = var.mysql_password
database_names = "employees"
}
destination_id = hevo_destination.redshift.id
destination_prefix = "mysql"
replication_type = "HISTORICAL_AND_INCREMENTAL"
load_mode = "MERGE"
schema_evolution = "BLOCK_ALL"
schedule = {
sync_type = "SCHEDULED"
frequency_minutes = 60
}
failure_handling_policy = {
object_failure_level = "PARTIAL"
object_failure_threshold = 5
}
latency_alert = {
enabled = true
threshold_minutes = 120
}
}The keys in source_config are the names of the Source connector's settings. To find them, read the connector with the hevo_connector_metadata data source. The documentation of each Source explains how to set up your Source and obtain the value of each setting. Provide credentials, such as database_password, through sensitive variables.
Set source_type to the type of your Source, such as AWS_RDS_MYSQL for an Amazon RDS MySQL database. The hevo_connector_metadata data source lists the Source types of a connector in source_types. If you do not specify source_type, Hevo creates the Source with the default type of the connector, such as MYSQL for the SOURCE_CONNECTOR_MYSQL connector.
Every Pipeline is created in the disabled state, so that you can select its objects before it starts to replicate data. After you select the objects, enable the Pipeline with the hevo_pipeline_state resource.
Authorizing a SaaS Source
Terraform can create a Pipeline for a SaaS Source, such as HubSpot, only if it can send the credentials of the Source. Use one of the following:
-
A static credential, such as an API key or a private app token, in
source_config. The hevo_connector_metadata data source lists the settings of the connector, including its credentials. -
The credentials of your own OAuth app, and a token that the app was issued, in
source_auth. Read Your Own OAuth App to obtain them.
Terraform cannot authorize a Source through Hevo's OAuth app, because that requires you to sign in through a browser. If you create a Pipeline for an OAuth Source without source_auth, the terraform apply command fails with an error that the Source configuration is incomplete.
To manage such a Pipeline with Terraform, create it in the Hevo UI, and then import it into Terraform. The imported Pipeline does not need source_config, so the Source keeps the authorization that you granted in the Hevo UI.
Pipeline Settings
The following table lists the settings that control how a Pipeline replicates data, such as what it loads, when it runs, and how it handles failures. Specify them as arguments of the hevo_pipeline resource:
| Setting | Description |
|---|---|
replication_type | The data that the Pipeline replicates. Set it to one of the following:
|
load_mode | The load mode of the Pipeline, which determines how Hevo writes data into the Destination. Set it to MERGE, APPEND, or HISTORY_TRACKING. You can set a different load mode for an object. HISTORY_TRACKING is available only for some Sources and objects. Refer to the respective Source documentation to check whether the Source supports History mode. If the Source does not support the load mode that you set, the terraform apply command fails with an error that the load mode is not valid. |
schema_evolution | The new objects and fields in the Source that Hevo replicates. Set it to one of the following:
|
schedule | When the Pipeline ingests data. Set sync_type to SCHEDULED and specify the interval in frequency_minutes. Alternatively, set sync_type to ON_DEMAND. Hevo ignores frequency_minutes for an ON_DEMAND Pipeline. An ON_DEMAND Pipeline does not run on a schedule, and enabling it does not start a load. To ingest data, trigger a sync. |
failure_handling_policy | The action that Hevo takes when objects fail to load. Specify the following:
|
latency_alert | An alert that Hevo sends when replication falls behind. Set enabled, and specify the latency in threshold_minutes. The threshold must be from 5 minutes to 15 days, and at least twice the sync frequency. If you remove latency_alert from your configuration, the alert in Hevo does not change. |
The terraform plan command checks these settings, and fails with an error that names the setting in the following cases:
-
object_failure_levelisPARTIAL, andobject_failure_thresholdis not specified. -
latency_alert.threshold_minutesis less than 5 minutes, more than 15 days, or less than twice the sync frequency.
Changing a Pipeline
To change a Pipeline, edit its configuration and run the terraform apply command. Each argument falls into one of the following groups:
-
Changes made in place: Hevo applies the change to the existing Pipeline.
-
Changes that replace the Pipeline: Terraform deletes the Pipeline and creates a new Pipeline.
Changes Made in Place
Hevo applies a change to name, source_name, load_mode, schema_evolution, schedule, failure_handling_policy.object_failure_level, or latency_alert to the existing Pipeline.
When you change source_config, the provider sends the changed settings, and Hevo tests the connection to the Source with the new settings. If the test fails, for example, because of an incorrect password, the terraform apply command fails with the reason, and the Pipeline keeps its previous settings and continues to replicate data.
Hevo does not return the settings of a Source, so the provider cannot detect a change made to source_config outside Terraform, and keeps the value from your configuration.
Note:
-
After you change
load_mode, Hevo resets the objects of the Pipeline, and they show the statusRESETTING. -
To change
schedulefromSCHEDULEDtoON_DEMAND, changesync_type. -
If you change
object_failure_leveltoPARTIAL, you must also specifyobject_failure_threshold. Check the plan before you apply the change, because a change toobject_failure_thresholdreplaces the Pipeline.
Changes That Replace the Pipeline
A change to source_connector_id, source_type, destination_id, destination_prefix, replication_type, or failure_handling_policy.object_failure_threshold replaces the Pipeline. Terraform deletes the Pipeline and its Source, and creates a new Pipeline and Source. The terraform plan command marks the change as a replacement with # forces replacement and displays a warning that the Pipeline will be replaced.
Terraform also replaces the
hevo_pipeline_objectsandhevo_pipeline_stateresources of the Pipeline, so the same apply selects the objects of the new Pipeline and enables it. If itsreplication_typeisHISTORICAL_AND_INCREMENTAL, the new Pipeline then loads all the historical data of its objects again. The tables of the old Pipeline remain in your Destination. Check the plan for# forces replacementbefore you apply a change.
Enabling and Disabling a Pipeline
The hevo_pipeline_state resource sets whether a Pipeline replicates data. Set paused to false to enable the Pipeline, or to true to disable it:
resource "hevo_pipeline_state" "mysql" {
pipeline_id = hevo_pipeline.mysql.id
paused = false
depends_on = [hevo_pipeline_objects.mysql]
}-
paused = false: Enables the Pipeline. A Pipeline can be enabled only if at least one of its objects is selected. Otherwise, theterraform applycommand fails with an error that no object of the Pipeline is selected. -
paused = true: Disables the Pipeline. Hevo stops replicating data, and lets the jobs that are already running finish.
Terraform selects the objects before it enables or disables the Pipeline only if you add depends_on to the hevo_pipeline_state resource. Without it, Terraform can run both changes at the same time, and the terraform apply command can fail, as Hevo does not accept a change to the objects while it disables the Pipeline.
If you remove the hevo_pipeline_state resource from your configuration, the Pipeline stays in its current state, enabled or disabled.
Handling Long-Running Changes
Hevo applies most changes to a Pipeline after it accepts the request. The provider waits until each change is applied before it reports the resource as created or updated. Creating a Pipeline can take about 30 seconds, and enabling it can take a few minutes. Any change can take a few minutes if Hevo is still applying an earlier change to the Pipeline.
While Hevo applies one change, it does not accept another change to the same Pipeline. The provider waits and sends the change again until Hevo accepts it.
The default limit for this wait is 20 minutes for a change to a hevo_pipeline, hevo_pipeline_objects, or hevo_pipeline_state resource. To change this limit, add a timeouts block:
resource "hevo_pipeline" "mysql" {
# ...
timeouts = {
create = "30m"
update = "30m"
delete = "30m"
}
}If the limit is reached, Terraform reports an error that the change did not finish. Hevo can still complete the change after the error, because it applies changes independently of Terraform. Run the terraform plan command to see the current state of the Pipeline before you apply again.
The request to create a Pipeline can fail when the limit is reached, or because of a network or server error. Hevo can still create the Pipeline after the error, so the provider looks for a Pipeline with the same name. If it finds the Pipeline, Terraform manages it, and displays a warning with the ID of the Pipeline.
Deleting a Pipeline
To delete a Pipeline, remove it from your configuration and run the terraform apply command, or run the terraform destroy command. Hevo deletes the Pipeline and its Source. The tables in your Destination are not deleted.
If Terraform also manages the objects and the state of the Pipeline, Terraform removes those resources first, and then deletes the Pipeline.
Arguments and Attributes
Arguments are the settings of the Pipeline that you specify in your configuration. Attributes are the values that Hevo assigns when it creates the Pipeline, such as its ID. You can refer to an attribute elsewhere in your configuration, for example, hevo_pipeline.mysql.id to set the Pipeline of a hevo_pipeline_objects resource.
The Arguments and Attributes sections describe the hevo_pipeline resource. The hevo_pipeline_state section describes the resource that enables or disables the Pipeline.
Arguments
The following table lists the arguments of the hevo_pipeline resource:
| Argument | Description |
|---|---|
name (String, Required) | The name of the Pipeline. |
source_name (String, Required) | The name of the Source of the Pipeline. |
source_connector_id (String, Required) | The ID of the Source connector, such as SOURCE_CONNECTOR_MYSQL. Changing it replaces the Pipeline. |
source_type (String, Optional) | The type of the Source, such as AWS_RDS_MYSQL. The source_types attribute of the hevo_connector_metadata data source lists the types of a connector. Default value: The default type of the connector. Changing it replaces the Pipeline. |
source_config (Dynamic, Optional) | The configuration settings of the Source. Each key is the name of a setting of the connector. Required when you create a Pipeline, unless the Source authenticates only with source_auth. |
source_auth (Dynamic, Optional, Sensitive) | The credentials of your own OAuth app, for a Source that authenticates with OAuth. |
destination_id (String, Required) | The ID of the Destination that the Pipeline loads data into. Changing it replaces the Pipeline. |
destination_prefix (String, Required) | The prefix that Hevo adds to the names of the tables that it creates in the Destination. Changing it replaces the Pipeline. |
replication_type (String, Required) | The data that the Pipeline replicates. Set it to HISTORICAL_AND_INCREMENTAL or INCREMENTAL_ONLY, as described in Pipeline Settings. Changing it replaces the Pipeline. |
load_mode (String, Required) | The load mode of the Pipeline. Set it to MERGE, APPEND, or HISTORY_TRACKING, as described in Pipeline Settings. |
schema_evolution (String, Required) | The new objects and fields in the Source that Hevo replicates. Set it to ALLOW_ALL, ALLOW_COLUMN_LEVEL, or BLOCK_ALL, as described in Pipeline Settings. |
schedule (Object, Required) | The schedule on which the Pipeline ingests data. Specify the following:
|
failure_handling_policy (Object, Required) | The action that Hevo takes when objects fail to load. Specify the following:
|
latency_alert (Object, Optional) | The alert that Hevo sends when replication falls behind. Specify the following:
|
timeouts (Object, Optional) | The time that the provider waits for a create, an update, or a delete to finish. Default value: 20 minutes each. Read Handling Long-Running Changes to change it. |
Attributes
The following table lists the attributes of the hevo_pipeline resource:
| Attribute | Description |
|---|---|
id (String) | The ID of the Pipeline. |
created_at (String) | The time when the Pipeline was created, in the Request for Comments (RFC) 3339 format. |
created_by (String) | The user who created the Pipeline. |
The state, health, and last change of a Pipeline change as it runs, so the resource does not include them. To read them, use the hevo_pipeline data source.
hevo_pipeline_state
The following table lists the arguments and attributes of the hevo_pipeline_state resource:
| Argument or attribute | Description |
|---|---|
pipeline_id (String, Required) | The ID of the Pipeline. |
paused (Boolean, Required) | false enables the Pipeline. true disables it. |
timeouts (Object, Optional) | The time that the provider waits for a create or update. Default value: 20 minutes each. |
id (String, Attribute) | The ID of the Pipeline. |