Managing Pipelines with Terraform

A Pipeline replicates data from a Source to a Destination. You create and manage a Pipeline, along with its Source, with the hevo_pipeline resource. Hevo creates the Source as part of the Pipeline, so you specify the Source settings in the Pipeline resource. The Destination can be a hevo_destination resource in your configuration, or an existing Destination that you read with the hevo_destination data source.

Two other resources complete a Pipeline:

Prerequisites

Creating a Pipeline

To create a Pipeline, specify its Source, its Destination, and its settings:

resource "hevo_pipeline" "mysql" {
  name                = "MySQL to Redshift"
  source_name         = "MySQL Production"
  source_connector_id = "SOURCE_CONNECTOR_MYSQL"
  source_type         = "AWS_RDS_MYSQL"

  source_config = {
    database_host     = var.mysql_host
    database_port     = 3306
    database_user     = var.mysql_user
    database_password = var.mysql_password
    database_names    = "employees"
  }

  destination_id     = hevo_destination.redshift.id
  destination_prefix = "mysql"
  replication_type   = "HISTORICAL_AND_INCREMENTAL"
  load_mode          = "MERGE"
  schema_evolution   = "BLOCK_ALL"

  schedule = {
    sync_type         = "SCHEDULED"
    frequency_minutes = 60
  }

  failure_handling_policy = {
    object_failure_level     = "PARTIAL"
    object_failure_threshold = 5
  }

  latency_alert = {
    enabled           = true
    threshold_minutes = 120
  }
}

The keys in source_config are the names of the Source connector's settings. To find them, read the connector with the hevo_connector_metadata data source. The documentation of each Source explains how to set up your Source and obtain the value of each setting. Provide credentials, such as database_password, through sensitive variables.

Set source_type to the type of your Source, such as AWS_RDS_MYSQL for an Amazon RDS MySQL database. The hevo_connector_metadata data source lists the Source types of a connector in source_types. If you do not specify source_type, Hevo creates the Source with the default type of the connector, such as MYSQL for the SOURCE_CONNECTOR_MYSQL connector.

Every Pipeline is created in the disabled state, so that you can select its objects before it starts to replicate data. After you select the objects, enable the Pipeline with the hevo_pipeline_state resource.

Authorizing a SaaS Source

Terraform can create a Pipeline for a SaaS Source, such as HubSpot, only if it can send the credentials of the Source. Use one of the following:

  • A static credential, such as an API key or a private app token, in source_config. The hevo_connector_metadata data source lists the settings of the connector, including its credentials.

  • The credentials of your own OAuth app, and a token that the app was issued, in source_auth. Read Your Own OAuth App to obtain them.

Terraform cannot authorize a Source through Hevo's OAuth app, because that requires you to sign in through a browser. If you create a Pipeline for an OAuth Source without source_auth, the terraform apply command fails with an error that the Source configuration is incomplete.

To manage such a Pipeline with Terraform, create it in the Hevo UI, and then import it into Terraform. The imported Pipeline does not need source_config, so the Source keeps the authorization that you granted in the Hevo UI.

Pipeline Settings

The following table lists the settings that control how a Pipeline replicates data, such as what it loads, when it runs, and how it handles failures. Specify them as arguments of the hevo_pipeline resource:

SettingDescription
replication_typeThe data that the Pipeline replicates. Set it to one of the following:
  • HISTORICAL_AND_INCREMENTAL: Loads the existing data in the Source, and then the changes to it.
  • INCREMENTAL_ONLY: Replicates only the data added or changed after the Pipeline is created. To replicate the existing data at a later time, resync the Pipeline.
load_modeThe load mode of the Pipeline, which determines how Hevo writes data into the Destination. Set it to MERGE, APPEND, or HISTORY_TRACKING. You can set a different load mode for an object. HISTORY_TRACKING is available only for some Sources and objects. Refer to the respective Source documentation to check whether the Source supports History mode. If the Source does not support the load mode that you set, the terraform apply command fails with an error that the load mode is not valid.
schema_evolutionThe new objects and fields in the Source that Hevo replicates. Set it to one of the following:
  • ALLOW_ALL: Replicates new objects and new fields as they appear.
  • ALLOW_COLUMN_LEVEL: Replicates new fields of the existing objects, but not new objects.
  • BLOCK_ALL: Replicates neither new objects nor new fields.
scheduleWhen the Pipeline ingests data. Set sync_type to SCHEDULED and specify the interval in frequency_minutes. Alternatively, set sync_type to ON_DEMAND. Hevo ignores frequency_minutes for an ON_DEMAND Pipeline. An ON_DEMAND Pipeline does not run on a schedule, and enabling it does not start a load. To ingest data, trigger a sync.
failure_handling_policyThe action that Hevo takes when objects fail to load. Specify the following:
  • object_failure_level: PARTIAL or ALL_OR_NONE.
  • object_failure_threshold: The number of objects that can fail, as a number greater than 0. Required when object_failure_level is PARTIAL. Changing it replaces the Pipeline.
latency_alertAn alert that Hevo sends when replication falls behind. Set enabled, and specify the latency in threshold_minutes. The threshold must be from 5 minutes to 15 days, and at least twice the sync frequency. If you remove latency_alert from your configuration, the alert in Hevo does not change.

The terraform plan command checks these settings, and fails with an error that names the setting in the following cases:

  • object_failure_level is PARTIAL, and object_failure_threshold is not specified.

  • latency_alert.threshold_minutes is less than 5 minutes, more than 15 days, or less than twice the sync frequency.

Changing a Pipeline

To change a Pipeline, edit its configuration and run the terraform apply command. Each argument falls into one of the following groups:

Changes Made in Place

Hevo applies a change to name, source_name, load_mode, schema_evolution, schedule, failure_handling_policy.object_failure_level, or latency_alert to the existing Pipeline.

When you change source_config, the provider sends the changed settings, and Hevo tests the connection to the Source with the new settings. If the test fails, for example, because of an incorrect password, the terraform apply command fails with the reason, and the Pipeline keeps its previous settings and continues to replicate data.

Hevo does not return the settings of a Source, so the provider cannot detect a change made to source_config outside Terraform, and keeps the value from your configuration.

Note:

  • After you change load_mode, Hevo resets the objects of the Pipeline, and they show the status RESETTING.

  • To change schedule from SCHEDULED to ON_DEMAND, change sync_type.

  • If you change object_failure_level to PARTIAL, you must also specify object_failure_threshold. Check the plan before you apply the change, because a change to object_failure_threshold replaces the Pipeline.

Changes That Replace the Pipeline

A change to source_connector_id, source_type, destination_id, destination_prefix, replication_type, or failure_handling_policy.object_failure_threshold replaces the Pipeline. Terraform deletes the Pipeline and its Source, and creates a new Pipeline and Source. The terraform plan command marks the change as a replacement with # forces replacement and displays a warning that the Pipeline will be replaced.

🚧

Terraform also replaces the hevo_pipeline_objects and hevo_pipeline_state resources of the Pipeline, so the same apply selects the objects of the new Pipeline and enables it. If its replication_type is HISTORICAL_AND_INCREMENTAL, the new Pipeline then loads all the historical data of its objects again. The tables of the old Pipeline remain in your Destination. Check the plan for # forces replacement before you apply a change.

Enabling and Disabling a Pipeline

The hevo_pipeline_state resource sets whether a Pipeline replicates data. Set paused to false to enable the Pipeline, or to true to disable it:

resource "hevo_pipeline_state" "mysql" {
  pipeline_id = hevo_pipeline.mysql.id
  paused      = false

  depends_on = [hevo_pipeline_objects.mysql]
}
  • paused = false: Enables the Pipeline. A Pipeline can be enabled only if at least one of its objects is selected. Otherwise, the terraform apply command fails with an error that no object of the Pipeline is selected.

  • paused = true: Disables the Pipeline. Hevo stops replicating data, and lets the jobs that are already running finish.

Terraform selects the objects before it enables or disables the Pipeline only if you add depends_on to the hevo_pipeline_state resource. Without it, Terraform can run both changes at the same time, and the terraform apply command can fail, as Hevo does not accept a change to the objects while it disables the Pipeline.

If you remove the hevo_pipeline_state resource from your configuration, the Pipeline stays in its current state, enabled or disabled.

Handling Long-Running Changes

Hevo applies most changes to a Pipeline after it accepts the request. The provider waits until each change is applied before it reports the resource as created or updated. Creating a Pipeline can take about 30 seconds, and enabling it can take a few minutes. Any change can take a few minutes if Hevo is still applying an earlier change to the Pipeline.

While Hevo applies one change, it does not accept another change to the same Pipeline. The provider waits and sends the change again until Hevo accepts it.

The default limit for this wait is 20 minutes for a change to a hevo_pipeline, hevo_pipeline_objects, or hevo_pipeline_state resource. To change this limit, add a timeouts block:

resource "hevo_pipeline" "mysql" {
  # ...
  timeouts = {
    create = "30m"
    update = "30m"
    delete = "30m"
  }
}

If the limit is reached, Terraform reports an error that the change did not finish. Hevo can still complete the change after the error, because it applies changes independently of Terraform. Run the terraform plan command to see the current state of the Pipeline before you apply again.

The request to create a Pipeline can fail when the limit is reached, or because of a network or server error. Hevo can still create the Pipeline after the error, so the provider looks for a Pipeline with the same name. If it finds the Pipeline, Terraform manages it, and displays a warning with the ID of the Pipeline.

Deleting a Pipeline

To delete a Pipeline, remove it from your configuration and run the terraform apply command, or run the terraform destroy command. Hevo deletes the Pipeline and its Source. The tables in your Destination are not deleted.

If Terraform also manages the objects and the state of the Pipeline, Terraform removes those resources first, and then deletes the Pipeline.

Arguments and Attributes

Arguments are the settings of the Pipeline that you specify in your configuration. Attributes are the values that Hevo assigns when it creates the Pipeline, such as its ID. You can refer to an attribute elsewhere in your configuration, for example, hevo_pipeline.mysql.id to set the Pipeline of a hevo_pipeline_objects resource.

The Arguments and Attributes sections describe the hevo_pipeline resource. The hevo_pipeline_state section describes the resource that enables or disables the Pipeline.

Arguments

The following table lists the arguments of the hevo_pipeline resource:

ArgumentDescription
name (String, Required)The name of the Pipeline.
source_name (String, Required)The name of the Source of the Pipeline.
source_connector_id (String, Required)The ID of the Source connector, such as SOURCE_CONNECTOR_MYSQL. Changing it replaces the Pipeline.
source_type (String, Optional)The type of the Source, such as AWS_RDS_MYSQL. The source_types attribute of the hevo_connector_metadata data source lists the types of a connector. Default value: The default type of the connector. Changing it replaces the Pipeline.
source_config (Dynamic, Optional)The configuration settings of the Source. Each key is the name of a setting of the connector. Required when you create a Pipeline, unless the Source authenticates only with source_auth.
source_auth (Dynamic, Optional, Sensitive)The credentials of your own OAuth app, for a Source that authenticates with OAuth.
destination_id (String, Required)The ID of the Destination that the Pipeline loads data into. Changing it replaces the Pipeline.
destination_prefix (String, Required)The prefix that Hevo adds to the names of the tables that it creates in the Destination. Changing it replaces the Pipeline.
replication_type (String, Required)The data that the Pipeline replicates. Set it to HISTORICAL_AND_INCREMENTAL or INCREMENTAL_ONLY, as described in Pipeline Settings. Changing it replaces the Pipeline.
load_mode (String, Required)The load mode of the Pipeline. Set it to MERGE, APPEND, or HISTORY_TRACKING, as described in Pipeline Settings.
schema_evolution (String, Required)The new objects and fields in the Source that Hevo replicates. Set it to ALLOW_ALL, ALLOW_COLUMN_LEVEL, or BLOCK_ALL, as described in Pipeline Settings.
schedule (Object, Required)The schedule on which the Pipeline ingests data. Specify the following:
  • sync_type: The type of schedule. Set it to SCHEDULED or ON_DEMAND.
  • frequency_minutes: The interval between syncs, in minutes. Required when sync_type is SCHEDULED. Hevo ignores it for an ON_DEMAND Pipeline.
failure_handling_policy (Object, Required)The action that Hevo takes when objects fail to load. Specify the following:
  • object_failure_level: The failure handling level. Set it to PARTIAL or ALL_OR_NONE.
  • object_failure_threshold: The number of objects that can fail, as a number greater than 0. Required when object_failure_level is PARTIAL. Changing it replaces the Pipeline.
latency_alert (Object, Optional)The alert that Hevo sends when replication falls behind. Specify the following:
  • enabled: The status of the alert. Set it to true to turn the alert on, or false to turn it off.
  • threshold_minutes: The latency that raises the alert, in minutes. It must be from 5 minutes to 15 days, and at least twice the sync frequency.
timeouts (Object, Optional)The time that the provider waits for a create, an update, or a delete to finish. Default value: 20 minutes each. Read Handling Long-Running Changes to change it.

Attributes

The following table lists the attributes of the hevo_pipeline resource:

AttributeDescription
id (String)The ID of the Pipeline.
created_at (String)The time when the Pipeline was created, in the Request for Comments (RFC) 3339 format.
created_by (String)The user who created the Pipeline.

The state, health, and last change of a Pipeline change as it runs, so the resource does not include them. To read them, use the hevo_pipeline data source.

hevo_pipeline_state

The following table lists the arguments and attributes of the hevo_pipeline_state resource:

Argument or attributeDescription
pipeline_id (String, Required)The ID of the Pipeline.
paused (Boolean, Required)false enables the Pipeline. true disables it.
timeouts (Object, Optional)The time that the provider waits for a create or update. Default value: 20 minutes each.
id (String, Attribute)The ID of the Pipeline.