Managing Existing Destinations and Pipelines with Terraform

Importing lets Terraform manage the Destinations and Pipelines that you created in the Hevo UI or with the Hevo API, without creating them again. To import a resource, you write its configuration, and Terraform links it to the existing Destination or Pipeline in your Hevo account. After that, Terraform manages the resource as if it had created it.

Prerequisites

Perform the following steps to import a Destination or Pipeline into Terraform.

Step 1: Find the IDs of your Resources

Terraform finds each existing resource in Hevo by its ID. A Destination uses its own ID, and the resources of a Pipeline all use the ID of the Pipeline.

The following table lists the ID to use for each resource:

ResourceID to use
hevo_destinationThe ID of the Destination.
hevo_pipelineThe ID of the Pipeline.
hevo_pipeline_objectsThe ID of the Pipeline.
hevo_pipeline_stateThe ID of the Pipeline.

To find the IDs, list your Destinations and Pipelines with the hevo_destinations and hevo_pipelines data sources. With the IDs of your resources, you can link each resource in your configuration to its Destination or Pipeline in Hevo.

Step 2: Link the Resources

Terraform links each resource in your configuration to its Destination or Pipeline in Hevo through an import block. Terraform can also generate the configuration of these resources for you.

Add an import block for each resource to your configuration:

import {
  to = hevo_destination.redshift
  id = "5382"
}

import {
  to = hevo_pipeline.mysql
  id = "4204"
}

import {
  to = hevo_pipeline_objects.mysql
  id = "4204"
}

import {
  to = hevo_pipeline_state.mysql
  id = "4204"
}

Then, run the terraform plan command with the -generate-config-out option, so that Terraform generates the configuration of these resources in a new file:

terraform plan -generate-config-out=generated.tf

Terraform reads each resource from Hevo and writes its settings to generated.tf. Move the resources from this file into your configuration. The generated Destination does not contain its credentials, so the command also displays an error for each missing credential. Terraform still writes the file, and you add the credentials in the next step. Terraform also displays a warning that generating configuration is experimental.

You can also write the resource blocks yourself, and import each resource with the terraform import command. The following command imports the Destination with the ID 5382 into the hevo_destination.redshift resource:

terraform import hevo_destination.redshift 5382

If no resource exists with the ID, Terraform displays an error that the object does not exist. Check the ID, and check that your provider is configured for the Hevo account that the resource belongs to.

The resources are linked to your configuration, and you can add the settings that Hevo does not return.

Step 3: Complete the Configuration

Hevo does not return some settings, so the generated configuration does not contain them. Add them before you apply the configuration:

  • Credentials of the Destination: Hevo returns credentials, such as database_password, only as ********, so the generated configuration leaves them out. Add each credential to config, through a sensitive variable.

  • Settings of the Source: Hevo does not return the settings of a Source, so the generated hevo_pipeline resource contains source_config = null. You can keep it, and the Source keeps its settings in Hevo. To manage the settings of the Source with Terraform, replace it with the settings of your Source, including its credentials.

  • Excluded fields: The provider does not read the excluded_fields of objects from Hevo. Add them to your configuration with the fields that the Pipeline excludes in Hevo. The first apply sends them again, which does not change anything in Hevo.

  • Credentials of your OAuth app: If your Source uses your own OAuth app, add source_auth.

  • References between resources: The generated configuration contains the IDs of the resources, such as destination_id = "5382". Replace them with references, such as hevo_destination.redshift.id and hevo_pipeline.mysql.id, so that Terraform creates and changes the resources in the right order. Also add depends_on = [hevo_pipeline_objects.mysql] to the hevo_pipeline_state resource. Read Ordering Object Selection and Pipeline State for the reason.

The configuration is complete, and you can apply it.

Step 4: Apply the Configuration

Terraform links your configuration to the existing Destinations and Pipelines when you apply it for the first time. It sends the credentials of each Destination, and records the Source settings that you added, without changing the other settings in Hevo.

Perform the following steps to apply the configuration:

  1. Run the terraform plan command, and review the changes. Before the first apply, the plan can show the following changes:

    • The credentials of the Destination that you added are shown as added, such as + database_password = (sensitive value).

    • If you added source_config, it is shown as added. Terraform does not send it to Hevo.

  2. Run the terraform apply command. In this first apply:

    • Terraform sends the credentials of the Destination, and Hevo tests the connection to the Destination.

    • If you added source_config, Terraform saves it in the state file without sending it to Hevo, and displays a warning that it recorded the settings without sending them.

  3. Run the terraform plan command again. The output shows No changes, and Terraform manages the resources from this point.

After this, every change to source_config is sent to Hevo, which tests the connection to the Source with the new settings. If you remove source_config later, Terraform does not send anything to Hevo, and the Source keeps its settings.