Best Practices for the Hevo Terraform Provider

Terraform changes your Hevo account directly from your configuration files. A credential in plain text, an unreviewed change, or a setting changed both in Terraform and in the Hevo UI can affect your Pipelines and the data that they replicate.

The following practices keep your Hevo configuration secure, predictable, and easy to maintain:

Providing Credentials Through Sensitive Variables

Store every credential, such as a database password, in a sensitive variable (sensitive = true). Provide its value through a TF_VAR_ environment variable, or as a secret in your CI/CD tool, such as GitHub Actions. Terraform then displays the value as (sensitive value) in the output of the terraform plan command.

Do not write a credential as a literal value in your configuration. Terraform displays literal values in plain text when you run the terraform plan and terraform apply commands, and anyone who can read the configuration files can read the credential.

Protecting the State File

Terraform saves the configuration of each resource, including the credentials of your Destinations and Sources, in plain text in its state file. This applies even to values that you provide through sensitive variables.

  • Store the state in a remote backend that encrypts it and limits access to it, instead of a local terraform.tfstate file.

  • Do not commit the state file to version control.

Reviewing the Plan Before You Apply

Run the terraform plan command before every terraform apply command, and check for changes that Terraform marks as a replacement with # forces replacement. A replacement deletes the resource and creates it again:

  • A replaced Destination is created again, and Hevo tests the connection to it. Terraform also replaces the Pipelines that it manages and that load data into the Destination. If a Pipeline that Terraform does not manage loads data into it, the replacement fails. Read Changing a Destination for the changes that replace a Destination.

  • A replaced Pipeline loads all the historical data of its objects again, if its replication_type is HISTORICAL_AND_INCREMENTAL.

Changing some arguments of a Pipeline replaces it. For such a change, the terraform plan command also displays a warning that the Pipeline will be replaced.

Applying Changes from your CI/CD System

Run Terraform from your Continuous Integration and Continuous Delivery (CI/CD) system, so that every change to your Hevo configuration is reviewed in a pull request before it is applied. For example, the following GitHub Actions workflow, saved as .github/workflows/hevo.yml in your repository, plans each pull request and applies the configuration after the pull request is merged:

name: Hevo Terraform

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

env:
  HEVO_ACCESS_KEY: ${{ secrets.HEVO_ACCESS_KEY }}
  HEVO_SECRET_KEY: ${{ secrets.HEVO_SECRET_KEY }}
  HEVO_HOST: https://us.hevodata.com
  TF_VAR_redshift_password: ${{ secrets.REDSHIFT_PASSWORD }}
  TF_VAR_mysql_password: ${{ secrets.MYSQL_PASSWORD }}

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: hashicorp/setup-terraform@v4
      - run: terraform init -input=false
      - run: terraform plan -input=false

  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    concurrency:
      group: hevo-terraform-apply
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v7
      - uses: hashicorp/setup-terraform@v4
      - run: terraform init -input=false
      - run: terraform apply -input=false -auto-approve

The workflow works as follows:

  • plan: The job that runs when a pull request is opened or updated. It runs the terraform plan command, and reviewers read the plan in the log of the job. Check the plan for replacements before you approve the pull request.

  • apply: The job that runs when the pull request is merged into the main branch. It runs the terraform apply command, which creates a new plan from the merged configuration and applies it. The -auto-approve option applies the plan without asking for confirmation.

  • concurrency: The group of the apply job, which ensures that only one apply changes your Hevo account at a time. If more pull requests are merged while an apply runs, GitHub keeps only the latest one waiting and cancels the others. This does not skip any change, because the latest run applies the complete configuration of the main branch.

  • env: The block that provides your API credentials and the host of your Hevo account through the provider environment variables, and the credentials of your Destination and Source through TF_VAR_ environment variables. Store each value as a GitHub secret, and add a TF_VAR_ line for each variable that your configuration declares.

Each run starts on a new machine, with no state from the previous run. Store the state in a remote backend, so that Terraform does not try to create your Destinations and Pipelines again.

Ordering Object Selection and Pipeline State

Terraform changes the object selection of a Pipeline before it enables or disables the Pipeline only if you add depends_on to the hevo_pipeline_state resource:

resource "hevo_pipeline_state" "mysql" {
  pipeline_id = hevo_pipeline.mysql.id
  paused      = false

  depends_on = [hevo_pipeline_objects.mysql]
}

Without depends_on, Terraform can send both changes at the same time, and the terraform apply command can fail, as Hevo does not accept a change to the objects while the Pipeline is being disabled.

Managing Similar Pipelines with for_each

If you replicate several databases with the same settings, define the Pipeline once and create a Pipeline for each database with the for_each argument:

locals {
  mysql_databases = {
    sales     = "sales_db"
    inventory = "inventory_db"
  }
}

resource "hevo_pipeline" "mysql" {
  for_each = local.mysql_databases

  name                = "MySQL ${each.key} to Redshift"
  source_name         = "MySQL ${each.key}"
  source_connector_id = "SOURCE_CONNECTOR_MYSQL"
  source_type         = "AWS_RDS_MYSQL"

  source_config = {
    database_host     = var.mysql_host
    database_port     = 3306
    database_user     = var.mysql_user
    database_password = var.mysql_password
    database_names    = each.value
  }

  destination_id     = hevo_destination.redshift.id
  destination_prefix = each.key
  replication_type   = "HISTORICAL_AND_INCREMENTAL"
  load_mode          = "MERGE"
  schema_evolution   = "BLOCK_ALL"

  schedule = {
    sync_type         = "SCHEDULED"
    frequency_minutes = 60
  }

  failure_handling_policy = {
    object_failure_level     = "PARTIAL"
    object_failure_threshold = 5
  }
}

Terraform creates one Pipeline for each entry in mysql_databases, and identifies each Pipeline by its key, such as hevo_pipeline.mysql["sales"]. To add a Pipeline, add an entry. If you remove an entry, Terraform deletes its Pipeline.

To select the objects of these Pipelines and enable them, add for_each = local.mysql_databases to the hevo_pipeline_objects and hevo_pipeline_state resources as well, and identify each Pipeline by its key, such as pipeline_id = hevo_pipeline.mysql[each.key].id.

Detecting Drift

Drift is a difference between your configuration and your Hevo account, caused by a change made outside Terraform, such as in the Hevo UI. To find drift, run the terraform plan command in refresh-only mode:

terraform plan -refresh-only

The output lists each resource that was changed outside Terraform, along with the settings that changed. To resolve the drift, do one of the following:

  • To keep the change, update your configuration to match it.

  • To undo the change, run the terraform apply command. Terraform restores the values from your configuration.

To check for drift from a script or a scheduled job, run the terraform plan -refresh-only -detailed-exitcode command. It exits with code 2 when it finds a difference, with 0 when it finds none, and with 1 when the command fails, for example, because of an incorrect API key.

The provider cannot detect a change made outside Terraform to a credential, to the settings of a Source, or to the excluded fields of an object, because the provider does not read their values from Hevo.

Managing Each Setting from One Place

Manage each Destination and Pipeline either with Terraform or in the Hevo UI, not both. A change made in the Hevo UI to a setting that Terraform manages is undone by the next terraform apply command. This does not apply to credentials, the settings of a Source, or excluded fields, because the provider cannot detect changes to them.

To stop managing a Pipeline with Terraform but keep the Pipeline in Hevo, remove the hevo_pipeline, hevo_pipeline_objects, and hevo_pipeline_state resources of the Pipeline from the state file with the terraform state rm command:

terraform state rm hevo_pipeline.mysql
terraform state rm hevo_pipeline_objects.mysql
terraform state rm hevo_pipeline_state.mysql

After you remove the resources from the state file, remove their blocks from your configuration. The Pipeline stays in your Hevo account as it is. If you remove the blocks first, the next terraform apply command deletes the Pipeline.

Staying Within the Rate Limits

Each request that the provider sends counts toward the rate limits of your team, along with the requests of every other API client of your team. Each provider configuration sends up to four requests at a time. If Hevo returns a 429 response, the provider waits for the number of seconds in the Retry-After header, and then sends the request again. During the wait, Terraform displays the usual progress messages, such as Still reading..., and the operation takes longer to complete. Terraform does not display an error.