Terraform changes your Hevo account directly from your configuration files. A credential in plain text, an unreviewed change, or a setting changed both in Terraform and in the Hevo UI can affect your Pipelines and the data that they replicate.
The following practices keep your Hevo configuration secure, predictable, and easy to maintain:
Providing Credentials Through Sensitive Variables
Store every credential, such as a database password, in a sensitive variable (sensitive = true). Provide its value through a TF_VAR_ environment variable, or as a secret in your CI/CD tool, such as GitHub Actions. Terraform then displays the value as (sensitive value) in the output of the terraform plan command.
Do not write a credential as a literal value in your configuration. Terraform displays literal values in plain text when you run the terraform plan and terraform apply commands, and anyone who can read the configuration files can read the credential.
Protecting the State File
Terraform saves the configuration of each resource, including the credentials of your Destinations and Sources, in plain text in its state file. This applies even to values that you provide through sensitive variables.
-
Store the state in a remote backend that encrypts it and limits access to it, instead of a local
terraform.tfstatefile. -
Do not commit the state file to version control.
Reviewing the Plan Before You Apply
Run the terraform plan command before every terraform apply command, and check for changes that Terraform marks as a replacement with # forces replacement. A replacement deletes the resource and creates it again:
-
A replaced Destination is created again, and Hevo tests the connection to it. Terraform also replaces the Pipelines that it manages and that load data into the Destination. If a Pipeline that Terraform does not manage loads data into it, the replacement fails. Read Changing a Destination for the changes that replace a Destination.
-
A replaced Pipeline loads all the historical data of its objects again, if its
replication_typeisHISTORICAL_AND_INCREMENTAL.
Changing some arguments of a Pipeline replaces it. For such a change, the terraform plan command also displays a warning that the Pipeline will be replaced.
Applying Changes from your CI/CD System
Run Terraform from your Continuous Integration and Continuous Delivery (CI/CD) system, so that every change to your Hevo configuration is reviewed in a pull request before it is applied. For example, the following GitHub Actions workflow, saved as .github/workflows/hevo.yml in your repository, plans each pull request and applies the configuration after the pull request is merged:
name: Hevo Terraform
on:
pull_request:
branches: [main]
push:
branches: [main]
env:
HEVO_ACCESS_KEY: ${{ secrets.HEVO_ACCESS_KEY }}
HEVO_SECRET_KEY: ${{ secrets.HEVO_SECRET_KEY }}
HEVO_HOST: https://us.hevodata.com
TF_VAR_redshift_password: ${{ secrets.REDSHIFT_PASSWORD }}
TF_VAR_mysql_password: ${{ secrets.MYSQL_PASSWORD }}
jobs:
plan:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: hashicorp/setup-terraform@v4
- run: terraform init -input=false
- run: terraform plan -input=false
apply:
if: github.event_name == 'push'
runs-on: ubuntu-latest
concurrency:
group: hevo-terraform-apply
cancel-in-progress: false
steps:
- uses: actions/checkout@v7
- uses: hashicorp/setup-terraform@v4
- run: terraform init -input=false
- run: terraform apply -input=false -auto-approveThe workflow works as follows:
-
plan: The job that runs when a pull request is opened or updated. It runs theterraform plancommand, and reviewers read the plan in the log of the job. Check the plan for replacements before you approve the pull request. -
apply: The job that runs when the pull request is merged into themainbranch. It runs theterraform applycommand, which creates a new plan from the merged configuration and applies it. The-auto-approveoption applies the plan without asking for confirmation. -
concurrency: The group of theapplyjob, which ensures that only one apply changes your Hevo account at a time. If more pull requests are merged while an apply runs, GitHub keeps only the latest one waiting and cancels the others. This does not skip any change, because the latest run applies the complete configuration of themainbranch. -
env: The block that provides your API credentials and the host of your Hevo account through the provider environment variables, and the credentials of your Destination and Source throughTF_VAR_environment variables. Store each value as a GitHub secret, and add aTF_VAR_line for each variable that your configuration declares.
Each run starts on a new machine, with no state from the previous run. Store the state in a remote backend, so that Terraform does not try to create your Destinations and Pipelines again.
Ordering Object Selection and Pipeline State
Terraform changes the object selection of a Pipeline before it enables or disables the Pipeline only if you add depends_on to the hevo_pipeline_state resource:
resource "hevo_pipeline_state" "mysql" {
pipeline_id = hevo_pipeline.mysql.id
paused = false
depends_on = [hevo_pipeline_objects.mysql]
}Without depends_on, Terraform can send both changes at the same time, and the terraform apply command can fail, as Hevo does not accept a change to the objects while the Pipeline is being disabled.
Managing Similar Pipelines with for_each
If you replicate several databases with the same settings, define the Pipeline once and create a Pipeline for each database with the for_each argument:
locals {
mysql_databases = {
sales = "sales_db"
inventory = "inventory_db"
}
}
resource "hevo_pipeline" "mysql" {
for_each = local.mysql_databases
name = "MySQL ${each.key} to Redshift"
source_name = "MySQL ${each.key}"
source_connector_id = "SOURCE_CONNECTOR_MYSQL"
source_type = "AWS_RDS_MYSQL"
source_config = {
database_host = var.mysql_host
database_port = 3306
database_user = var.mysql_user
database_password = var.mysql_password
database_names = each.value
}
destination_id = hevo_destination.redshift.id
destination_prefix = each.key
replication_type = "HISTORICAL_AND_INCREMENTAL"
load_mode = "MERGE"
schema_evolution = "BLOCK_ALL"
schedule = {
sync_type = "SCHEDULED"
frequency_minutes = 60
}
failure_handling_policy = {
object_failure_level = "PARTIAL"
object_failure_threshold = 5
}
}Terraform creates one Pipeline for each entry in mysql_databases, and identifies each Pipeline by its key, such as hevo_pipeline.mysql["sales"]. To add a Pipeline, add an entry. If you remove an entry, Terraform deletes its Pipeline.
To select the objects of these Pipelines and enable them, add for_each = local.mysql_databases to the hevo_pipeline_objects and hevo_pipeline_state resources as well, and identify each Pipeline by its key, such as pipeline_id = hevo_pipeline.mysql[each.key].id.
Detecting Drift
Drift is a difference between your configuration and your Hevo account, caused by a change made outside Terraform, such as in the Hevo UI. To find drift, run the terraform plan command in refresh-only mode:
terraform plan -refresh-onlyThe output lists each resource that was changed outside Terraform, along with the settings that changed. To resolve the drift, do one of the following:
-
To keep the change, update your configuration to match it.
-
To undo the change, run the
terraform applycommand. Terraform restores the values from your configuration.
To check for drift from a script or a scheduled job, run the terraform plan -refresh-only -detailed-exitcode command. It exits with code 2 when it finds a difference, with 0 when it finds none, and with 1 when the command fails, for example, because of an incorrect API key.
The provider cannot detect a change made outside Terraform to a credential, to the settings of a Source, or to the excluded fields of an object, because the provider does not read their values from Hevo.
Managing Each Setting from One Place
Manage each Destination and Pipeline either with Terraform or in the Hevo UI, not both. A change made in the Hevo UI to a setting that Terraform manages is undone by the next terraform apply command. This does not apply to credentials, the settings of a Source, or excluded fields, because the provider cannot detect changes to them.
To stop managing a Pipeline with Terraform but keep the Pipeline in Hevo, remove the hevo_pipeline, hevo_pipeline_objects, and hevo_pipeline_state resources of the Pipeline from the state file with the terraform state rm command:
terraform state rm hevo_pipeline.mysql
terraform state rm hevo_pipeline_objects.mysql
terraform state rm hevo_pipeline_state.mysqlAfter you remove the resources from the state file, remove their blocks from your configuration. The Pipeline stays in your Hevo account as it is. If you remove the blocks first, the next terraform apply command deletes the Pipeline.
Staying Within the Rate Limits
Each request that the provider sends counts toward the rate limits of your team, along with the requests of every other API client of your team. Each provider configuration sends up to four requests at a time. If Hevo returns a 429 response, the provider waits for the number of seconds in the Retry-After header, and then sends the request again. During the wait, Terraform displays the usual progress messages, such as Still reading..., and the operation takes longer to complete. Terraform does not display an error.