A Terraform configuration describes the Destinations and Pipelines that you want in your Hevo account. The following steps show how to create a Pipeline with Terraform, using an Amazon RDS MySQL Source and an Amazon Redshift Destination as an example. The configuration creates the Destination and the Pipeline, selects two tables as the objects of the Pipeline, and enables the Pipeline. To use another Source or Destination, change the connector and its settings, as described in the Find the Connectors and Their Settings section.
Prerequisites
-
An active Hevo account. You can start with a 14-day full-feature, free trial account.
-
You are assigned the required role for the actions that you want to perform.
-
You have generated your API credentials. Read Authentication for the steps to generate them.
-
The machine or CI/CD runner that runs Terraform uses Terraform version 1.5 or higher.
-
An Amazon RDS MySQL database configured for replication.
-
An Amazon Redshift cluster configured as a Hevo Destination.
-
You have the connection details and credentials of the Amazon RDS MySQL database and the Amazon Redshift cluster.
Step 1: Set Up the Provider
Terraform manages your Hevo account through the Hevo provider.
Perform the following steps to declare the provider in a new configuration, provide the API credentials and the host of your Hevo account, and download the provider:
-
Create a folder for your configuration, and in it, create a file named
main.tfwith the following content:terraform { required_providers { hevo = { source = "hevoio/hevo" version = "~> 1.0" } } } provider "hevo" {}The
versionargument sets the versions of the provider that Terraform can install. -
In the shell where you run Terraform, set your API credentials and the host of your Hevo account:
export HEVO_ACCESS_KEY="<your access key>" export HEVO_SECRET_KEY="<your secret key>" export HEVO_HOST="https://us.hevodata.com"Replace
https://us.hevodata.comwith the host of your Hevo account. To provide your credentials through Terraform variables instead, read the Providing your API Credentials section. -
Run the terraform init command to download the provider:
terraform initTerraform downloads the provider and confirms that the configuration is initialized.
The provider is set up, and you can find the connectors for the Destination and the Pipeline.
Step 2: Find the Connectors and Their Settings
A connector is the integration that Hevo uses for a type of Source or Destination. The Amazon RDS MySQL Source uses the SOURCE_CONNECTOR_MYSQL connector, and the Amazon Redshift Destination uses the redshift connector. A Source connector can have more than one Source type, such as Amazon RDS MySQL and Google Cloud MySQL for the MySQL connector.
Perform the following steps to find the settings that each connector requires:
-
Add the following data sources to
main.tf:data "hevo_connector_metadata" "mysql" { kind = "source" connector_id = "SOURCE_CONNECTOR_MYSQL" } data "hevo_connector_metadata" "redshift" { kind = "destination" connector_id = "redshift" } output "mysql_required_settings" { value = data.hevo_connector_metadata.mysql.required_config_keys } output "redshift_required_settings" { value = data.hevo_connector_metadata.redshift.required_config_keys } output "mysql_source_types" { value = data.hevo_connector_metadata.mysql.source_types } -
Run the terraform plan command. The output lists the settings that each connector requires, and the Source types of the MySQL connector, such as
AWS_RDS_MYSQL. The list contains only the settings that a connector always requires. A setting that is required only for a particular choice, such as the credentials for an authentication type, is described in theschema_jsonattribute of the hevo_connector_metadata data source.To find the connector for another Source or Destination, use the hevo_connectors data source.
You know the settings that the Destination and the Pipeline require, and you can define the Destination.
Step 3: Define the Destination
The Destination is the Amazon Redshift cluster that the Pipeline loads data into. You define it with a hevo_destination resource, which reads the connection details of the cluster from variables.
Perform the following steps to define the Destination:
-
Add variables for the connection details of your Redshift cluster. Declare the password as sensitive, so that Terraform does not display it:
variable "redshift_host" { type = string } variable "redshift_user" { type = string } variable "redshift_password" { type = string sensitive = true } -
Add the Destination:
resource "hevo_destination" "redshift" { name = "Redshift Warehouse" connector_id = "redshift" config = { cluster_identifier = var.redshift_host database_port = 5439 database_name = "analytics" database_user = var.redshift_user database_password = var.redshift_password } }Specify the endpoint of your Redshift cluster, without the port, in
cluster_identifier. For example,examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com.
The Destination is defined, and you can define the Pipeline that loads data into it.
Step 4: Define the Pipeline
The Pipeline replicates data from the Amazon RDS MySQL database into the Destination that you defined in the previous step. You define the Pipeline and its Source with a hevo_pipeline resource, which reads the connection details of the database from variables.
Perform the following steps to define the Pipeline:
-
Add variables for the connection details of your MySQL database:
variable "mysql_host" { type = string } variable "mysql_user" { type = string } variable "mysql_password" { type = string sensitive = true } -
Add the Pipeline:
resource "hevo_pipeline" "mysql" { name = "MySQL to Redshift" source_name = "MySQL Production" source_connector_id = "SOURCE_CONNECTOR_MYSQL" source_type = "AWS_RDS_MYSQL" source_config = { database_host = var.mysql_host database_port = 3306 database_user = var.mysql_user database_password = var.mysql_password database_names = "employees" } destination_id = hevo_destination.redshift.id destination_prefix = "mysql" replication_type = "HISTORICAL_AND_INCREMENTAL" load_mode = "MERGE" schema_evolution = "BLOCK_ALL" schedule = { sync_type = "SCHEDULED" frequency_minutes = 60 } failure_handling_policy = { object_failure_level = "PARTIAL" object_failure_threshold = 5 } }The
source_typeargument sets the type of your Source, here Amazon RDS MySQL. If you do not specify it, Hevo uses the default Source type of the connector, such asMYSQLfor the MySQL connector.
The Pipeline is defined. The provider creates it in the disabled state, so that you can select its objects before it starts to replicate data.
Step 5: Select the Objects and Enable the Pipeline
To start the Pipeline, select the tables that it replicates and enable it. You select the objects with a hevo_pipeline_objects resource, and enable the Pipeline with a hevo_pipeline_state resource.
Perform the following steps to select the objects and enable the Pipeline:
-
Add the objects that you want the Pipeline to replicate. For a MySQL Source, name each table as
<database>.<table>:resource "hevo_pipeline_objects" "mysql" { pipeline_id = hevo_pipeline.mysql.id select = "listed" objects = { "employees.departments" = {} "employees.dept_manager" = {} } }To find the names of the objects of an existing Pipeline, use the hevo_pipeline_objects data source.
-
Enable the Pipeline:
resource "hevo_pipeline_state" "mysql" { pipeline_id = hevo_pipeline.mysql.id paused = false depends_on = [hevo_pipeline_objects.mysql] }The depends_on argument makes Terraform select the objects before it enables the Pipeline. A Pipeline cannot be enabled until at least one of its objects is selected.
The configuration is complete, and you can apply it.
Step 6: Apply the Configuration
Terraform creates the Destination, the Pipeline, its objects, and its state in your Hevo account only when you apply the configuration.
Perform the following steps to apply the configuration:
-
Set the values of your variables in
TF_VAR_environment variables:export TF_VAR_redshift_host="<your cluster endpoint>" export TF_VAR_redshift_user="<your Redshift user>" export TF_VAR_redshift_password="<your Redshift password>" export TF_VAR_mysql_host="<your MySQL host>" export TF_VAR_mysql_user="<your MySQL user>" export TF_VAR_mysql_password="<your MySQL password>" -
Preview the changes with the terraform plan command:
terraform planThe plan lists the four resources that Terraform creates. Hevo tests the connection to the Destination and the Source when Terraform creates them, so a connection error is reported only when you apply the configuration.
-
Create the resources with the terraform apply command:
terraform applyTerraform creates the Destination, then the Pipeline, then selects the objects, and then enables the Pipeline. Enabling the Pipeline can take a few minutes.
-
Run the
terraform plancommand again. The output showsNo changes, which confirms that your Hevo account matches your configuration.
The Pipeline replicates the two tables. You can view it on the Pipelines page of the Hevo UI, or read its jobs with the hevo_pipeline_jobs data source.
Terraform saves the state of your resources, including the passwords, in the terraform.tfstate file. Protect the state file by storing it in a location that only authorized users can access.