Creating your First Pipeline with Terraform

A Terraform configuration describes the Destinations and Pipelines that you want in your Hevo account. The following steps show how to create a Pipeline with Terraform, using an Amazon RDS MySQL Source and an Amazon Redshift Destination as an example. The configuration creates the Destination and the Pipeline, selects two tables as the objects of the Pipeline, and enables the Pipeline. To use another Source or Destination, change the connector and its settings, as described in the Find the Connectors and Their Settings section.

Prerequisites

Step 1: Set Up the Provider

Terraform manages your Hevo account through the Hevo provider.

Perform the following steps to declare the provider in a new configuration, provide the API credentials and the host of your Hevo account, and download the provider:

  1. Create a folder for your configuration, and in it, create a file named main.tf with the following content:

    terraform {
      required_providers {
        hevo = {
          source  = "hevoio/hevo"
          version = "~> 1.0"
        }
      }
    }
    
    provider "hevo" {}

    The version argument sets the versions of the provider that Terraform can install.

  2. In the shell where you run Terraform, set your API credentials and the host of your Hevo account:

    export HEVO_ACCESS_KEY="<your access key>"
    export HEVO_SECRET_KEY="<your secret key>"
    export HEVO_HOST="https://us.hevodata.com"

    Replace https://us.hevodata.com with the host of your Hevo account. To provide your credentials through Terraform variables instead, read the Providing your API Credentials section.

  3. Run the terraform init command to download the provider:

    terraform init

    Terraform downloads the provider and confirms that the configuration is initialized.

The provider is set up, and you can find the connectors for the Destination and the Pipeline.

Step 2: Find the Connectors and Their Settings

A connector is the integration that Hevo uses for a type of Source or Destination. The Amazon RDS MySQL Source uses the SOURCE_CONNECTOR_MYSQL connector, and the Amazon Redshift Destination uses the redshift connector. A Source connector can have more than one Source type, such as Amazon RDS MySQL and Google Cloud MySQL for the MySQL connector.

Perform the following steps to find the settings that each connector requires:

  1. Add the following data sources to main.tf:

    data "hevo_connector_metadata" "mysql" {
      kind         = "source"
      connector_id = "SOURCE_CONNECTOR_MYSQL"
    }
    
    data "hevo_connector_metadata" "redshift" {
      kind         = "destination"
      connector_id = "redshift"
    }
    
    output "mysql_required_settings" {
      value = data.hevo_connector_metadata.mysql.required_config_keys
    }
    
    output "redshift_required_settings" {
      value = data.hevo_connector_metadata.redshift.required_config_keys
    }
    
    output "mysql_source_types" {
      value = data.hevo_connector_metadata.mysql.source_types
    }
  2. Run the terraform plan command. The output lists the settings that each connector requires, and the Source types of the MySQL connector, such as AWS_RDS_MYSQL. The list contains only the settings that a connector always requires. A setting that is required only for a particular choice, such as the credentials for an authentication type, is described in the schema_json attribute of the hevo_connector_metadata data source.

    To find the connector for another Source or Destination, use the hevo_connectors data source.

You know the settings that the Destination and the Pipeline require, and you can define the Destination.

Step 3: Define the Destination

The Destination is the Amazon Redshift cluster that the Pipeline loads data into. You define it with a hevo_destination resource, which reads the connection details of the cluster from variables.

Perform the following steps to define the Destination:

  1. Add variables for the connection details of your Redshift cluster. Declare the password as sensitive, so that Terraform does not display it:

    variable "redshift_host" {
      type = string
    }
    
    variable "redshift_user" {
      type = string
    }
    
    variable "redshift_password" {
      type      = string
      sensitive = true
    }
  2. Add the Destination:

    resource "hevo_destination" "redshift" {
      name         = "Redshift Warehouse"
      connector_id = "redshift"
    
      config = {
        cluster_identifier = var.redshift_host
        database_port      = 5439
        database_name      = "analytics"
        database_user      = var.redshift_user
        database_password  = var.redshift_password
      }
    }

    Specify the endpoint of your Redshift cluster, without the port, in cluster_identifier. For example, examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com.

The Destination is defined, and you can define the Pipeline that loads data into it.

Step 4: Define the Pipeline

The Pipeline replicates data from the Amazon RDS MySQL database into the Destination that you defined in the previous step. You define the Pipeline and its Source with a hevo_pipeline resource, which reads the connection details of the database from variables.

Perform the following steps to define the Pipeline:

  1. Add variables for the connection details of your MySQL database:

    variable "mysql_host" {
      type = string
    }
    
    variable "mysql_user" {
      type = string
    }
    
    variable "mysql_password" {
      type      = string
      sensitive = true
    }
  2. Add the Pipeline:

    resource "hevo_pipeline" "mysql" {
      name                = "MySQL to Redshift"
      source_name         = "MySQL Production"
      source_connector_id = "SOURCE_CONNECTOR_MYSQL"
      source_type         = "AWS_RDS_MYSQL"
    
      source_config = {
        database_host     = var.mysql_host
        database_port     = 3306
        database_user     = var.mysql_user
        database_password = var.mysql_password
        database_names    = "employees"
      }
    
      destination_id     = hevo_destination.redshift.id
      destination_prefix = "mysql"
      replication_type   = "HISTORICAL_AND_INCREMENTAL"
      load_mode          = "MERGE"
      schema_evolution   = "BLOCK_ALL"
    
      schedule = {
        sync_type         = "SCHEDULED"
        frequency_minutes = 60
      }
    
      failure_handling_policy = {
        object_failure_level     = "PARTIAL"
        object_failure_threshold = 5
      }
    }

    The source_type argument sets the type of your Source, here Amazon RDS MySQL. If you do not specify it, Hevo uses the default Source type of the connector, such as MYSQL for the MySQL connector.

The Pipeline is defined. The provider creates it in the disabled state, so that you can select its objects before it starts to replicate data.

Step 5: Select the Objects and Enable the Pipeline

To start the Pipeline, select the tables that it replicates and enable it. You select the objects with a hevo_pipeline_objects resource, and enable the Pipeline with a hevo_pipeline_state resource.

Perform the following steps to select the objects and enable the Pipeline:

  1. Add the objects that you want the Pipeline to replicate. For a MySQL Source, name each table as <database>.<table>:

    resource "hevo_pipeline_objects" "mysql" {
      pipeline_id = hevo_pipeline.mysql.id
      select      = "listed"
    
      objects = {
        "employees.departments"  = {}
        "employees.dept_manager" = {}
      }
    }

    To find the names of the objects of an existing Pipeline, use the hevo_pipeline_objects data source.

  2. Enable the Pipeline:

    resource "hevo_pipeline_state" "mysql" {
      pipeline_id = hevo_pipeline.mysql.id
      paused      = false
    
      depends_on = [hevo_pipeline_objects.mysql]
    }

    The depends_on argument makes Terraform select the objects before it enables the Pipeline. A Pipeline cannot be enabled until at least one of its objects is selected.

The configuration is complete, and you can apply it.

Step 6: Apply the Configuration

Terraform creates the Destination, the Pipeline, its objects, and its state in your Hevo account only when you apply the configuration.

Perform the following steps to apply the configuration:

  1. Set the values of your variables in TF_VAR_ environment variables:

    export TF_VAR_redshift_host="<your cluster endpoint>"
    export TF_VAR_redshift_user="<your Redshift user>"
    export TF_VAR_redshift_password="<your Redshift password>"
    export TF_VAR_mysql_host="<your MySQL host>"
    export TF_VAR_mysql_user="<your MySQL user>"
    export TF_VAR_mysql_password="<your MySQL password>"
  2. Preview the changes with the terraform plan command:

    terraform plan

    The plan lists the four resources that Terraform creates. Hevo tests the connection to the Destination and the Source when Terraform creates them, so a connection error is reported only when you apply the configuration.

  3. Create the resources with the terraform apply command:

    terraform apply

    Terraform creates the Destination, then the Pipeline, then selects the objects, and then enables the Pipeline. Enabling the Pipeline can take a few minutes.

  4. Run the terraform plan command again. The output shows No changes, which confirms that your Hevo account matches your configuration.

The Pipeline replicates the two tables. You can view it on the Pipelines page of the Hevo UI, or read its jobs with the hevo_pipeline_jobs data source.

Terraform saves the state of your resources, including the passwords, in the terraform.tfstate file. Protect the state file by storing it in a location that only authorized users can access.