An object is an entity in your Source application or database that contains the data Hevo must fetch, such as a table in a database, or companies in HubSpot. The hevo_pipeline_objects resource sets which objects a Pipeline replicates, and how each object is loaded.
The resource describes the complete selection of the Pipeline. When you apply it, Hevo selects and deselects objects to match your configuration.
Prerequisites
-
An active Hevo account. You can start with a 14-day full-feature, free trial account.
-
You are assigned the required role for the actions that you want to perform.
-
You have generated your API credentials. Read Authentication for the steps to generate them.
-
The machine or CI/CD runner that runs Terraform uses Terraform version 1.5 or higher.
-
You have configured the Hevo provider. Read Configuring the Hevo Terraform Provider for the steps.
-
An existing Pipeline, or a
hevo_pipelineresource in your configuration. Read Managing Pipelines with Terraform for the steps to create one.
Selecting Objects
The select argument sets how the provider reads your objects list:
-
listed: The Pipeline replicates only the objects inobjects. Hevo deselects every other object. Deselecting an object stops its replication, but leaves its Destination table and the data already loaded into it. Selecting an object again starts its replication, and its historical load runs in the next run of the Pipeline. -
all: The Pipeline replicates every object in the Source, except the objects that you list withenabled = false. The plan does not list the objects that this selects. To see and control each object in the plan, uselisted. To check the objects thatallselects before the Pipeline replicates any data, apply the selection while the Pipeline is disabled, and read the selected objects with the hevo_pipeline_objects data source, withstatuses = ["ACTIVE"], or review them in the Hevo UI.
The following example replicates only two tables from a MySQL database:
resource "hevo_pipeline_objects" "mysql" {
pipeline_id = hevo_pipeline.mysql.id
select = "listed"
objects = {
"employees.departments" = {}
"employees.dept_manager" = {}
}
}The following example replicates every table except two:
resource "hevo_pipeline_objects" "mysql" {
pipeline_id = hevo_pipeline.mysql.id
select = "all"
objects = {
"employees.salaries" = { enabled = false }
"employees.employees" = { enabled = false }
}
}If you later change select to listed, Hevo deselects every object that is not in objects.
Name each object by its fully qualified name, as Hevo lists it for the Pipeline. For example, a MySQL table is named <database>.<table>, such as employees.departments, and a SQL Server table is named <database>.<schema>.<table>, such as sales.dbo.orders. A SaaS object is named by its object name, such as companies in HubSpot. To see the names, read the objects of the Pipeline with the hevo_pipeline_objects data source.
If the hevo_pipeline_objects resource already exists, the provider checks the objects that you add when you run the terraform plan command. If the resource does not exist yet, Hevo checks the objects when you run the terraform apply command. In both cases, the command fails with an error that names an object that Hevo does not find.
If Terraform creates the Pipeline in the same apply, the provider cannot check the names during the plan. Hevo checks them when Terraform selects the objects. If Hevo does not find an object, the terraform apply command fails with an error that names the object.
Object Settings
Each object in objects accepts the following optional settings:
| Setting | Description |
|---|---|
enabled (Boolean) | The replication setting of the object. Set it to true to replicate the object, or false to deselect it. Default value: true. |
load_mode (String) | The load mode for this object, used instead of the load mode of the Pipeline. Set it to MERGE, APPEND, or HISTORY_TRACKING. The object must support the load mode. HISTORY_TRACKING is available only for some Sources and objects. Refer to the respective Source documentation to check whether the Source supports History mode. |
excluded_fields (Set of String) | The fields of this object that the Pipeline does not replicate. |
The following example loads the departments table in APPEND mode, and does not replicate the to_date field of the dept_manager table:
objects = {
"employees.departments" = { load_mode = "APPEND" }
"employees.dept_manager" = { excluded_fields = ["to_date"] }
}When you apply excluded_fields, Hevo stops replicating those fields. If you remove a field from excluded_fields, or remove excluded_fields from an object, Hevo replicates the field again.
The provider does not read the excluded fields from Hevo. As a result, the terraform plan command does not show a change made to the excluded fields of an object outside Terraform, such as in the Hevo UI.
Changes Made Outside Terraform
If an object is selected or deselected outside Terraform, for example, in the Hevo UI, the next terraform plan command shows this drift, and the terraform apply command restores your configuration.
While Hevo applies an earlier change to the Pipeline, it does not accept a new selection. The provider waits and sends the selection again until Hevo accepts it, for up to 20 minutes by default.
Removing the Resource
If you remove the hevo_pipeline_objects resource from your configuration, or destroy it, the Pipeline keeps its current selection. The provider does not deselect any object in Hevo.
Arguments and Attributes
Arguments are the settings of the object selection that you specify in your configuration. Attributes are the values that the provider sets when it creates the resource, such as its ID.
Arguments
The following table lists the arguments of the hevo_pipeline_objects resource:
| Argument | Description |
|---|---|
pipeline_id (String, Required) | The ID of the Pipeline for which the resource selects objects. Changing it replaces the resource. |
select (String, Required) | The objects that the Pipeline replicates. Set it to listed or all, as described in Selecting Objects. |
objects (Map, Optional) | The objects of the Pipeline, keyed by their fully qualified names, each with its object settings. |
timeouts (Object, Optional) | The time that the provider waits for a create or an update to finish. Default value: 20 minutes each. |
Attributes
The following table lists the attribute of the hevo_pipeline_objects resource:
| Attribute | Description |
|---|---|
id (String) | The ID of the Pipeline. |