Pagination

When an API request returns a large number of records in a single response, it can slow performance and make data handling difficult. Pagination divides the response into smaller, manageable chunks called pages, controlling the number of records per request and making large datasets easier to work with.

Hevo uses cursor-based pagination through the cursor and limit parameters. A cursor marks the position of the last record that you received, so your next request continues from that point instead of starting again.

Refer to the following sections to page through a list:

• Query Parameters: The parameters that control each page.
• Paging through Results: Fetch every page in turn.

Query Parameters

Query parameters are a part of the URL in the HTTP request and can be used to filter or paginate the API results. They are added to the URL after a question mark (?) and are given as key-value pairs. For example, ?limit=20 asks for 20 records in the response.

Every list endpoint accepts the following query parameters:

A list response returns the following fields in addition to code:

Send a cursor only to the endpoint that returned it. Send the next_cursor value exactly as you received it. A cursor that cannot be decoded returns a 400.

A cursor that decodes but came from another list is not always rejected. It can return an unexpected part of the list instead of an error. The cursor of the List All Jobs for a Pipeline endpoint marks a position in the list rather than a job, so send the same limit and filters with each cursor. Jobs are listed newest first, so a job can appear on two pages if new jobs start while you page through the list. A limit that is not an integer, or that is outside the accepted range, returns a 400.

Paging through Results

Send your first request without a cursor, then send the next_cursor value from each response back as the cursor of the next request. Stop when the has_more field value is false.

Note: In the example requests below, replace <ACCESS_KEY> and <SECRET_KEY> with your API credentials. Read Generating your API Credentials for the steps to generate them.

Request 1: Fetch the first page. No cursor is sent, so Hevo returns the records from the beginning, and limit=20 asks for up to 20 records in this response:

curl --request GET \
  --url 'https://asia.hevodata.com/api/v2/pipelines?limit=20' \
  --header 'accept: application/json' \
  --user '<ACCESS_KEY>:<SECRET_KEY>'

The response returns the first 20 Pipelines, a cursor, and has_more as true, which tells you that more Pipelines are available:

{
  "code": "Success",
  "data": [ ... 20 Pipelines ... ],
  "next_cursor": "MTI0NQ==",
  "has_more": true
}

Request 2: Fetch the next page by sending the next_cursor value from the first response as your cursor. The value must be URL-encoded, so MTI0NQ== is sent as MTI0NQ%3D%3D:

curl --request GET \
  --url 'https://asia.hevodata.com/api/v2/pipelines?limit=20&cursor=MTI0NQ%3D%3D' \
  --header 'accept: application/json' \
  --user '<ACCESS_KEY>:<SECRET_KEY>'

Some characters have a special meaning in a URL, such as the = character at the end of a cursor value. To send such a character as part of a value, replace it with its code. The code for = is %3D, so MTI0NQ== becomes MTI0NQ%3D%3D. cURL does not make this replacement, so make it yourself before you add the cursor value to the URL, as in the example above. Most HTTP client libraries make the replacement automatically.

The response returns the next 20 Pipelines and a new cursor:

{
  "code": "Success",
  "data": [ ... 20 Pipelines ... ],
  "next_cursor": "MTI5OQ==",
  "has_more": true
}

Repeat this until has_more is false and next_cursor is null, which means that you have retrieved every record:

{
  "code": "Success",
  "data": [ ... 7 Pipelines ... ],
  "next_cursor": null,
  "has_more": false
}

The data array on the last page can return fewer records than the limit you requested, as in the response above. Use has_more to decide whether to fetch another page, rather than counting the records returned.