Flex Connector Pagination

This document describes the different pagination configuration options of the Flex Connector. Pagination can help in two different aspects:

1. Allow the Flex Connector to get all the desired data our of the API
2. Improve efficiency of API calls

The available configurations of the pagination will be determined by the API method selected. First, you will need to enable pagination and then set the configuration based on the API specifications.

GET

• Get next page value - Will determine if the next page information can be found in the previous API call's response or if there's a need in a standard pagination. If the next page information is in the API call's response, select Dynamic. If the the pagination is incremental, select Static.

Static pagination

• Start page - Select the first page. The default is 1
• Page step - Select the increment size between pages. The default is 1
• Use next page value - Define how will the next page value will be used:
  • As a URL parameter value - Add an additional URL parameter and paginate using the value of this parameter. For example page
  • As a URL postfix - Extend the base URL with an increment for each page. for example, if the base URL is https://apiservice.com/api then the pagination will be https://apiservice.com/api/1 for the first page.
• URL/Body key - The parameter name. This will be added to the URL parameters when paginating across the results. This value supports nested parameters like param1.param2. You can escape dot character with a backslash: param1\.param2
• URL parameters to remove - Select which URL parameters should not be included during pagination. Enter a ; delimited list with all the desired URL parameters.
• URL parameters to add - Select which URL parameters should be included during pagination. Enter a & separated parameters with the desired parameters and their values.
• Stop iterating when - Determine when the Flex Connector has extracted all the data and the pagination should stop:
  • Empty response - The response from the API is empty
  • Empty data - There is no data returned from the API. This is based on the selected data key.
  • Stop key-value pair in response - There is a key-value pair in the response of the API that determines if there is another page to extract. You will need to enter the Key in response and Value in response. For example:
    • Key in response: has_next_page
    • Value in response: False
  • Total pages in response - There is a field in the response that determines how many pages are there to extract. When reaching the last page, the pagination will stop. You will need to enter the Key in response. For example: total_pages.
  • Total objects in response is smaller than page size - Identify if the response has less records returned compared to the page size. You will need to enter the Key in response to determine the returned number of records and Objects per page to determine how many records were expected. For example:
    • Key in response: results
    • Objects per page: 50

Dynamic pagination

• Next page key in response - Enter the path to the value in the response that determines the next page. In some cases, the dynamic pagination will need to use a value from the last record received. For example: { "per_page": 2, "total_objects": 100, "data": [{"id": 1, "city": "city1"}] [{"id": 2, "city": "city2"}]}. In such cases you should use the [-1] to navigate to the last record. In the example above, use data[-1].id
• Use next page value - Define how will the next page value will be used:
  • As a URL parameter value - Add an additional URL parameter and paginate using the value of this parameter. For example page
  • As a whole URL - Replace the base URL with the URL returned from the API call's response
  • As a URL postfix - Extend the base URL with an increment for each page. for example, if the base URL is https://apiservice.com/api then the pagination will be https://apiservice.com/api/1 for the first page.
• URL/Body key - The parameter name. This will be added to the URL parameters when paginating across the results. This value supports nested parameters like param1.param2. You can escape dot character with a backslash: param1\.param2
• URL parameters to remove - Select which URL parameters should not be included during pagination. Enter a ; delimited list with all the desired URL parameters.
• URL parameters to add - Select which URL parameters should be included during pagination. Enter a & separated parameters with the desired parameters and their values.
• Stop iterating when - Determine when the Flex Connector extracted all the data and the pagination should stop:
  • No next page in response - The API call's response does not include in it the next page information
  • Empty response - The response from the API is empty
  • Empty data - There is no data returned from the API. This is based on the selected data key.
  • Stop key-value pair in response - There is a key-value pair in the response of the API that determines if there is another page to extract. You will need to enter the Key in response and Value in response. For example:
    • Key in response: has_next_page
    • Value in response: False

POST (Raw input)

• Get next page value - Will determine if the next page information can be found in the previous API call's response or if there's a need in an incremental pagination. If the next page information is in the API call's response, select Dynamic. If the the pagination is incremental, select Static.

Static pagination

• Start page - Select the first page. The default is 1
• Page step - Select the increment size between pages. The default is 1
• Use next page value - Define how will the next page value will be used:
  • As a URL parameter value - Add an additional URL parameter and paginate using the value of this parameter. For example page
  • As a URL postfix - Extend the base URL with an increment for each page. for example, if the base URL is https://apiservice.com/api then the pagination will be https://apiservice.com/api/1 for the first page.
  • As a parameter in body - Add an additional parameter to the body of the API call and paginate using the value of this parameter. For example page
• URL/Body key - The parameter name. This will be added to the URL parameters when paginating across the results. This value supports nested parameters like param1.param2. You can escape dot character with a backslash: param1\.param2
• POST raw data to remove - Select which POST raw parameters should not be included during pagination. Enter a ; delimited list with all the desired raw data parameters.
• POST raw data to add - Select which POST raw parameters should be included during pagination. Enter a JSON or XML with all the desired raw data parameters and their values.
• URL parameters to remove - Select which URL parameters should not be included during pagination. Enter a ; delimited list with all the desired URL parameters.
• URL parameters to add - Select which URL parameters should be included during pagination. Enter a & separated parameters with the desired parameters and their values.
• Stop iterating when - Determine when the Flex Connector extracted all the data and the pagination should stop:
  • Empty response - The response from the API is empty
  • Empty data - There is no data returned from the API. This is based on the selected data key.
  • Stop key-value pair in response - There is a key-value pair in the response of the API that determines if there is another page to extract. You will need to enter the Key in response and Value in response. For example:
    • Key in response: has_next_page
    • Value in response: False
  • Total pages in response - There is a field in the response that determines how many pages are there to extract. When reaching the last page, the pagination will stop. You will need to enter the Key in the response. For example: total_pages.
  • Total objects in response is smaller than page size - Identify if the response has less records returned compared to the page size. You will need to enter the Key in response to determine the returned number of records and Objects per page to determine how many records were expected. For example:
    • Key in response: results
    • Objects per page: 50

Dynamic pagination

• Next page key in response - Enter the path to the value in the response that determines the next page. In some cases, the dynamic pagination will need to use a value from the last record received. For example: { "per_page": 2, "total_objects": 100, "data": [{"id": 1, "city": "city1"}] [{"id": 2, "city": "city2"}]}. In such cases you should use the [-1] to navigate to the last record. In the example above, use data[-1].id
• Use next page value - Define how will the next page value will be used:
  • As a URL parameter value - Add an additional URL parameter and paginate using the value of this parameter. For example page
  • As a whole URL - Replace the base URL with the URL returned from the API call's response
  • As a URL postfix - Extend the base URL with an increment for each page. for example, if the base URL is https://apiservice.com/api then the pagination will be https://apiservice.com/api/1 for the first page.
  • As a parameter in body - Add an additional parameter to the body of the API call and paginate using the value of this parameter. For example page
• URL/Body key - The parameter name. This will be added to the URL or body parameters when paginating across the results. This value supports nested parameters like param1.param2. You can escape dot character with a backslash: param1\.param2
• POST raw data to remove - Select which POST raw parameters should not be included during pagination. Enter a ; delimited list with all the desired raw data parameters.
• POST raw data to add - Select which POST raw parameters should be included during pagination. Enter a JSON with all the desired raw data parameters and their values.
• URL parameters to remove - Select which URL parameters should not be included during pagination. Enter a ; delimited list with all the desired URL parameters.
• URL parameters to add - Select which URL parameters should be included during pagination. Enter a & separated parameters with the desired parameters and their values.
• Stop iterating when - Determine when the Flex Connector extracted all the data and the pagination should stop:
  • No next page in response - The API call's response does not include in it the next page information
  • Empty response - The response from the API is empty
  • Empty data - There is no data returned from the API. This is based on the selected data key.
  • Stop key-value pair in response - There is a key-value pair in the response of the API that determines if there is another page to extract. You will need to enter the Key in response and Value in response. For example:
    • Key in response: has_next_page
    • Value in response: False

POST (GraphQL input)

• Get next page value - Will determine if the next page information can be found in the previous API call's response (using cursors) or if there's a need in an incremental pagination. If the next page information is in the API call's response, select Cursor. If the the pagination is incremental, select Static.

Static pagination

• Use next page value - Define how will the next page value will be used:
  • As a parameter in graphQL body - Add an additional parameter to the graphQL query and paginate using the value of this parameter. For example offset
• Body key - The parameter name. This will be added to the graphQL parameters when paginating across the results.
• Stop iterating when - Determine when the Flex Connector extracted all the data and the pagination should stop:
  • Empty data - There is no data returned from the API. This is based on the selected data key.
• Objects per page key - graphQL parameter that defines how many records will return in each page
• Objects per page - The value of the parameter above

Cursor pagination

• Next cursor in response - Path to the value of the next endCursor in the response
• Stop key in response - Path to the stop parameter in the response. Based on this parameter the Flex Connector will know when to stop paginating
• Stop value in response - The value of the parameter above that
• Objects per page key - graphQL parameter that defines how many records will return in each page
• Objects per page - The value of the parameter above