ElasticSearch Advanced Settings

For users who have some experience working with their data in Panoply, there are a number of items that can be customized for this data source.

1. Incremental Key: By default, Panoply fetches all of your ElasticSearch data on each run. If you only want to collect some of your data, enter a column name to use as the incremental key. The column must be logically incremental. Panoply will keep track of the maximum value reached during the previous run and will start there on the next run. See Incremental Loads for more information.

2. Destination Schema: This is the name of the target schema to save the data. The default schema for data warehouses built on Google BigQuery is panoply. The default schema for data warehouses built on Amazon Redshift is public. This cannot be changed once a source has been collected.
3. Destination: This is the name of the table Panoply will create with your data. The default for ElasticSearch is `elasticsearch_{index){_type}.
4. Primary Key: Primary Keys are the column(s) values that uniquely identify a row. Panoply uses the defaul primary key of {_type}#{_id} as provided by ElasticSearch.
5. Exclude: The Exclude option allows you to exclude certain data, such as names, addresses, or other personally identifiable information. Enter the column names of the data to exclude.
6. Parse String: If the data to be collected contains JSON, include the JSON text attributes to be parsed.
7. Truncate: Truncate deletes all the current data stored in the destination tables, but not the tables themselves. Afterwards Panoply will recollect all the available data for this data source.
8. Click Save Changes and then Collect.
   • The data source appears grayed out while the collection runs.
   • You may add additional data sources while this collection runs.
   • You can monitor this collection from the Jobs page or the Data Sources page.
   • After a successful collection, navigate to the Tables page to review the data results.