File Upload Data Dictionary

Because data in your uploaded files is unique, Panoply cannot provide a complete data dictionary. However, we can provide you information about how Panoply automates the data schema and how certain file and data types are handled.

Supported File Types

Panoply supports the following file types for the File Upload data source:

• Archive (.tar and .gzip)
• JSON (.json) See our sample .json file
• Character-delimited files (.csv and .tsv) See our sample .csv file or sample .tsv file
• Microsoft Excel (.xlsx) See our sample .xlsx file
• Plain text (.txt)

Maximum size for uploading files is 100 MB. For larger files, consider using our Amazon S3 data source.

Special Requirements / Handling

Several of the supported file types have special requirements or handling.

Excel (.xslx) Files

Panoply collects the file into the destination table you entered. To have each sheet (tab) within the file into its own table, users must enter additional parameters.

Text (.txt) Files

Text files txt files can be tab delimited, which Panoply will automatically use to mark the columns. Other delimiters may also be used, but the user will have to specify that delimeter in the Advanced Settings.

Archives with unsupported files

Collection will fail if you try to upload an archive file (such as .tar or .gzip) that includes unsupported file types or other archive files. To resolve this issue, review your archive, remove unsupported files, upload the modified archive file, and collect again.

Header Handling

The first row of the spreadsheet or file with values will be considered the header for the table. If a column has data but nothing in the header row, Panoply will add a header consisting of the word column + the column letter, such as column A.

For duplicate headers, Panoply will rename the second instance of the header by appending the header with column and the column letter, such as Name Column A. Columns with a header and no data will not be collected.

Metadata

The following metadata columns are added by Panoply to the destination table(s):

• id: If the user does not enter a primary key, and no id column exists in the source, Panoply will insert an id. Formatted as a GUID, such as 2cd570d1-a11d-4593-9d29-9e2488f0ccc2
• __filename: This is the name of the file Panoply collected
• __senttime: Formatted as a datetime, such as 2018-06-26T01:26:14.695Z
• __updatetime: Formatted as a datetime, such as 2018-06-26T01:26:14.695Z
• __sheetname: The name of the sheet (Excel files.
• __xml_root: The name of the XML root element.
• __xml_item: The name of a child element (XML)