Storage connectors support two import modes: Fixed file and Most recent file. This article covers both modes, with extra detail on Most recent file.
Fixed File
This import mode will choose one individual file name.
Just input the file name and location (ex: folder1/folder2/file.csv) and Select on Import.
This is a good set up if your file’s name will not change or you only wanted to update a single file with the storage connector. If the configuration is saved and imported again, it will take the latest version of that file, as long as the name hasn’t changed.
Most recent file
This import mode allows you to define a path and Pigment will upload the most recent file depending on a defined naming convention that must contain a ISO-8601 compatible date format within it.
Naming convention in storage connector
When naming your files in the storage connector, your file name must follow a consistent naming convention with the date included. It doesn’t matter where the date is in the name as long as it uses the ISO-8601 format and it remains consistent.
For example, you could have 2022-09-30-rev.csv, where the date is in the front and -rev is at the end. You could also have rev-2022-09-30.csv. The important thing is to remain consistent with all files after that. Once you define the naming convention, you’ll have to write the correct path and naming convention in Pigment.
Here are some examples of file names for a rev.csv with the ISO-8601 compatible date formats located at the start.
ISO-8601 compatible date formats | example of file names {{date}}-rev.csv naming convention |
|---|---|
| 2022-09-30-rev.csv |
| 2022-09-30T00:00:00-rev.csv |
| 2022-09-30T00:00:00Z-rev.csv |
When writing the file path in Pigment
Similar to the Fixed file, you must first define the path with the location(ex: folder1/folder2/ ) ,then you must define file naming convention with a date included in the file name. When entering the case sensitive path in Pigment, use {{date}} where the date is located.
It should look similar to this: folder1/folder2/file-{{date}}.csv
Here are some examples of file names and the results depending on what path you entered.
If the bucket contains the following files:
/myfolderA/2022-09-15-salaries.csv
/myfolderA/2022-10-15-salaries.csv
/myfolderA/2022-10-16-revenue.csv
/myfolderB/2022-11-15-salaries.csv
Here are the results I would get depending on my file path :
path "myfolderA/{{date}}-salaries.csv" will import
/myfolderA/2022-10-15-salaries.csvbecause this is the most recent file in myfolderA.path "myfolderB/{{date}}-salaries.csv" will import
/myfolderB/2022-11-15-salaries.csvbecause this is the most recent file in myfolderB.path "/{{date}}-salaries.csv" will import nothing because its missing the folder location.
⚠️ Important
In order to get the Start Import button to be Selectable, you must have {{date}} and .csv and in the pattern.
Example
Here is an example from a an S3 folder with an example of a naming convention.
Select Show Content below to see the correct path needed to pull the most recent file.
Show content
folder/Country-{{date}}.csv is the correct file path and naming convention.
It is important to match case and to include {{date}} where the date is located.
Select Show Content below to see which file would be uploaded
Show Content
Country-2022-07-26TO81417.csv