When working with storage connectors, there are two different import modes, Fixed file or Most recent file. This article will discuss Fixed file but go into a more in-depth look at how to use Most recent file.
Table of Contents
Fixed File
This import mode will choose one individual file name.
Just input the file name and location (ex: folder1/folder2/file.csv) and click on Import.
This is a good set up if your file’s name will not change or you only wanted to update a single file with the storage connector. If the configuration is saved and imported again, it will take the latest version of that file, as long as the name hasn’t changed.
Most recent file
This import mode allows you to define a path and Pigment will upload the most recent file depending on a defined naming convention that must contain a ISO-8601 compatible date format within it.
Naming convention in storage connector
When naming your files in the storage connector, your file name must follow a consistent naming convention with the date included. It doesn’t matter where the date is in the name as long as it uses the ISO-8601 format and it remains consistent.
For example, you could have 2022-09-30-rev.csv, where the date is in the front and -rev is at the end. You could also have rev-2022-09-30.csv. The important thing is to remain consistent with all files after that. Once you define the naming convention, you’ll have to write the correct path and naming convention in Pigment.
Here are some examples of file names for a rev.csv with the ISO-8601 compatible date formats located at the start.
| ISO-8601 compatible date formats | example of file names {{date}}-rev.csv naming convention |
|---|---|
yyyy-MM-dd | 2022-09-30-rev.csv |
yyyy-MM-ddThh:mm:ss | 2022-09-30T00:00:00-rev.csv |
yyyy-MM-ddThh:mm:ssZ | 2022-09-30T00:00:00Z-rev.csv |
When writing the file path in Pigment
Similar to the Fixed file, you must first define the path with the location(ex: folder1/folder2/ ) ,then you must define file naming convention with a date included in the file name. When entering the case sensitive path in Pigment, use {{date}} where the date is located.
It should look something like this folder1/folder2/file-{{date}}.csv
Here are some examples of file names and the results depending on what path you entered.
If the bucket contains the following files:
- /myfolderA/2022-09-15-salaries.csv
- /myfolderA/2022-10-15-salaries.csv
- /myfolderA/2022-10-16-revenue.csv
- /myfolderB/2022-11-15-salaries.csv
Here are the results I would get depending on my file path :
- path "myfolderA/{{date}}-salaries.csv" will import
/myfolderA/2022-10-15-salaries.csvbecause this is the most recent file in myfolderA. - path "myfolderB/{{date}}-salaries.csv" will import
/myfolderB/2022-11-15-salaries.csvbecause this is the most recent file in myfolderB. - path "/{{date}}-salaries.csv" will import nothing because its missing the folder location.
In order to get the Start Import button to be clickable, you must have {{date}} and .csv and in the pattern.
Example
Here is an example from a an S3 folder. Take a look at the naming convention and see if you can input the correct
Click Show Content below to see the correct path needed to pull the most recent file.
folder/Country-{{date}}.csv is the correct file path and naming convention.
It is important to match case and to include {{date}} where the date is located.
Click Show Content below to see which file would be uploaded
Country-2022-07-26TO81417.csv