Reading several files at the same time

In the main exercise, you used an Input step to read a single file. But suppose you have several files, all with the same structure. Reading those files at the same time is not much different from what you did.

For this quick tutorial, we will take as a source several files with the same structure as sales_data.csv, but separated by region. This is the list of files which you will find in the SAMPLEFILES folder:

sales_data_APAC.csv
sales_data_EMEA.csv
sales_data_Japan.csv

You have two options here. The first way for reading several files at the same time would be to provide the detailed list of filenames:

1. Open the Transformation that reads the sales_data.csv file.
2. Fill the grid with every filename, as shown next:

Reading several files at the same time

1. Select the Additional output fields tab. In the Short filename field textbox, type file_name.
2. Run a preview. You should see something like this:

Previewing the content of several files

As you can see, the preview shows data coming from all the specified files. We added the name of the file as an extra field, just for you to see where each order comes from.

When the names of the files in your list follow a pattern, then instead of a detailed list of filenames, you could use regular expressions. This option is also useful when you don't know the exact names of the files beforehand:

1. Open the Transformation that reads several files and double-click on the Input step.
2. Delete the lines with the names of the files.

1. In the first row of the grid, under the File/Directory column, type the full path of the input folder, for example, D:/SAMPLEFILES/. Under the Wildcard (RegExp) column, type sales_data_.+\.csv. Click on the Show filename(s)... button. You will see the list of files that match the expression:

Files that match a regular expression

1. Close the tiny window and click on Preview rows to confirm that the file names match the expression you typed.