Linux: parallel SFTP transfers using a simple one-liner.
Recently i came across a legacy bash script that worked fine; its job is to find files in a directory and sftp them. The script was quite…
Recently i came across a legacy bash script that worked fine; its job is to find files in a directory and sftp them. The script was quite long for such a simple job, about 20 lines and i was wondering if i could make it shorter, also i was wondering how i can improve the script since it transferred the files one by one despite i had enough bandwidth.
The steps of the scripts are:
- find files of this pattern in a directorydata_2022_09_19_partner1.csv
2. Then sftp these files that in their filename have a date stamp of YYYY_MM_DD minus one day from the current execution date.
One liner creation
Find the files with specified pattern
I will try to make this long script to an one-liner, lets first find files that match the pattern without the date.$ find . -name 'data_*_partner?.csv'
./data_2022_09_19_partner1.csv
./data_2022_09_18_partner3.csv
./data_2022_09_18_partner2.csv
./data_2022_09_16_partner1.csv
./data_2022_09_17_partner1.csv
./data_2022_09_16_partner3.csv
./data_2022_09_17_partner2.csv
./data_2022_09_16_partner2.csv
./data_2022_09_18_partner1.csv
./data_2022_09_19_partner3.csv
./data_2022_09_19_partner2.csv
./data_2022_09_17_partner3.csv
Filter the files with a date stamp of current date -1 day
That was easy! now we have to filter out only files with yesterday timestamp in the filename, at this time of execution the date is 2022–09–20 so we need to find files with a pattern of data_2022_09_19_partner?.csv to do this we pass the find output to grep which uses date to generate a matching string of current date — 1 day.$ find . -name 'data_*_partner?.csv' | grep "$(date +"%Y_%m_%d" -d "1 day ago")"
./data_2022_09_19_partner1.csv
./data_2022_09_19_partner3.csv
./data_2022_09_19_partner2.csv
Create the sftp command to use with xargs
We almost done! next step is to do the actual sftp transfer! now we will use xargs; xargs allows to build commands from stdin input, we can even execute those commands in parallel.
In this example we will use xargs with the -I {} argument, -I allows to alias input from stdin with {} and use it to build your command you want to execute, in our case we want to create a valid sftp command.
The sftp command i wanted to create was, where {} will replaced from xargs with the passed filename.echo 'put {} /EXPORTS/PARTNERS/'
The xargs statement to accomplish this isxargs -I {} bash -c "echo 'put {} /EXPORTS/PARTNERS/'"
If you run the one-liner it will create the following output, which will be commands passed to sftp!$ find . -name 'data_*_partner?.csv' | grep "$(date +"%Y_%m_%d" -d "1 day ago")" | xargs -I {} bash -c "echo 'put {} /EXPORTS/PARTNERS/'"put ./data_2022_09_19_partner1.csv /EXPORTS/PARTNERS/
put ./data_2022_09_19_partner3.csv /EXPORTS/PARTNERS/
put ./data_2022_09_19_partner2.csv /EXPORTS/PARTNERS/
Execute the sftp command
And finally our last step! to execute the sftp commands, the sftp command syntax in my case where i use ssh keys issftp -oPort=22 sftp_user@sftpsrv01
Of course in your case you should change port, username and sftp server!,
Gluing all together would create this one-liner, maybe a bit difficult to read but still better than a long script!find . -name 'data_*_partner?.csv' | grep "$(date +"%Y_%m_%d" -d "1 day ago")" | xargs -I {} bash -c "echo 'put {} /EXPORTS/PARTNERS/'" | sftp -oPort=22 sftp_user@sftpsrv01
Making transfers parallel
But we are not ready yet! as i promised i will show you how control the number of parallel transfers! to do this we need to use the -P parameter in xargs, -P4 instructs xargs to execute 4 commands in parallel!find . -name 'data_*_partner?.csv' | grep "$(date +"%Y_%m_%d" -d "1 day ago")" | xargs -P4 -I {} bash -c "echo 'put {} /EXPORTS/PARTNERS/'" | sftp -oPort=22 sftp_user@sftpsrv01
If everything was done correctly you should see your sftp transfers starting! i hope this cute one-liner help you avoid creating long scripts and make Linux fun again!