Linux: A cool tar and curl one-liner with many capabilities

I had the following requirement! i needed to compress all files in a directory and then transfer them over scp to a host, plus that i needed to rate limit the transfer in order not to saturate the network!

How it works

I used the following one liner

tar czf - /path/to/files | curl --limit-rate 1M -u username:sftp_password --upload-file - sftp://remote-server/path/to/destination/

Lets break down the command

This part compresses files in mentioned directory

tar czf - /path/to/files

Next part accepts input from tar as defined by the | character, the output is passed to curl which uploads over sftp protocol data to a remote host

| curl --limit-rate 1M -u username:sftp_password --upload-file - sftp://remote-server/path/to/destination/

• - -limit-rate 1M: Limit the transfer rate to 1MB per second
• -u usernane:sftp_passwords ofc you can use ssh keys instead!
• - — upload-file — : Upload whatever comes from stdin “-”
• sftp://remote-server : Use sftp protocol and connect to remote-server

Conclusion

One cool thing about curl is that supports multiple protocols! this means that with minimal changes you can compress and transfer files over http!, i hope you enjoyed this article as much i enjoyed writing this article!