How to Automate FTP transfers in Linux Shell Scripting

Whether you are responsible for uploading files regularly to a remote web server, or syncing configuration files among a number of machines, most of the time you’ll find yourself using the FTP protocol.

But what about automating the process?

Could you simply add a shell script to a Crontab job and let it take care of all the FTP login process, file transfers, and essentially has the job done for you while you’re away from your desk?

If this sounds interesting, read along. Here, you will learn how to automate FTP transfers in Linux Shell Scripting.

What is FTP?

FTP stands for File Transfer Protocol. The protocol specifications were written by Abhay Bhushan. In April 1971, the Request for Comments (RFC) 114 was published to describe the newly developed protocol.

How FTP Works?

Using interactive mode
OK, so before we delve in, let’s have a quick refresher on FTP commands and usage for those who are accustomed to using GUI applications like FileZilla. In its simplest form, you can connect to a remote server using FTP using one of the following ways:

1. Type FTP then type open hostname, where hostname is the IP address or the hostname of the remote server you’re connecting to. You’ll be asked to enter your username and password. An example of this is in the following figure:

2. Type ftp hostname to avoid having to type open after starting the FTP session

3. Type ftp username@hostname where username is the username used to connect to the server. This will let you jump right into the password prompt.

Working with files

So now we’re connected to the remote server, what’s next? The following is a list of most common commands you’ll need in your FTP session:

Tip: You can always execute a command on your local machine while in an FTP session with the remote server by prefixing it with an exclamation mark (!). Example: !ls will list the files and directories on your local machine’s current path.

Automating the process

So now you have a basic idea about how to connect to a remote server using FTP and execute basic commands. Let’s see how you can automate the process.

What are we trying to do?

The rest of the article describes how to automate logging into a remote server using FTP and working with files. To accomplish this, you’ll log in to the FTP server in a slightly different manner than the previous methods. We are trying to automate the following example:

The FTP command switches

In the previous example, we used a long way to connect to FTP. First, we specified the -n switch to disable the auto-login feature (FTP expects the login user to be the current user), we manually specified the hostname using the open command, then we specified the username and password using the user command. A couple of additional important switches can be used in this task:
-i: Turns off interactive prompting during multiple file transfers. (The normal FTP behavior is to ask you before uploading or downloading files if they are multiple, a behavior you definitely don’t want in a background job)
-v: for verbose mode. This is not strictly needed, but it’s good for debugging purposes and troubleshooting.

The heredocs

We are going to use a bash feature called “here documents” or heredocs. The main use of this feature is to enable entering multiple lines of input into an interactive prompt. It starts with any string preceded by << (Example <<EOF), then you add the commands line by line, finally you end the block by adding the same string you used, but on a new line and without the << (Example EOF).
To give you a more illustrated idea about heredocs, consider this example:
The echo command may be fine if you want to output a single line of text to the screen. But what if you want to output a number of lines? You can use cat with heredocs as follows:

cat <<EOT
This is line 1
This is line 2
Thanks
EOT

And the output of this would be:

This is line 1
This is line 2
Thanks

The idea here is feeding the command with a block of lines and forcing it to treat them as a single argument. So how can this be useful to our FTP automation task? It’s the same concept: FTP is expecting a command as an argument (like user username password), that will be followed by a number of get, put and other FTP commands.

Automate FTP transfers script

Ok, now you have all the tools you need to automate an FTP job under your belt, let’s start writing the script.

#!/bin/bash
HOST=ftp.example.com
USER=ftpuser
PASSWORD=P@ssw0rd

ftp -inv $HOST <<EOF
user $USER $PASSWORD
cd /path/to/file
mput *.html
bye
EOF

Pretty simple right?
1. You start by specifying that you are going to use bash as your shell language.
2. Assign the hostname to variable HOST.
3. Assign the username to variable USER.
4. Assign the password to variable PASSWORD.
5. Issue the FTP command in non-interactive, verbose mode, instructing the program not to use auto-login. Also, you start a heredocs block to start feeding the FTP program your command block.
6. Enter the user and password.
7. Navigate to the destination directory.
8. Upload all files with extension .html to the destination directory
9. End the FTP session
10. Mark the end of the heredocs
Now save this script as ftpScript.sh for example, and add the appropriate permissions by issuing chmod 700 ftpScript.sh. Put this script in a crontab job and you’re done.

If you are not familiar with Shell Scripting, You can have a look at this blog to get the fundamentals of Shell Scripting clear. You can use Shell Scripting for a variety of things such as generating random numbers and generate random data or passwords.

But do you see any problem in this code?

Yes, you guessed right: the password is stored in clear text! Of course, you can set very strict permissions on the script so that only you have read access. But imagine you want to change the files you want to upload/download from .html to .php, and while doing so, your new colleague is sitting beside you to learn FTP automation. In such a case, your password will be accidentally visible to your colleague and to anyone who happens to have your screen in his/her sight.

A not very secure way to encrypt your password
Ok, let’s agree first that the following method is NOT by any means a cryptographically secure way of string encryption. This is just a way to “obfuscate” your password so that it is not visible in cleartext.

Having said that, let’s get to know the base64 command. Basically, this command converts the string you supply to and from base64 format. So assuming that you have P@ssw0rd as your FTP password, you can encrypt it as follows:

echo P@ssw0rd | base64
UEBzc3cwcmQK

As you can see the base64 representation of your password is UEBzc3cwcmQK. So how can we decrypt it to get back our text?

echo UEBzc3cwcmQK | base64 --decode
P@ssw0rd

Now let’s make a slight modification to our script:

#!/bin/bash
HOST=ftp.example.com
USER=ftpuser
PASSWORD=`echo UEBzc3cwcmQK | base64 --decode`

ftp -inv $HOST <<EOF
user $USER $PASSWORD
cd /path/to/file
mput *.html
bye
EOF

The changed part here is line 4. We used the base64 decryption method to get the password string and assign it to the password variable. To make it work, we enclosed the command in “ to instruct bash to execute the command and not treat it as literal text.

This way, if somebody had an accidental look at your file, the base64-encoded password will be much more difficult to memorize than the clear text one.

Conclusion
In this article, we had a quick overview of the FTP command and how to use it in the bash shell instead of using GUI applications. Then we explained how to use command-line switches and heredocs to automate FTP file transfer operations. Finally, we showed a very basic way to encrypt clear text passwords. Although not considered “secure enough” by any security standard, it is much better than letting the password stay in cleartext.

Meanwhile, If you are a beginner and want to gain in-depth knowledge about the subject, try the “Linux Shell Programming for Beginners” course. The course comes with 9.5 hours of video that covers 12 sections such as command lines and tips, customizing your shell, conditions and loops, command-line options, and much more.

Also, Try out the Learn Linux from Scratch online tutorial for FREE! It comes with 4 hours of video that covers 12 sections including an introduction to Linux Systems, installation, hardware, file management and much more.

I hope I saved somebody’s time and effort through this post. Thanks for reading and see you in later posts.

Read More:

• Linux Over The Years: An Open-Source Revolution!
• 10 Best IDEs For Linux In 2020!