How To Automate Backups with DigitalOcean Spaces

Introduction

Backing up crucial information is an part that is essential of any computer infrastructure. Although everyone’s needs are different when it comes to performing backups, maintaining backup data at an location that is offsite a good training.

The means of giving copies of information to an offsite location was once a significant challenge that is logistical. But with the advent of cloud-based storage services like Crashplan and Dropbox, as well as the development of object storage solutions like DigitalOcean Spaces, it has now become a much simpler task. Despite this, remembering to back up files and taking the time to still upload them can be a roadblock for many.

This is excatly why individuals go for different tools to execute routine, automated backups of these data that are important. In this tutorial, we will build a script around the s3cmd command line tool which can be used to quickly upload data to DigitalOcean Spaces. We will then use crontab to invoke the backups regularly script and upload files to your room.

Prerequisites

For this guide, you’ll need:

Having some knowledge of shell scripting while the cron task scheduler may be helpful. For many guidance and context that is additional give consideration to reading “An Introduction to Shell Scripting” and “How To Schedule Routine Tasks With Cron and Anacron on a VPS.”

With our prerequisites in position, we’re willing to start our automation that is backup procedure.

Building our Backups Script

There are many tools and this can be regularly upload files that are backup an object storage service automatically and at regular intervals. However, these can be difficult to configure and may not offer much in the real method of freedom. Utilizing a straightforward shell script may be an infinitely more elegant and approach that is straightforward automating item storage space backups.

For this guide, we’ll compose a bash that is basic that will produce a back-up of a file or directory making use of tar. The script will likely then upload that back-up to areas utilising the s3cmd demand line energy.

To begin, log into the Droplet and navigate to your residence folder:

Once in your home folder, we’ll usage nano to generate a clear file in which we are able to compose within our script:

We are now actually willing to start composing our backups script within the text editor. In order, section by section.( as we build the script, we will explain each part of it*********)

Initiating Our Script

At this aspect, bkupscript.sh is an text file that is empty. In order for our computer to invoke our executable file as commands, we need to begin a hashbang to our script. A hashbang is an interpreter directive that allows scripts or data become run as commands.

In our instance, the hashbang can look similar to this:

bkupscript.sh

#!/bin/bash

By including this near the top of the script, we're telling the shell to perform the commands that are file’s bash.

Declaring Variables

Next, we’ll inform our script the factors it'll need to understand to work properly. We are able to include these straight underneath the hashbang near the top of the written text file:

bkupscript.sh

...
DATETIME=`date +%y%m%d-%H_percentM_percentS`
SRC=$1
DST=$2
GIVENNAME=$3

Let’s proceed through that which we are assigning each one of these factors to:

  • DATETIME: This adjustable holds a timestamp to affix toward ensuing filename so every file copied to your area has a name that is unique. This timestamp is made by invoking the date demand and formatting the production to exhibit the very last two digits of the season (%y), both digits for the thirty days (%m), both digits for the time (%d), the hour (percentH), the moments (percentM) while the moments (percentS).
  • SRC: here is the source course the file or folder we should backup. The $1 shows we are using this value from very first parameter passed away toward script.
  • DST: This adjustable represents the file’s destination. This is the name of the Space to which we are uploading the backup in our case. This name will come from the parameter that is second toward script as suggested by the $2.
  • GIVENNAME: This adjustable hosts the name that is user-chosen the destination file. The filename that is resulting focus on GIVENNAME and can have the DATETIME concatenated into it. This title arises from the parameter that is third toward script ($3).

Providing Some Assistance

whenever composing a script, it is beneficial to then add recommendations or advice that is general will help users troubleshooting if their tries to make use of it fail.

For our backups script, we'll include a function called showhelp() below our factors. This may print a few communications to greatly help users troubleshoot whenever the script fails. Whenever including a function in bash, our syntax shall seem like this :

bkupscript.sh

...
showhelp(){

}

This function provides assistance communications by echoing a few use directions in the display screen. Each instruction should really be presented as a string enclosed with dual quotes. You’ll notice within our instance below that a number of the strings have actually t or n written at their end or beginning. They are escape figures which offer certain directions for the way the sequence should come in the script’s production:

  • t shows a tab room
  • n shows a line break

Feel absolve to include any use details that could be ideal for you among the curly brackets (keep in mind to precede any sequence with echo). For demonstration purposes, we'll include the ( that is following*********)

bkupscript.sh

echo "nn############################################"
echo "# bkupscript.sh                            #"
echo "############################################"
echo "nThis script will backup files/folders into an individual compressed file and can keep it in the present folder."
echo "to work, this script requires these three parameters within the detailed purchase: "
echo "t- the path that is full the folder or file you intend to backup."
Echo "t- the true title for the area in which you intend to keep the back-up at."
Echo "t- the true title the back-up file (timestamp will likely to be put into the start of the filename)n"
echo "Example: sh bckupscript.sh ./testdir testSpace backupdatan"<^>

The last showhelp function should look something similar to this:

bkupscript.sh

...
showhelp(
        echo "nn############################################"
        echo "# bkupscript.sh                            #"
        echo "############################################"
        echo "nThis script will backup files/folders into an individual compressed file and can keep it in the present folder."
        echo "to work, this script requires these three parameters within the detailed purchase: "
        echo "t- the path that is full the folder or file you intend to backup."
        Echo "t- the true title for the area in which you intend to keep the back-up at."
        Echo "t- the true title the back-up file (timestamp will likely to be put into the start of the filename)n"
        echo "Example: sh bckupscript.sh ./testdir testSpace backupdatan"
}

With our assistance text in position, we are able to proceed to gathering the files we wish to back-up within our room.

Gathering Data

Before our script can move any such thing to your area, it first must gather the best files and combine them into a package that is single us to upload. We can accomplish this by using the tar utility and a statement that is conditional. Because we’re making use of tar to generate an archive file (often referred to as a “zip” file), we shall phone this function tarandzip().

Let’s begin by declaring the event and including another echo demand to allow users realize that the script has started files that are gathering*********)

bkupscript.sh

...
tarandzip(){
    echo "n##### Gathering files #####n"
}

Below the echo demand, we are able to include a tar demand that will perform some work of gathering and compressing the files into a output that is single.

bkupscript.sh

tarandzip(){
    echo "n##### Gathering files #####n"
    tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC
}

You’ll observe that this tar demand is invoked with a few choices and factors:

  • c: This banner informs tar to compress the production file.
  • z: This instructs tar to compress the file making use of gzip.
  • v: This signifies the verbose choice, which instructs tar to exhibit more info within the production.
  • f: This banner instructs tar to truly save the file with all the filename suggested next.
  • $GIVENNAME-$DATETIME.tar.gz: The script calls these factors which we declared at the start to produce the file name that is new. It does this by combining the**********)$ that is(***********) and $DATETIME factors and including the .tar.gz expansion toward end to make the file that is new.
  • $SRC: This adjustable represents the source files or files which we're instructing tar to backup.

This function should now have the ability to do that which we want to buy to, but we are able to include some more echo calls to offer the consumer some information that is extra the way the script is working. This is often carried out by including several conditional statements, similar to this:

bkupscript.sh

    if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
        echo "n##### Done gathering files #####letter"
        return 0
    else
        echo "n##### Failed to gather files #####letter"
        return 1
    fi

As soon as the if clause is invoked, it's going to perform the tar demand and wait for outcome. In the event that total outcome the demand is good (meaning, it went effectively), the lines between then and else will likely to be performed. They are:

  • Echoing an email your script has effectively finished the tar procedure
  • Returning one rule of 0 so the part of the rule which invokes this function understands that every thing worked fine.

The else part of this function will simply be performed in the event that tar demand discovers one while performing. Theelse branch of the clause will:

in this case

  • Echo an email indicating your tar demand failed
  • Return one rule of 1, showing that one thing went incorrect

Finally, we end the if/then/else clause with a fi, that bash language means the if clause is finished.

The finished tarandzip() function can look similar to this:

bkupscript.sh

tarandzip(){
    echo "n##### Gathering files #####n"
    if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
        echo "n##### Done gathering files #####n"
        return 0
    else
        echo "n##### Failed to gather files #####n"
        return 1
    fi
}

With our tarandzip() function in position, we're willing to set the script as much as go our backups.

Transferring Data to Object Space

At this aspect, we are able to get our back-up script to move a file to your area utilizing the s3cmd demand. Much like tarandzip, we are able to additionally echo some strings and use an if/then/else declaration to help keep users up to date with the way the script is being employed as it really is operating.

First we'll declare our function. Let’s once again keep this simple and easy title it movetoSpace():

bkupscript.sh

...
movetoSpace(){

}

Now we are able to utilize s3cmd while the factors we declared early in the day to construct the demand that will push our back-up files to your area:

bkupscript.sh

movetoSpace(){
    ~/s3cmd-2.0.1/s3cmd placed $GIVENNAME-$DATETIME.tar.gz s3://$DST
}

Here’s exactly what each section of this demand means:

  • ~/s3cmd-2.0.1/s3cmd: This invokes s3cmd, a demand line device employed for handling item storage space buckets.
  • put: this is certainly a demand employed by s3cmd for uploading information into a bucket.
  • $GIVENNAME-$DATETIME.tar.gz: This is actually the true name of the backup that will be uploaded to our Space. It consists of the fourth and first variables we declared, followed by .tar.gz, and is created by the**********)tarandzip( that is( function from early in the day.
  • s3://$DST;: here is the location in which we should upload the file. s3:// is a schema that is URI-like especially to explain item storage space places on line, while $DST; may be the 3rd variable we declared early in the day.

We are in possession of a function which could upload our files that are archived our Space. However, it doesn’t notify the user about its status. Let’s change this by echoing a string before the command to let the user know whether it was successful that it has been started, and after the function is complete, to let us know.

Let’s begin by notifying the consumer your procedure has started:

bkupscript.sh

movetoSpace(){
    echo “n##### MOVING TO SPACE #####n”
    ~/s3cmd-2.0.1/s3cmd place $GIVENNAME-$DATETIME.tar.gz s3://$DST
}

Because the demand will likely to be either effective or unsuccessful (meaning, it's going to either upload the files to your area or it won’t), we are able to allow users understand whether it worked by echoing 1 of 2 strings held in an if/then/else declaration, similar to this:

bkupscript.sh

...
if ~/s3cmd-2.0.1/s3cmd place $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
    echo "n##### Done moving files to s3://"$DST" #####letter"
    return 0
else
    echo "n##### Failed to move files to the Space #####letter"
    return 1
fi

This conditional declaration informs bash “If our s3cmd demand functions properly, then allow individual realize that the script is performed going files to your area. Otherwise, allow individual realize that the method failed.”

If the s3cmd procedure completes effectively then function images an email toward display screen (the initial echo sequence within the then declaration) showing therefore, and comes back a value of 0, which notifies the calling function your procedure happens to be finished. A1 so the rest of the script is aware that an error occurred.( if the process fails, the then clause simply prints the error message (the second echo string), and returns*********)

Altogether, the**********)movetoSpace( that is( function should seem like this:

bkupscript.sh

movetoSpace(){
    echo "n##### MOVING TO SPACE #####n"
    if ~/s3cmd-2.0.1/s3cmd place $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
        echo "n##### Done moving files to s3://"$DST" #####n"
        return 0
    else
        echo "n##### Failed to move files to the Space #####n"
        return 1
    fi
}

With the movetoSpace() function written, we are able to proceed to making sure the script is initiated to phone functions within the anticipated purchase through making use of conditional statements for movement control.

Setting Up Flow Control

(we have not provided an order for the script to complete those functions********)Though we have set up our script with functions. At this true point, we are able to introduce a calling function that will inform all of those other script just how when to perform others functions we’ve written.

Assuming every thing happens to be configured properly, once we operate the script it will see the input demand, assign the values from this every single adjustable, perform the tarandzip() function, and follow that with all the movetoSpace() function. If the script fail between some of those true points, it will print the production of our showhelp() function to greatly help users with troubleshooting. We are able to purchase this and get mistakes by the addition of a few if/then/else statements in the bottom for the file:

bkupscript.sh

...
if [ ! -z "$GIVENNAME" ]; then
    if tarandzip; then
        movetoSpace
    else
        showhelp
    fi
else
    showhelp
fi

The very first if declaration within the area above checks your 3rd variable passed away just isn't empty. It can therefore within the following method:

  • [ ]: the brackets that are square that what lies between them is a test. The test is for a specific variable to not be empty.( in this case****************************)
  • !: in this instance, this icon means not.
  • -z: this program shows an empty sequence. Therefore, with the !, we're seeking not an string that is empty********************************).
  • $GIVENNAME: we're showing right here your sequence we don’t wish to be empty may be the value assigned toward adjustable $GIVENNAME. The reason why we decided this process is really because this adjustable is assigned the worth passed away by the parameter that is third calling the script from the command line. So it will assign an empty string and this test will fail.( if we pass fewer than 3 parameters to the script, the code will not have a third parameter to assign the value to $GIVENNAME,****************************)

Assuming that test that is first successful, it will then move on to the next if statement and so on. Thethen clause will call the showhelp function and help text will be displayed in the output if any of the if statements return an error. Really, exactly what this does could it be glues together most of the past functions we’ve written down and provides bash the details it requires to perform them within the order that is correct

Our script has become complete! It is possible to validate that the script seems like the script that is complete built-in the area below.

The Complete Script

The finished backups script we created should seem like this:

bkupscript.sh

#!/bin/bash
DATETIME=`date +%y%m%d-%H_percentM_percentS`
SRC=$1
DST=$2
GIVENNAME=$3
showhelp(){
        echo "nn############################################"
        echo "# bkupscript.sh                            #"
        echo "############################################"
        echo "nThis script will backup files/folders into an individual compressed file and can keep it in the present folder."
        echo "to work, this script requires these three parameters within the detailed purchase: "
        echo "t- the path that is full the folder or file you intend to backup."
        Echo "t- the true title for the area in which you intend to keep the back-up at (perhaps not the url, simply the title)."
        Echo "t- the true title the back-up file (timestamp will likely to be put into the start of the filename)n"
        echo "Example: sh bkupscript.sh ./testdir testSpace backupdatan"
}
tarandzip(){
    echo "n##### Gathering files #####n"
    if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
        echo "n##### Done gathering files #####n"
        return 0
    else
        echo "n##### Failed to gather files #####n"
        return 1
    fi
}
movetoSpace(){
    echo "n##### MOVING TO SPACE #####n"
    if ~/s3cmd-2.0.1/s3cmd place $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
        echo "n##### Done moving files to s3://"$DST" #####n"
        return 0
    else
        echo "n##### Failed to move files to the Space #####n"
        return 1
    fi
}
if [ ! -z "$GIVENNAME" ]; then
    if tarandzip; then
        movetoSpace
    else
        showhelp
    fi
else
    showhelp
fi

Once you've got confirmed your script, make sure you conserve and shut the file (CTRL-x, y, then ENTER) before leaving nano.

Testing the Script

Now that we’re completed with building the script, we are able to proceed to testing it down. Not just will this inform us whether we’ve written the script properly, however it shall additionally provide us with an opportunity to exercise utilising the script to discover it doing his thing.

whenever trying out a script similar to this, it is frequently a idea that is good use dummy files. Even with some unimportant files though we know it isn’t capable of destroying or removing data, it’s smart to play it safe by testing it. We’ll first create a directory utilising the mkdir demand:

Next, we’ll generate two files that are empty this directory making use of touch:

  • sudo touch backupthis/file1.txt
  • sudo touch backupthis/file2.txt

We is now able to test the script by uploading the backupthis directory and its own articles to your area. This is actually the structure we shall need certainly to use within purchase to invoke the script:

  • sh bkupscript.sh ./backupthis name_of_your_space testrun

Note as the movetoSpace() function immediately prepends s3:// toward location adjustable (in other words., the title of one's room), this variable should merely end up being the title of one's room and never its complete URL. As an example, in the event your Space’s Address is “https://example-space-name.nyc3.digitaloceanspaces.com”, you'll compose the test demand similar to this:

  • sh bkupscript.sh ./backupthis example-space-name testrun

The demand above will set the script in movement and you ought to see production similar to this:

Output

##### Gathering files ##### ./backupthis/ ./backupthis/file1.txt ./backupthis/file2.txt ##### Done gathering files ##### ##### MOVING TO SPACE ##### upload: 'testrun-180119-15_09_36.tar.gz' -> 's3://name_of_your_space /testrun-180119-15_09_36.tar.gz' [1 of 1] 162 of 162 100percent in 8s 19.81 B/s done ##### Done moving files to s3://name_of_your_space #####

(that it matches our example********)If you encounter any errors, please review your script to ensure. Also, make sure that your installation of s3cmd has been properly configured and that both the access key and the key that is secret making use of are proper.

Automating Backups with Crontab

After effectively testing the backups script, we are able to put up a cron task that will make use of the script to execute backups that are regular our Space. Every minute.( for the purpose of this tutorial, we’ll set it up to execute our backups script*********)

First, we must result in the script ( that is executable*********)

Now your script may be performed as a demand, we are able to modify the crontab file to perform the script every moment:

The very first time it will ask you to select an editor from a list:( that you run crontab -e,*********)

no crontab for root - making use of a clear one
Pick an editor.  To improve later on, run 'select-editor'.
  1. /bin/ed
  2. /bin/nano        <---- simplest
  3. /usr/bin/vim.basic
  4. /usr/bin/vim.tiny
Select 1-4 [2]: 

You can find the standard nano, or any other text editor of one's option.

Once in crontab, we'll include the line that is following the base of the values which are currently here:

/tmp/crontab.example/crontab

* * * * * ~/bkupscript.sh ~/backupthis nameofyourspace cronupload

To save the modifications, press CTRL-x, then y, then ENTER.

After a moment or more, you'll see a file that is new your Space’s dashboard!

If you leave the cron task operating without making any modifications, you should have a brand new file copied towards area every moment. When you’ve verified that cron is operating effectively, please feel free to reconfigure crontab to backup your files at your desired period.

You are in possession of a script that occasionally compresses and vessels your backups to a DigitalOcean area!

Conclusion

In this guide, we now have reviewed how exactly to produce frequently planned offsite backups of one's files that are important a bash script, crontab, and DigitalOcean Spaces. Although the script presented in this tutorial is only intended for demonstration purposes, it can be used as a foundation to build a version that is production-ready could later on be incorporated with a CI/CD solution like Jenkins, Drone, or Travis CI.

(you can do so by reading the following tutorials:

If you want to learn more about CI/CD tools,*********)

LEAVE A REPLY

Please enter your comment!
Please enter your name here