How To Automate the Scaling of Your Web Application on DigitalOcean Ubuntu 16.04 Droplets

Introduction

In this guide we shall show utilizing the DigitalOcean API to horizontally measure your host setup DOProxy that is using a Ruby script that, as soon as configured, provides a demand line screen to measure your HTTP application host tier up or down.

DOProxy had been written especially for this tutorial to offer an easy method for producing and deleting application host Droplets utilizing the DigitalOcean API in addition to handling their account to an load balancer that is HAProxy. This scaling that is basic permits users to get into the application through HAProxy host that may, consequently, forward them towards the back-end application servers in lots balanced fashion.

DOProxy executes three main functions:

  • Creates Droplets and adds them towards the load balancer
  • Deletes Droplets and eliminates them through the load balancer
  • Maintains a listing regarding the Droplets until they are deleted( that it has created********)

DOProxy create

Note: the purpose that is primary of tutorial is to teach the minimum required concepts that are necessary to programmatically scale your DigitalOcean server architecture through the API. You should not run DOProxy in a production environment it performs only very basic error checking since it was not designed with resiliency in mind and. With that said, getting acquainted this script is a way that is great enable you to get started with researching horizontal scaling through DigitalOcean API.

Prerequisites

This guide makes use of the technologies that are following you might want to learn about before continuing:

Because DOProxy is written in Ruby, understanding of Ruby are useful. To get a lot more of a familiarity with Ruby, it is possible to read our show on the best way to Code in Ruby. We provide some pseudocode to explain the gist of the DOProxy code if you’re less familiar with Ruby. To simplify our telephone calls towards the API, we have been utilizing DropletKit which will be the state DigitalOcean Ruby wrapper.

(on a server****)Before we get into the details of how DOProxy works we will install and use it.

Let’s install DOProxy on an Ubuntu 16.04 Droplet now.

Install DOProxy

First, create an Ubuntu 16.04 Droplet into the region that is NYC3 the region DOProxy uses by default. You will need to configure the region variable in the doproxy.yml( if you wish to use another region,********************) file after setting up DOProxy. This Droplet will run the load that is HAProxy and the DOProxy scaling script, so choose a size that you think will be adequate for your desired scale potential. Because this tutorial is a basic demonstration of scaling with no traffic that is real, the 512MB dimensions are most likely sufficient.

For along this document, we shall relate to this Droplet due to the fact DOProxy host.

Next, log to the host and proceed with the Installation and Configuration (including doproxy config and Userdata) parts into the GitHub that is DOProxy repository to install DOProxy on this server. Be sure to replace the YOUR_DO_API_TOKEN and YOUR_SSH_KEY_FINGERPRINT values in the configuration that is DOproxy and/or script won't work.

Now you have actually DOProxy and HAProxy set up in your host, let us try to measure the surroundings.

Run DOProxy

Log into the server that is DOProxy as************)root and progress to the directory in which you cloned DOProxy.

Run DOProxy with no arguments:

This should print from available commands:

Output

Commands: doproxy.rb printing # Print backend Droplets in stock file doproxy.rb create # Create a backend that is new and reload doproxy.rb delete <LINE_NUMBER> # Delete a Droplet and reload Doproxy.rb reload # Generate HAProxy reload and config HAProxy doproxy.rb generate # generate config that is HAProxy on stock

At this aspect, DOProxy has not yet produced any Droplets. Let us produce some getting our HTTP solution on the web, and scale up.

Scale Up (Make)

Run the create demand generate the Droplet that is first that handled by DOProxy:

This takes time before time for the prompt (since the script produces a Droplet that is new via API and waits because of it on top of that up). We will speak about the way the API call is manufactured whenever we have the pseudocode.

Once the script is performed, you need to see a message that is success offers the Droplet ID:

Output

Success: 4202645 produced and put into backend.

It is recommended since it is possible that the userdata script may not have yet run and hence, the HAProxy may not have started passing traffic.( that you wait a couple minutes after the prompt has returned before proceeding with the next steps*****)

Once you're prepared to carry on, check out your DOProxy host's general public internet protocol address in an internet browser. You ought to see a typical page that lists your brand-new Droplet's hostname, id, and public internet protocol address.

We'll make use of DOProxy generate two more Droplets, for an overall total of three. Please feel free generate more if you would like:

  • ruby doproxy.rb make
  • ruby doproxy.rb make

Now check out your DOProxy host's general public internet protocol address in an internet browser once more. That you’ve created if you refresh the page, you will notice that the information on the page will change while cycling through the Droplets. This is because they are all load that is being by HAProxy which included each Droplet to its setup whenever made up of DOProxy.

If someone happens to check into the DigitalOcean control interface, you will observe these droplets that are new be detailed here (combined with the remainder of one's Droplets):

Droplets in Control Panel

Let's simply take a better consider the Droplets which were produced by taking a look at DOProxy's stock.

DOProxy provides a print demand which will print away the Droplets which are element of its stock:

You should see production that appears something similar to this:

Output

0) auto-nginx-0 (pvt internet protocol address: 192.0.2.175, status: active, id: 4202645) 1) auto-nginx-1 (pvt internet protocol address: 192.0.2.176, status: active, id: 4205587) 2) auto-nginx-2 (pvt internet protocol address: 192.0.2.172, status: active, id: 4205675)

In the instance production, we come across information regarding the 3 Droplets that people created, including their hostnames, status, and Droplet IDs. The hostnames and IDs should match that which you saw within browser whenever you accessed the load that is HAProxy (via DOProxy's general public internet protocol address).

As you may possibly have noticed, DOProxy just printed information regarding Droplets so it created. This is certainly it creates.( because it maintains an inventory of the Droplets*****)

Check from articles regarding the inventory file now:

You should begin to see the ID of each and every Droplet, one per line. Every time a Droplet is done, its ID is saved within stock file.

(them.

As you may have guessed, DOProxy's print command iterates through the Droplet IDs in the inventory file and performs an API call to retrieve information about each of*****)

It must certanly be noted that keeping your host stock in one single file isn't the greatest solution it demonstrates a simple implementation that works— it can easily be corrupted or deleted — but. A key that is distributed shop, such as for example etcd, is a much better solution. You'll would also like to save lots of more than simply the Droplet ID into the stock (you want to look at certain Droplet information).( so you don't have to make API calls every time*****)

Scale Down (Delete)

DOProxy also offers a delete demand that allows you to delete Droplets within stock. The delete demand calls for you supply the line range the Droplet to delete (as presented by the print demand).

Before operating this demand you certainly will want to print probably your stock:

So, like, you would supply 2 as the line number:( if you want to delete the third Droplet,*****)

  • ruby doprorxy.rb delete 2

After a minute, you will see the verification message:

Output

Success: 4205675 deleted and taken from backend.

The delete demand deletes the Droplet through the API, eliminates it through the configuration that is HAProxy deletes it from the inventory. Feel free to verify that the Droplet was deleted by using the DOProxy print command or by checking the control panel that is digitalOcean. You shall additionally observe that it really is not the main load balancer.

HAProxy Configuration

The final little bit of DOProxy that people have not talked about yet is exactly how HAProxy is configured.

whenever you operate the create or delete DOProxy demand, the data for every Droplet into the stock is retrieved plus some regarding the info is accustomed alter an HAProxy setup file. Specifically, the Droplet ID and IP that is private are acclimatized to include each Droplet as a backend host.

Look during the final couple of lines regarding the generated haproxy.cfg file similar to this:

You should see something similar to this:

tail of haproxy*****)

( that is.cfg(****************************)    frontend www-http
       bind 203.0.113.43:80
       reqadd X-Forwarded-Proto: http
       default_backend www-backend

    backend www-backend

       host www-4202645 192.0.2.175:80 always check # id**************************************************************************************) that is:( hostname:auto-nginx-0
       host www-4205587 192.0.2.176:80 always check # id*************************************************************************************) that is:( hostname:auto-nginx-1

The frontend area should support the IP that is public of one's DOProxy host, and backend area should include lines that relate to all the Droplets which were produced.

Note: now, you might want to delete other Droplets which were made up of DOProxy (ruby doproxy.rb delete 0 until the servers have died).

Now that you have seen DOProxy's scaling doing his thing, let us simply take a better consider the rule.

DOProxy Code

In this area, we shall consider the files that are pertinent lines of rule that produce DOProxy work. Seeing exactly how DOProxy had been implemented should supply a few ideas of ways to make use of the API to handle and automate your host infrastructure.

(you can look at the files there or you can look at the files at the DOProxy repository (https://github.com/scanevari/doproxy).

Since you cloned the repository to your server,*****)

Important files:

  • doproxy.rb: DOProxy Ruby script. Offers the demand line logic and interface behind DOProxy
  • doproxy.yml: DOProxy setup file. Offers the API token and specifies Droplet make choices
  • haproxy.cfg.erb: HAProxy setup template. Always produce load balancer setup with appropriate backend host information
  • inventory: Droplet stock file. Stores IDs of created Droplets
  • user-data.yml: Userdata file. A file that is cloud-config will run using a fresh Droplet if it is produced

Let's plunge to the setup files very first.

doproxy.yml

These will be the crucial lines in doproxy.yml:

doproxy.yml

token: YOUR_DO_API_TOKEN
ssh_key_ids:
  - YOUR_SSH_KEY_FINGERPRINT
...
droplet_options:
  hostname_prefix: auto-nginx
  area: nyc3
  size: 1gb
  image: ubuntu-16-04-x64

The token home may be the the one that must hold your read and write API token.

The other lines specify the choices that'll be utilized whenever DOProxy produces a fresh Droplet. Like, installing the specified SSH key (by ID or fingerprint) and prefixing the hostnames with "auto-nginx".

More information regarding legitimate Droplet choices are located in the DigitalOcean API documents.

user-data.yml

This may be the file that'll be performed by cloud-init whenever each Droplet that is new is. This means that you can supply a file that is cloud-config a script to set up the application computer software for each brand new Droplet.

The test userdata file contains an easy bash script that installs Nginx on an Ubuntu host and replaces the Droplet’s hostname to its default configuration file, ID, and general public internet protocol address:

user-data.yml

#!/bin/bash

Apt-get update that is-y
Apt-get install nginx that is-y
export DROPLET_ID=$(curl http://169.254.169.254/metadata/v1/id)
export HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)
export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)
echo Droplet: $HOSTNAME, ID: $DROPLET_ID, internet protocol address: $PUBLIC_IPV4 > /var/www/html/index.html

Those curl commands are retrieving the data concerning the Droplet (hostname, ID, and internet protocol address) utilising the DigitalOcean Metadata solution.

In a manufacturing execution, this file would support the commands to, for example, install and configure the application. You may also make use of this to automate the integration of one's Droplets into the infrastructure that is overall by such things as immediately setting up SSH secrets and linking towards setup administration or monitoring tools.

To look over more about userdata, cloud-config, and metadata, always check these links out:

haproxy.cfg.erb

The HAProxy setup template contains the majority of the load balancer setup, with a few code that is ruby is changed with backend Droplet information.

We'll simply consider the Ruby area that yields the backend setup:

haproxy.cfg.erb

backend www-backend
   <percent @Droplets.each_with_index do |droplet, index| %>
   host www-<percent= droplet.id %> <percent= droplet.private_ip %>:80 always check # id:<percent= droplet.id %>, hostname:<percent= droplet.name -%>
   <percent end %>

This rule iterates through all the Droplets into the stock and adds a fresh backend that is HAProxy for every one of these (on the basis of the personal internet protocol address).

For instance, a line similar to this is produced for every Droplet:

haproxy.cfg

server www-4202645 192.0.2.175:80 always check # id:4202645, hostname:auto-nginx-0

Whenever a Droplet is done or deleted, DOProxy yields a fresh configuration that is HAProxy containing the modifications.

doproxy.rb

This Ruby script consists primarily of a class that is DOProxy offers the techniques that perform the Droplet creation and removal, stock administration and HAProxy setup generation.

If you recognize Ruby, take a look at file on GitHub: https://github.com/scanevari/doproxy/blob/master/doproxy.rb.

If that you do not comprehend Ruby, listed here is some simplified pseudocode which explains each technique. It may possibly be beneficial to compare this contrary to the real code that is ruby allow you to determine what is occurring.

def initialize

Executed everytime DOProxy runs with any arguments:( that are valid*****)

  • Read doproxy.yml setup file and acquire API token and Droplet choices.
def get_inventory

Retrieves information for every Droplet into the stock file. It should be performed before some of the other techniques are performed.

  • Read stock file (containing Droplet IDs)
  • For each Droplet ID, make use of the API to recover Droplet information
def print_inventory

This technique images Droplet information for every regarding the Droplet IDs into the stock file. It's invoked with all the doproxy.rb printing demand.

  • For each Droplet into the stock printing the hostname, personal internet protocol address, status, and ID
def create_server

whenever invoked through doproxy.rb create demand, this technique produces a Droplet that is new and it to the inventory file. Then, it calls reload_haproxy to re-generate the configuration that is HAProxy and reload force balancer.

  • Read the userdata file
  • Use the API generate a Droplet according to provided userdata and choices
  • Wait for Droplet status to be "active" — usage API to recover Droplet information every 15 moments until status modifications
  • whenever status is "active", include the Droplet ID towards the stock file
  • Call reload_haproxy to regenerate the configuration that is HAProxy and reload force balancer
def delete_server(line_number)

once the doproxy.rb delete demand is employed, this technique deletes the specified Droplet and deletes its ID through the stock file. After that it calls reload_haproxy to re-generate the configuration that is HAProxy and reload force balancer.

  • Remove the specified line through the stock file (delete Droplet ID)
  • Use API to delete Droplet by its ID
  • Call reload_haproxy to re-generate the configuration that is HAProxy and reload force balancer
def generate_haproxy_cfg

This is a supporting technique that produces HAProxy that is new files on the basis of the Droplets into the stock.

  • Open the configuration that is HAProxy (haproxy.cfg.erb)
  • For each Droplet in stock, include a matching backend host entry
  • Write ensuing haproxy.cfg file to disk
def reload_haproxy

This is another supporting technique that copies the HAProxy setup file to the location that is proper reloads HAProxy. This depends on generate_haproxy_cfg.

  • Copy HAProxy setup file haproxy.cfg towards the location in which HAProxy will appear because of it on reload
  • Reload HAProxy

That's the crucial rule which makes work that is DOProxy. The thing that is last will talk about is DropletKit, the API wrapper that people utilized in DOProxy.

DropletKit Gem

DOProxy makes use of the DropletKit treasure which will be the DigitalOcean that is official API Ruby wrapper that facilitates calls designed to the DigitalOcean API. DropletKit we can effortlessly compose Ruby programs that such things as:

  • Create brand new Droplets
  • Delete current Droplets
  • Get information regarding current Droplets such as for example status, internet protocol address, Droplet ID, area, etc

This tutorial dedicated to these specific API endpoints, but remember that there are lots of other endpoints that will help facilitate management that is programmatic of DigitalOcean host infrastructure.

Conclusion

Now that you have seen exactly how a script that is simple assist measure a host environment by leveraging the DigitalOcean API, cloud-config, and metadata, it is possible to use all of these principles to measure your host setup. Although DOProxy is not designed for manufacturing usage, it will supply good pair of a few ideas for applying your scaling solution.

Remember your setup that is scaling right here with DOProxy is informational, however it could possibly be significantly enhanced from it together with our monitoring system. This might permit you to immediately measure the application host tier along dependent on specific conditions such as for example host resource utilization.

LEAVE A REPLY

Please enter your comment!
Please enter your name here