Early Experimentation Toward Automating Web Application Deployment

The Problem:

"Okay, we just finished writing our application and it's ready for prime time and public consumption! Now what?"

The answer isn't usually too difficult to sort out. You need to find space on a server, make sure it's web-accessible, it can communicate with its database, and a few other things. The process may vary, but the steps are usually pretty similar and the stakes may not be very high for a first time launch. But...

"Okay, our application has been up and running for a while and a lot of people depend on it. We've made some changes and need to get them into production without causing chaos. Now what?"

That's a really common question in web development, but really good answers to it have only started cropping up in the past four or five years. Deploying new software the first time is easy. Improving it without breaking things is entirely different. Usually if you want to change something, you need to try out your new code in a testing & development environment, verify everything is working as intended, then move that new code into a production server. This isn't so bad when things work like they should. In practice though, emergencies come up and a problem arising in production may need to be fixed right there on the fly, leading to an inevitable falling out of sync between test & production environments.

Backing changes out is another problem. If things don't work as expected though, you need a good way to return to the previously working code without disrupting your end users. When copying files around manually, you need to keep a copy of the old files close by when you update them in case you need to move the old ones back in place. That leads to web servers which slowly start looking like this:

• the-real-file.html
• the-real-file.html.backup~
• the-real-file-testing-deleteme.html
• the-real-file (Copy).html
• the-real-file (Copy (Copy)).html

Okay, maybe not that bad, but you get the idea.

The Solution:

We have been thinking carefully about how we can improve our web deployment strategies to be sane, consistent, repeatable, automated, documented, logged, and most of all, safe from breakage. One of the tools emerging as a leader in this arena is called Capistrano. Basically, Capistrano steps through a list of tasks to retrieve code from a repository (Github), copy it to a server, and swap it into place for use. But it also manages those old versions on the server too, allowing a seamless and instantaneous return to the previous version when things don't work as planned without the (Copy (Copy)).backup litter.

To make sure things do work as planned before end users ever see the new features, Capistrano can run a whole series of pre-flight checks. These checks may include verifying that the files that contain database passwords are present and correct, running automated software tests, ensuring the application can communicate with a 3rd party resource, or just about anything else we might need.

Looks like we're all clear to deploy our code!

If everything passes, the new code release is swapped instantly in place of the old, and if anything fails, the whole process rolls back without downtime for end users. We can correct the issue, and try again when it's ready.

Something isn't right, but Capistrano was smart enough to notice and back out to the previous working state.

Our Experiment:

As a first proof of concept, we've setup the Affinity String Statistics web application to be deployable with Capistrano. Since it mostly stands alone without deep dependencies on our other web applications, and because a little work was being done on it, it seemed like a good first candidate.

Now, because it only stands alone mostly and not entirely, it is also a good test bed to uncover the kinds of issues we can expect to encounter when fitting other applications for Capistrano deployment. Chiefly, we will have to work out some new ways of organizing files in the UL web space, but recent work Dave has done on configuration file management gives us a really sound push toward accomplishing that.

It's an exciting project, and  in an ideal world we will eventually be able to add "push-button" deployments to all of our custom code on the web for a fast, safe, repeatable path from our desktop workstations through wider testing and into production.