Motivation

I have a 12-year old daughter. Like many others she spend a lot of time online, Skype & Facebook of course, but mostly a horse site called net-hesten.dk (warning: in danish!) I think she has probably written thousands of messages on their forum One of the main things you can do there is to create your own farm where you can buy/sell horses etc. As part of making your farm, you create a profile for your farm. And here all kinds of creativity starts.

Over the summer she has taught herself HTML, CSS, some Javascript, Gimp etc. just to be able to create a “cool” profile page. Nothing beats motivation for learning new stuff! Now the time has come where she wants to create her own site (the precise mammal is not known yet and probably not important). She is inspired by other girls that have learned PHP, MySQL etc. in order to build things. And while the results are not always impressive when viewed with the eyes of a professional, the dedication is.

So I decided to help her getting some of the basic components up and running and figured I could kill two birds with one stone:

1. Try out some new technology & services that I’ve been wanting to play with for a while
2. Get some quality time with my daughter (How often do you hear “I’ve got a cool dad”, when you proclaim you can build a web application  :-)

Ground rules

This is a side project, so low cost & high return on time invested is important. So I’ll try to see how little code I can get away with writing. It’ll also function as a test-bed for the highly (too?) popular MVP concept from the lean startup camp: Only focus on the minimal amount of functionality needed.

I’ll not be doing the layout & design, but will leave that to my daughter (which probably has better abilities for this anyway)

The idea is to blog about some of the code components that (hopefully!) will be created during this project and put some, if not all, the code on github.

The components

• The Scala language
• Scala IDE for Eclipse
• LIft web framework
• MongoDB
• Akka
• …and probably more

The Services

Low cost is important and I don’t expect a lot of traffic here, so many of the online service providers have free plans that can be useful:

• CloudBees for hosting JVM applications. As opposed to e.g. Google AppEngine you can actually use the full JVM functionality (e.g. create threads for Lift’s comet support)
• MongoHQ for hosted mongodb
• SendGrid for sending out emails

So, it took a little longer than expected to write this article which continues where the previous stopped. But all good things come to he who waits The software used in the previous article all had major updates in the meantime:

• Scala 2.8 (2.8.1 is just around the corner)
• Eclipse 3.6
• Scale IDE for Eclipse (though a nightly build is currently needed for Eclipse 3.6)
• Gradle 0.9 RC1
• Lift 2.1 RC2

Installing the software

I’m writing this on OS X 10.6.4, using the default Java installation:

java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)

Eclipse

I’m using the latest Eclipse 3.6 (Helios), which can be found here. Install it and launch with a new workspace.

JRebel

JRebel allows you to make code changes and have the changes reflected in your running app without restarting. This is very handy for a rapid development cycle. It’s not perfect and doesn’t work with all changes, but most of the time it picks up the changes ok. Note that you can get a free license if you only work with Scala files!

Follow step 1-3 in the instructions here to download and install JRebel as well as the JRebel Eclipse plugin.

Scala IDE for Eclipse

Next, install the Scala IDE for Eclipse from this update site: http://download.scala-ide.org/nightly-update-helios-2.8.0.final/

Use this guide as a starting point: https://www.assembla.com/wiki/show/scala-ide/Requirements_and_Installation

Try creating a small sample Scala project to verify your installation:

1. Select the Scala perspective (click on ‘+’ the first time and select “Other..”)



2. Right-click in Package Explorer and select New->Scala Project
3. Enter helloworld as Project name
4. Expand the project, right-click on the src folder and select New->Scala Application
5. Enter “HelloWorld” as Object name
6. Add println(“Hello, World”)
7. Right-click in the editor and select Run As->Scala Application

Note, that since this is a nightly build, there may be times when the plugin is not in a good shape. Seek help on the scala-ide mailing list

Creating the Lift Eclipse project

Since we’re using Gradle to manage the dependencies to external JARs (see previous article for details), we need to tell Eclipse where in the file system those jars can be found. You need to find this directory on your system. It’s usally in ~/.gradle/cache which for me maps to /Users/Jeppe/.gradle/cache.

So we create a classpath variable, GRADLE_CACHE  that points to this directory. Open Eclipse preferences:

Next, create the actual Eclipse files (.classpath and .project) by running

$ ./gradlew eclipse
:eclipseClasspath
:eclipseProject
:eclipseWtp
:eclipse
BUILD SUCCESSFUL

Total time: 11.27 secs

Finally, import the project into Eclipse:

1. Right-click in Package Explorer and select “Import…”
2. Select General->Existing projects into Workspace
3. Select the location of your build.gradle file as root

The created Lift application contains a main object that runs an embedded Jetty container which is very useful for development. It’s located in src/test/scala/RunWepApp.scala. Right-click this file and select Run As->Scala Application. Now Jetty starts up and you can navigate to http://localhost:8080 to see the application.

The final step is to enable JRebel for the Run configuration just created. Select “Run Configurations” and find RunWebApp$, select the JRebel tab and enable the JRebel agent.

Restart the application. JRebel should print something to the console.

JRebel in action

Now try to change something. Go to src/main/scala/example/snippet/HelloWorld.scala.

1. Change the output of the howdy snippet by e.g. replacing “d.toString()” with some text.
2. Save the file in order to compile it.
3. Refresh the page in the browser.
4. Watch the new content be displayed in the browser!
5. Watch the console output that says JRebel has reloaded the class

As mentioned earlier, not everything can be reloaded by JRebel and not everything in Lift can be changed during runtime.

If JRebel cannot reload the changes, you usually get a runtime error (ClassNotFound, ClassCastException etc) and you’ll have to restart. If JRebel does pickup your changes and nothing happens, chances are that Lift cannot dynamically pick up the changes. Examples of the latter are:

• Things that happen in Boot. Your application is not re-booted to pickup the changes. This unfortunately includes SiteMap changes.
• Addition/removal of mapper fields. The fields of a mapped object are determined at startup and cannot change at runtime.

So, this completes the post. Enjoy! And please provide feedback if you have any ideas to improve the development cycle!

One area where Lift is lacking is in documentation, so I’ll try to write a few posts here that can help people getting started using Lift. I’ll be writing about the workflow we use, since

1. It works for us
2. It is not using Maven, a source of some ….. frustration for many

Introduction

There are many tutorials available that shows how to create a new Lift project, building and running using Maven. If you like maven and think that it’s fine, you should probably follow one of those, since this is the most common setup and you’re likely to find help on the mailing list. We’ve decided to use Gradle.

Why not SBT?

The Simple Build Tool is a cool build tool aimed primarily at Scala projects. It has nifty support for continuous compilation and testing. I haven’t used SBT, but hear it’s quite good (Lift may eventually move to SBT).

But I think many of the features that are cool about SBT (ie. the continuous compilation and testing) are most useful during development if you’re just using a text editor. If you’re using an IDE (such as Eclipse) it already handles this.

I mostly use a build tool for automating the build/test/deployment process and as such, I think Gradle is better suited since it is scripted in Groovy and has complete support for existing Ant tasks. But I really need to try out the Emacs+SBT combo at some point to see what all the fuzz is about

Prerequisites

Ok, lets get started. To create and run your Lift projects, you need these packages installed and working:

• A Java 1.5 (1.6 preferred) JRE installed
• Git (not strictly necessary as you can download the files as zip/tar package from Github)

That’s it. No need for Maven, Gradle, Scala or anything else since we’re using a the nice Wrapper feature of Gradle that automatically downloads and installs the correct version of Gradle.

A minor note about versions

In these posts I’ll use the versions of various software that I know work. Both Scala, Lift and Gradle are about to release new versions which will improve many things, but I’m only using the Gradle pre-release. I’ll update the posts when the new Scala & Lift versions are more stable.

Creating a Lift project

Creating a project is as simple as cloning a project from Git:

$ git clone git@github.com:jeppenejsum/liftstart.git

This will fetch all the files from github. The project just created is a basic Lift application with database support (using H2) and simple user registration. The files follow a standard directory scheme.

The scala files are in src/main/scala and the web files (html, images etc) are in src/main/webapp. The file src/main/scala/bootstrap/liftweb/Boot.scala is where you customize most Lift settings, it is run once during startup. Check out the Lift documentation for more details.

Building and running the project

Gradle reads the build.gradle file to determine the tasks that are available in the project. The build.gradle in this project is fairly simple, mostly adding the scala-tools.org repository and specifying the dependencies needed to compile and test the project.

// Minimal build.gradle for Lift project
apply {
    plugin 'scala'
    plugin 'war'
    plugin 'jetty'
}

scalaVersion = '2.7.7'
liftVersion = '2.0-M4'

jettyRun.contextPath = "/"

repositories {
    mavenCentral()
    mavenRepo name:'scala-releases', urls:'http://scala-tools.org/repo-releases/'
}

dependencies {
    scalaTools "org.scala-lang:scala-compiler:$scalaVersion",
        "org.scala-lang:scala-library:$scalaVersion"

    compile "org.scala-lang:scala-library:$scalaVersion",
    	"net.liftweb:lift-webkit:$liftVersion",
    	"net.liftweb:lift-mapper:$liftVersion",
    	"net.liftweb:lift-json:$liftVersion",
    	"net.liftweb:lift-util:$liftVersion",
    	"net.liftweb:lift-common:$liftVersion",
        "com.h2database:h2:1.1.117"

    testCompile "junit:junit:4.5",
        "org.scala-tools.testing:specs:1.6.1",
        "org.scala-lang:scala-compiler:$scalaVersion",
        'org.mortbay.jetty:jetty:6.1.22',
        'org.mortbay.jetty:jetty-util:6.1.22',
        'org.mortbay.jetty:jetty-management:6.1.22'

    providedCompile 'javax.servlet:servlet-api:2.5'
}

task wrapper(type: Wrapper) {
    gradleVersion = '0.9-preview-1'
}

In order to run a task you use the syntax

$ ./gradlew taskName

This will install gradle if it’s not already installed and then execute the specified task in the build script. So to compile the project and load it within the built-in Jetty web server, use the following command:

$ ./gradlew jettyRun

Running this (and ignore the warnings about multiple SLF4J bindings, this is a bug in the built-in jettyRun task) and  then browsing to http://localhost:8080 should show something like this:

To package the project such that it can be moved to a standalone web server, you can create a war file by executing

$ ./gradlew war

This will create the file build/libs/liftstart.war which can be copied to the apps directory of any servlet container.

Next steps

Ok, so you’ve created a Lift project and know how to run it using Gradle. So go crazy and start hacking!

But as you’ve probably noted, this is not exactly an efficient workflow since compiling everything from scratch takes quite a while. In a future post I’ll explain how to setup a fairly rapid development environment using Eclipse and JRebel.

Background

One of the nice things about starting a new project in a startup is the freedom of choice when it comes to selecting platform and tools. You can spend an awful lot of time mulling between the different choices, but in the end, it is usually not the choice of programming language that kills a startup.

First, a little background. I’ve been programming for more than 15 years in C++,VB,C#, Java,Perl & PHP. The last 5-6 years it has been mostly enterprise Java. When I started at my last job (another startup) we had a product implemented in Java that we turned into a SaaS platform. The core remained in Java  and much of the web frontend was implemented in PHP. I really liked the productivity we got out of PHP as compared to the Java code. Very fast turnaround times. But somehow the language/platform doesn’t really  turn me on.

So many tools, so little time

So I knew I didn’t want to use pure Java or PHP,  in the end I ended up choosing between

• Ruby + Rails
• Python + Django
• Scala + Lift

Despite the success of Rails in the past few years, it never really clicked with me. Granted I haven’t used it for more than a few weeks, but it never really felt natural.

Same with Django. I’ve used python on several occasions for scripting tasks, but never really for webapps.

So I ended up with Scala and Lift. I looked briefly at Scala in 2007, liking what I saw, but didn’t use it for anything serious. In the end, what made the difference was:

1. Well known development and deployment platform (JVM, Build tools, IDEs, app servers etc)
2. Java interop, both for use of existing libraries but also as a fallback plan if everything failed.
3. Lift’s clean templating. We didn’t really need any of the real-time features (ie. comet support) in Lift, although the Ajax support looked nice.

Status

So here we are, 6 months after development began. How has it been? In short: Not bad. We have a platform up and running, productivity is pretty good and getting better all the time.

When compared with Java code, our Scala/Lift code is not very verbose, but compared to Rails & Django code I still feel it is (slightly) more verbose. I think this is because of the static typing that (despite type inferencing) requires quite a few type annotations. Otoh, when something type checks, it usually works

Scala

Pros

Scala the language has a lot of benefits, many of which have already been described in other articles on the net. Here are some others  that I find important:

• Very concise, you can skip most types, parentheses, semicolons etc. Except when using parameterized types (generics). Here’s Hello World in Scala (using the Application trait is considered bad mojo for various reasons but ignore that for this example )

  object HelloWorld extends Application {
    println("Hello World")
  }
  

• You can start coding as you would in Java and gradually ease into the Functional Programming paradigm (if you want)
• You can use all existing Java libraries
• Very nice testing frameworks such as Specs, ScalaTest & ScalaCheck
• You can write very concise, readable code. E.g we have a dataset to be charted. It has a number of dataseries each having a number of values. To get the total of all values, we can just write this method

  def sum = series.map(_.sum).foldLeft(0.0)(_+_)

which says, for each series, calculate the sum and then add all the results

• Lazy parameters make logging code (which I think is very important, but is responsible for a lot of boilerplate) very readable. In Java you usually guard calls to logging to avoid the expensive computation of the string to be logged:

if (log.isDebugEnabled() {
  log.debug("Here is the XML blob:"+xml)
}

in Scala (or rather, Lift) you can, without loss of performance, just write

  log.debug("Here is the XML blob:"+xml)

The string to be logged will not be computed if debug log statements should not be logged.

• Very easy to write meaningful DTOs. In Java, a lot of boiler plate is needed to make a class wih e.g. 3 properties. Often I would use an array (which will be untyped) or a tuple. Both suffer from the problem that you’ll have to know that the 2nd field is the customer id and the 3rd is the total amount. In Scala you can simply write a case-class and get named fields with typed access:

   case class MyRecord(id:Int, custNo:String, amount: Double)
  

Cons

There are also a number of drawbacks to using the Scala language

• Tooling. We’re using the Scala 2.7.x plugin for Eclipse. While it works, there are a lot of bugs that means you’ll often have to rebuild the entire project for your changes to be seen. The 2.7.x version is basically abandoned and all work is going on in the 2.8 version which is supposedly much better. Unfortunately, we can’t switch to 2.8 until all the other tools we use are running on 2.8 (which is not in beta yet)
• While type inference works, it’s not as powerful as in ML (and probably never will be. Scala’s type system has inheritance making it much harder to inference the correct types). It frequently requires you to add some type info. After a while you figure out where this is needed though.
• The Scala compiler is rather slow. Fortunately, the Eclipse incremental compiler works most of the time.
• The Scala book is very well written, but the API documentation is rather sparse. With most Java code I seldom have to use the source, but for the scala code it seems easier to start with the source.
• The syntax can get quite ugly until you recognize the patterns. This is a problem since the documentation is rather bad. Parameterized types can can be especially bad I think. This is an example from the scala-user mailing list:

implicit def TraversableBind[M[X] <: Traversable[X]] = new Bind[M] {
  def bind[A, B](r: M[A], f: A => M[B])(implicit w: CanBuild[B, M[B]]): M[B] = r.flatMap(f)(breakOut)
}

• No binary compatibility between Scala versions. This means that code compiled with Scala 2.7.3 is not (in general) compatible with code compiled with 2.7.5 and needs to be recompiled. This is not a big deal for our own code, but it does make it difficult to move to another version since all the Scala libraries used needs to be released first. I think this is also what has slowed down use of the forthcoming 2.8 release: people with sizable apps are waiting for the entire tool chain to be available and since 2.8 is not yet in beta, you need to agree on a version of 2.8 for everything.

Lift

Overall, I like the Lift framework and how it utilizes the Scala language. The fundamental approach to request handling seem very well thought out and makes it easy to handle both traditional web apps, Ajax, REST APIs etc. The focus on Lift seems to get things done, not so much to create the perfect web framework abstraction that has all corner cases covered. This means most code has been battle tested, but sometimes you’ll wander along an untrodden path and strange things will happen.

Pros

• Without a doubt, the best thing about Lift is the community. Very friendly people, fast response to most questions. Eager to get newbies up to speed.
• The Eclipse incremental compiler, Jetty & the JRebel plugin makes for fairly rapid turnaround times even if it’s not in the same league as e.g. PHP & Python. But many changes can be reloaded on the fly without having to restart the server.
• Very fast to get a real app up and running. Prebuilt archetypes for a basic app with authentication and DB access.
• Focus on security. Much effort has been put into making Lift apps secure. E.g. you cannot repost a form since the field ids are unique
• The Lift ORM, Mapper, is fairly simple, but does provide a very quick way to get CRUD functionality up and running.
• Very easy to add Ajax functionality (I can’t speak on the Comet functionality)
• Pretty good i18n support. Strings, templates etc. can be locale dependent and the locale can be computed for each request if needed.
• The clean separation between UI and code (it is impossible to have code in templates) makes it easy for designers to modify the layout and styling with breaking anything.
• The focus on getting things done means if there’s a real use case, new features can be added fairly quickly. Same goes for bugs in production code.

Cons

• API Documentation is very scarce. Usually, I use the source to figure out how stuff is working, but sometimes you need the big picture in order to get a good understanding. There is a free book available, but I usually don’t use this on a day to day basis.
• Form handling is very basic. There isn’t really any form specific code in Lift, so making a real, usable form (with i18n, validation, etc) takes some effort.
• Lift apps are very stateful. This is not a problem for us, but for apps that needs to be very horizontally scalable, this is a showstopper. The problem is not easily solved, since it requires serialization of closures.
• The Lift ORM, Mapper, is fairly simple, which means that for more complex scenarios it is somewhat difficult to extend, loosing some of the RAD capabilities
• The focus on getting things done means that some areas are not really consistent or complete.
• While you can’t get code into your templates, it’s easy to get UI into your code, which is (almost) just as bad. The dynamic part of the UI is done by snippets and they of course need to emit HTML. But it is easy to put all kinds of style, class attributes as well as other things which belong in the template, into this dynamically generated code. This makes it difficult for designers to modify the layout and styling without touching the Scala code.

Ok, installed ffmpeg on OS X using MacPorts:

sudo port install ffmpeg +gpl +lame +x264 +xvid

Tried to convert a file. Got this error:

[libmp3lame @ 0x1804600]lame: output buffer too small (buffer index: 8360, free bytes: 1432)

Turns out this is a known bug with version 3.98.2 of libmp3lame, which, incidentally is the latest version on MacPorts.

I tried version 3.98 and it seems to work.
This guide tells how to install an older version of a port.

To save you some trouble, these are the steps I took to make it work:

cd /tmp
curl -O http://trac.macports.org/export/38059/trunk/dports/audio/lame/Portfile
sudo port install

Voila! ffmpeg can now generate mp3s without errors.

And then you wish you belong to the (small?) group of people who write down every change to a log file, so you can answer this question.

I’m not in this group but have had this need often enough that I’ve tried to automate at least some of the logging.

Twitter to the rescue

Lately, I’ve started using twitter to maintain a timestamped log of the changes we’re making to our production system. It’s very easy to create a dummy twitter account (and make it private if you don’t want to expose all your production info to the public!) for the purpose of tracking these changes.

Since everybody else in the company is already using twitter, it’s very easy, for those who wants this info, to follow this user and get updates whenever something interesting happens.

Posting an update

Twitter has a decent API to access the functionality and posting a new update can be done from the command line, using curl. This makes it easy to post updates from all kinds of places such as build scripts, boot scripts, cron jobs etc.

Since we’re using Gradle to handle all our builds and automation, we can define a Groovy function called tweet, which can be used throughout the scripts:

def tweet(msg) {
  ant.exec(executable:"curl") {
    arg(line: "--user twitteruser:twitterpassword")
    arg(line: "--data-ascii \"status=$msg\"")
    arg(line: "--silent")
    arg(value: "http://twitter.com/statuses/update.json")
  }
}

If you’re doing any manual changes (but of course you don’t, right? Everything is scripted and put under version control ), you can always manually post an update for the account by using a multi-account aware twitter client.

What to post

So, what can you use this for? Well, you probably shouldn’t post highly sensitive stuff, even if it’s on a private twitter account. But, basically as much useful info as possible. This is when we currently do updates:

• Whenever a new deployment is about to start
• When a deployment is complete (with links to new features, bug solved etc)
• When a new EC2 instance is booting
• Any changes made to the AWS setup
• Etc.

Scala has a nice mechanism for creating lists with a for comprehension:

val l = for (...) yield computeResult(...)

Basically, l will become the list of return values from computeResult. Cool.

Scala also has Range class that, together with the for comprehension mimics a standard for-loop construct:

for(i <- 1 to 20) {
 ...
}

Problem

I had the following case. I needed to create some objects in a database and return the results in a list. So I created something that looked like the following:

val databaseObjects = for(i <- 1 to 20) yield {
  val o = CreateObject
  o
}

When I ran this code, nothing happened?! I checked the DB, no records to be found anywhere. I tried printing the object o inside the loop. Nothing got printed. Tried the construct in the Scala REPL. Everything worked fine. Then I tried printing databaseObjects.length after the loop. Shows that 20 objects were created. Hmmmm. Then I tried printing all elements of databaseObjects. They were created just fine. Hmmmmm.

I recalled  a discussion about the Range being created by the expression “1 to 20″ is a lazy data structure which is not evaluated before the results are needed. Turns out this lazyness is contagious (more details can be found here)  Ahhhh. Everything makes sense now, since nothing happened until I tried to actually fetch the objects from the list (and the REPL implicitly retrieves the objects to print out the results!)

Solution

The fix is easy, you need to force the lazy range:

val databaseObjects = for(i <- (1 to 20).force) yield {

Finally, we’re coming to the end of the road. The only things left in order to finish the continuous deployment cycle are these steps:

• Run integration tests
• Shutdown instance
• Add new nightly job to Hudson that runs the integration tests

Running integration tests

Normally, you would use something like Selenium to do browser-based integration tests, but configuring Selenium RC is outside the scope of these articles (though, you could provision new EC2 instances and run Selenium Grid to test the app from various browsers, even running on Windows. Hmmm, maybe there’s some material here for another article series )

To demonstrate the principle, lets create a simple task that performs a crude test to verify that the site is at least running

createTask('integrationTest', dependsOn: 'testDeploy') {
	try {
		client = new org.apache.commons.httpclient.HttpClient()
		method = new org.apache.commons.httpclient.methods.GetMethod("http://$createInstance.dnsName/")
		client.executeMethod(method)
		def s = method.responseBodyAsString
		if (s.indexOf("Welcome to the final helloworld project!") < 0) {
			println s
			throw new RuntimeException("Test failed")
		}
	} finally {
		ec2.terminateInstances([createInstance.instanceId])
	}
 }

Note, that we shutdown the instance at the end.

Now it should be straightforward to add a new Hudson job that runs at night and executes the integrationTest task. This is left as an exercise to the reader!

Where to go from here

This concludes the article series on continuous deployment. The example webapp is very crude, and the target stack rather simple, but this series does describe a complete end-to-end example of how to automate the deployment and testing of a webapp. Expanding on each of the steps in order to match a more realistic scenario should not be insurmountable.

But what I’ve presented here is just the nimble beginning. I know Amazon AWS is not the cheapest hosting you can get, but it does provide you with a lot of flexibility. Here are just some of the possibilities:

• The single server architecture we’ve used in this example is probably too simplistic in all but the smallest setups. But it is easy to extend even your nightly builds to include a more realistic setup including a load balancer, a few web servers, a few app servers and a database server.
• If you’re using EBS to store your data, it is very simple to get a replica of your production data by taking a snapshot of your EBS volumes and then mount them in your testing environment. Then you can also test your DB upgrade scripts.
• Extend integration tests to do browser-based testing using different browsers. This should be possible now that EC2 supports windows instances.
• Use Chef to deploy a new version of the application without downtime.  You can easily start new app servers with the new code and start routing (perhaps just some?) requests to the new version. When everything looks ok, you can take down the old instances. Of course this would probably require support from the application, since it needs to coexist on the same database with different code bases etc.

Source code

All the code in this example is available on GitHub http://github.com/jeppenejsum/helloworld/tree/master

What we’ve done so far is a pretty basic Continuous Integration setup. To take this to the next level, we’ll add a new job, to be run at night, that provisions and configures a new EC2 instance, deploys the application and runs the integration tests.

The steps that needs to be executed are

• Start a new EC2 instance
• Configure the instance with all the software needed to run the application (in our case Jetty & nginx)
• Deploy the application
• Run the integration tests
• Shutdown the instance

Chef

Chef is a new tool for automated infrastructure management. Like Gradle it’s a task specific DSL (for managing infrastructure) based on a scripting language (Ruby).

For a full Chef implementation, you would run a Chef server that knows everything about your infrastructure. It also holds your Cookbook which is a collection of recipes, where each recipe describes how to install/configure some software on a node. New nodes will run chef-client to configure the node according to the “recipes” defined on the sever.

But the nice thing about Chef is that it also seems to scale down to the quite simple scenario we are going to use here, which won’t be using the client/server setup but the chef-solo command. This will still execute recipes to configure the node (and those same recipes can be used in the full scale setup), but will not need a Chef server.

The helloworld recipe

Whenever a node is brought up it should have a specific role (i.e. db server, web server, app server etc). I think it makes sense to create a recipe for each role and then have that recipe contain all the stuff that needs to be installed and configured for this role. In our example, we only have a single role, helloworld, which will contain

• nginx web server
• Jetty6 application server
• The helloworld application

This is specified in the recipe shown below. It basically includes recipes for the dependencies nginx and jetty (which in turn depends on java). After the prerequisites are installed, we download the helloworld application from S3 and put it into the Jetty webapps dir. Finally, we restart jetty6 to deploy the application.

Note that the source of the war file is specified as source node[:war]. This means the actual value is a property of the node configuration. In a full Chef setup the value will come from the Chef server, but when running chef-solo we will specify the node properties in a file called dna.json.

Unfortunately, until this bug is fixed in Chef, the URL cannot contain special characters.  When using S3, this means you cannot create a signed URL for the war file but need to make the file public readable. Hopefully it will be fixed soon (the fix is simple, see ticket)

include_recipe "jetty"
include_recipe "nginx"

remote_file "helloworld_war" do
        path "/usr/share/jetty6/webapps/ROOT.war"
        owner "jetty"
        mode  0640
        source node[:war]
end

service "jetty6" do
  action :restart
end

The jetty recipe

The previous recipe was very simple (which is a good thing . The jetty recipe is a little more involved and shows some of the power in Chef. It installs a few packages (using Apt ), downloads a number of deb files, installs them, configures the package defaults and enables the startup script.

include_recipe "java"
# Needed by jetty extra & jsp
%w{ ant libgnujaf-java libgnumail-java }.each do |pkg|
 package pkg do
 action :install
 end
end

jetty_version = "6.1.16_all"
jetty_debs = %w{libjetty6-java libjetty6-jsp-java libjetty6-extra-java jetty6}

jetty_debs.each do |deb|
 remote_file deb do
 path "/tmp/#{deb}_#{jetty_version}.deb"
 source "http://dist.codehaus.org/jetty/jetty-6.1.16/debs/#{deb}_#{jetty_version}.deb"
 end
end

bash "install-jetty" do
 code "cd /tmp && dpkg --install *jetty6*.deb"
end

bash "enable-jetty" do
 code "sed s/NO_START=1/NO_START=0/ -i /etc/default/jetty6"
end

service "jetty6" do
 supports :restart => true, :reload => true
 action [ :enable, :start ]
end

Chef comes with recipes for many packages, including java and nginx, and they are a good inspiration.

Bootstrapping an EC2 instance

Before we can begin cooking, we need to launch an EC2 instance. Since Gradle scripts are Groovy scripts, we can perform all this directly in our build scripts by creating a few new tasks & methods. By using the Java libraries Typica and Jets3t, it is relatively easy to perform these tasks. I’ll not go through all the code here, just highlight a few points.

createTask('testDeploy', dependsOn:['libs','createInstance']) {
  urls = uploadCookbooksToS3()
  bootstrap(urls[0], urls[1])
}

createTask('createInstance', dependsOn: 'libs') {
  def config = new LaunchConfiguration(imageId)
  config.keyName = ec2keyname
  def instances = ec2.runInstances(config)
  def instance = instances.instances.get(0)
  createInstance.instanceId = instance.instanceId
  println "Started instance with id $createInstance.instanceId"
}

The testDeploy task depends on the createInstance task, which is the task that actually launches the EC2 instance. I’m using a RightScale Ubuntu AMI which includes Ruby and rubygem. The instance id is stored as a property of the createInstance task. The uploadCookboksToS3 method, uploads the cookbook with all the recipes to S3 and creates and uploads the following dna.json file

{"war": "http://mys3bucket.s3.amazonaws.com/helloworld-1.war", "recipes": "helloworld"}

This file contains the URL to the war file to be used by the helloworld recipe as well as the list of recipes that should be installed on the node. uploadCookbooksToS3 returns URLs for the cookbook.tgz and dna.json files which are needed to download from S3. The bootstrap method first waits for the EC2 instance to fully boot and then performs the following tasks:

  sshExec("gem install --no-ri --no-rdoc ohai chef --source http://gems.opscode.com --source http://gems.rubyforge.org");
  ant.scp(passphrase:"", todir: "root@$createInstance.dnsName:", trust: true, keyfile: ec2KeyFile) {
		fileset(dir: "deployment") {
			include(name: "solo.rb")
		}
	}
  sshExec("chef-solo -c solo.rb -j '$dnaURL' -l info -r '$cookbookURL'");

Basically,

• Install Chef using gem
• Copy the solo.rb config file to the instance
• Configure the node by calling chef-solo with URLs for the dna & cookbook files

Chef will now download the recipes and run them to configure the node.

What we have accomplished so far is that when you type

$ gradle testDeploy

You can sit back and wait for a few minutes. When all is done, you can point your browser to the public DNS of the EC2 instance and …tadaaa…..you should see something like this:

This might not be a very impressive application but you can find resources on the net that enable you to write a Twitter clone in Lift in 30 minutes. You might need a little more advanced deployment architecture though

We can now build our application manually, but need to automate the actual build process. For this I’ll be using the Hudson Extensible continuous integration engine. I’ve been using CruiseControl for many years, and while it does the job, I’ve been annoyed enough that I’ll try something new.

Add the source to a repository

All CI systems need to get the source from a SCM so that the code that is being built is in a well defined state. So let’s  add the source of the webapp to a SCM, in this case github.com.

$ git init
$ git add *
$ git remote add origin git@github.com:jeppenejsum/helloworld.git
$ git commit
$ git push origin master

Now all the sample code is available here http://github.com/jeppenejsum/helloworld/tree/master

Install Hudson

While you could use Amazon EC2 to host your build server, it takes some time to configure with persistent storage, mail server etc. Since this is not the main focus of these articles, I’ll just use our existing Xen infrastructure. Provisioning a new server running Debian Lenny is simple (when xen-tools is configured correctly):

# xen-create-image --hostname buildserver --mac 00:16:3E:55:42:8E
# xm create /etc/xen/buildserver.cfg

Now sign in to the new VM and install the necessary prerequisite software:

# apt-get update && apt-get upgrade && apt-get install sun-java6-jdk git-core daemon subversion unzip
# git config --global user.email "you@example.com"
# git config --global user.name "Your Name"

There seems to be a Debian repository for Hudson, but it’s very unstable so we’ll just install directly from the .deb files:

# cd /tmp && wget http://hudson.gotdns.com/debian/binary/hudson_1.303_all.deb && dpkg --install hudson_1.303_all.deb

Install Gradle as previously described here.

Restart the server to make sure the service is started automatically. After you’ve done this, point your browser to the buildserver at port 8080. You should see something like this:

Install Hudson plugins

We need to configure a number of Hudson plugins, goto http://buildserver:8080/pluginManager/available and install the following plugins:

• Git plugin – For access to github.com
• Gradle plugin – Since we use Gradle to build the software

Restart Hudson for the changes to take effect

# /etc/init.d/hudson restart

Configure

The Gradle plugin needs to be configured. Go to http://buildserver:8080/configure and enter the path

You should also configure SMTP as appropriate for your system. If you have a Gmail account, you can use this for sending email.

Setup smoke test to build on new commits

Now that we’ve installed and configured Hudson, it’s (finally) time to configure a smoke test to build whenever changes are comitted to the application. This build should be quick in order to provide quick feedback to developers on their last commit. I usually include the following:

• Compile the entire application
• Run all unit tests

but don’t include any packaging, integration tests or other long running test cases in this task. Incidentally? the above is eactly what happens when we execute the default test task on our Gradle build file.

The steps are

• Click “New Job”
• Enter jobname “Helloworld smoke test”
• The type should be “Build a free-style software project”
• Click “Ok”

On the next page, fill out the following sections and tweak as you see fit:

Verify

Press the “Build Now” link and the code should build. If you push any changes to github, the code should compile and you should receive an email if anything fails (compilation or tests).