The importance of having a solid build process

Modern web applications are large and complicated pieces of engineering, using many different technologies, sometimes deployed on hundreds (or even thousands) of servers throughout the world, and used by people from dozens of locales. Such applications cannot efficiently be developed without relying on a solid build process to do all the dirty and repetitive work of reliably putting all the pieces together.

Apache Ant

Many tools (make, gnumake, nmake, jam, etc.) are available today to build applications. However, when it comes to building a web application, my personal favorite is definitely Apache Ant (here is a good explanation of Ant’s benefits over the other tools) This short tutorial will assume you already have some basic knowledge of Ant (if you don’t, you should flip through its user manual beforehand) This article will focus mainly on building front-end code using Ant.

Build types

It is often useful to build an application differently at different stages of its life cycle. For instance, a development build will have assertions and tons of logging code, while a production version will be stripped of all that. You may also need some intermediate build to hand off to QA, and still be able to debug things easily while running actual production code. With Ant, you can create one target per build type, and have a different dependency list for each target. Properties can also be used to customize specific build types. Here is an example:

<target name="dev" depends="load.properties.dev, js.preprocess, js.check.syntax, copy.jsp, copy.image.files, copy.css.files, copy.js.files, compile.jsp, copy.properties.files, compile.webapps, copy.libs"
    description="Development build">
</target>
<target name="prod" depends="load.properties.prod, js.preprocess, js.check.syntax, js.concatenate, js.minify, copy.jsp, copy.image.files, copy.css.files, copy.js.files, compile.jsp, copy.properties.files, compile.webapps, copy.libs"
    description="Full production build">
</target>
<target name="load.properties">
    <property file="ant.properties"/>
</target>
<target name="load.properties.dev" depends="load.properties">
    <property name="js.preprocess.switches" value="-P -DDEBUG_VERSION=1"/>
    <property name="js.compressor.switches" value="--nomunge --line-break"/>
</target>
<target name="load.properties.prod" depends="load.properties">
    <property name="js.preprocess.switches" value="-P -DDEBUG_VERSION=0"/>
    <property name="js.compressor.switches" value=""/>
</target>

Concatenate your JavaScript and CSS files

Concatenating JavaScript and CSS files contributes to making your site faster according to Yahoo!’s Exceptional Performance team. File concatenation using Ant is trivial using the concat task. However, it is often important to concatenate JavaScript and CSS files in a very specific order (If you are using YUI for example, you want yahoo.js to appear first, and then dom.js/event.js, and then animation.js, etc. in order to respect module dependencies) This can be accomplished using a filelist, or a combination of filesets. Here is an example:

<target name="js.concatenate">
    <concat destfile="${build.dir}/concatenated/foo.js">
        <filelist dir="${src.dir}/js"
            files="a.js, b.js"/>
        <fileset dir="${src.dir}/js"
            includes="*.js"
            excludes="a.js, b.js"/>
    </concat>
</target>

It is also possible to prepend some comments to the destination file, which is often used for license and copyright information, using a nested header element (see the Apache Ant manual) Just keep in mind that minifying your code will make these comments go away, so you may want to do this later in the build process.

Preprocess your JavaScript files using CPP

Preprocessing JavaScript code is very useful to facilitate the development process. It allows you to define your own macros, and conditionally compile blocks of code. One easy way to preprocess JavaScript code is to use the C preprocessor, which is usually installed by default on UNIX machines. It is available on Windows machines as well using cygwin. Here is a snippet of an Ant build.xml file illustrating the use of cpp to preprocess JavaScript files:

<target name="js.preprocess" depends="js.concatenate">
    <apply executable="cpp" dest="${build.dir}/preprocessed">
        <fileset dir="${build.dir}/concatenated"
            includes="foo.js"/>
        <arg line="${js.preprocess.switches}"/>
        <srcfile/>
        <targetfile/>
        <mapper type="identity"/>
    </apply>
</target>

You can now write JavaScript code that looks like the following:

include.js:

#if DEBUG_VERSION
function assert(condition, message) {
    ...
}
#define ASSERT(x, ...) assert(x, ## __VA_ARGS__)
#else
#define ASSERT(x, ...)
#endif

foobar.js:

#include "include.js"
function myFunction(arg) {
    ASSERT(YAHOO.lang.isString(argvar), "arg should be a string");
    ...
#if DEBUG_VERSION
    YAHOO.log("Log this in debug mode only");
#endif
    ...
}

Just a word of caution here: make sure you use the UNIX EOL character before preprocessing your files. Otherwise, you’ll have some issues with multi-line macros. You may use the fixcrlf Ant task to automatically fix that for you.

Minify your JavaScript and CSS files

Minifying your JavaScript and CSS files will help make your application faster according to Yahoo!’s Exceptional Performance team. I warmly recommend you use the YUI Compressor, available for download on this site. There are two ways to call the YUI Compressor from Ant. Here is the first one using the java task:

<target name="js.minify" depends="js.preprocess">
    <java jar="yuicompressor.jar" fork="true">
        <arg value="foo.js"/>
    </java>
</target>

Here is another way using the apply task that allows you to pass several files to the compressor using a fileset:

<target name="js.minify" depends="js.preprocess">
    <apply executable="java" parallel="false">
        <fileset dir="." includes="foo.js, bar.js"/>
        <arg line="-jar"/>
        <arg path="yuicompressor.jar"/>
        <mapper type="glob" from="*.js" to="*-min.js"/>
    </apply>
</target>

Update: Starting with version 2.2.x of the YUI Compressor, the ant target described above needs to be slightly modified:

<target name="js.minify" depends="js.preprocess">
    <apply executable="java" parallel="false">
        <fileset dir="." includes="foo.js, bar.js"/>
        <arg line="-jar"/>
        <arg path="yuicompressor.jar"/>
        <srcfile/>
        <arg line="-o"/>
        <mapper type="glob" from="*.js" to="*-min.js"/>
        <targetfile/>
    </apply>
</target>

Also consider the following ant target to minify CSS files:

<target name="js.minify" depends="js.preprocess">
    <apply executable="java" parallel="false">
        <fileset dir="." includes="*.css"/>
        <arg line="-jar"/>
        <arg path="yuicompressor.jar"/>
        <arg line="--line-break 0"/>
        <srcfile/>
        <arg line="-o"/>
        <mapper type="glob" from="*.css" to="*-min.css"/>
        <targetfile/>
    </apply>
</target>

Work around caching issues

Adequate cache control can really enhance your users’ experience by not having them re-download static content (usually JavaScript, CSS, HTML and images) when coming back to your site. This has a downside however: when rev’ing your application, you want to make sure your users get the latest static content. The only way to do that is to change the name of these files (for example by adding a time stamp to their file name, or using their checksum as their file name) You must also propagate that change to all the files that refer to them.

The copy and file name replacement will be handled by a custom Ant task named FileTransform (see FileTransform.java) The first step is to build the task from the source and define the custom task:

<target name="-setup.build.tools" depends="-load.properties">
    <mkdir dir="${build.dir}/tools/classes"/>
    <javac srcdir="tools/src"
           destdir="${build.dir}/tools/classes"
           includes="**/*.java">
        <classpath>
            <pathelement />
        </classpath>
    </javac>
    <taskdef name="FileTransform"
        classname="com.yahoo.platform.build.ant.FileTransform"
        classpath="${build.dir}/tools/classes"/>
</target>

(Note the use of a “-” in front of the name of the target. This is used to make the target “private” i.e. not invokable directly) The second part is the copy of the static content that is going to be cached using the newly defined FileTransform task:

<target name="-copy.js.files" depends="-setup.build.tools">
    <mkdir dir="${build.dir}/js"/>
    <FileTransform todir="${build.dir}/js"
        changefilenames="true"
        propertiesfile="${build.dir}/js.files.mapping.properties">
        <fileset dir="site/js" includes="*.js"/>
    </FileTransform>
</target>

The mapping between the old names and the new names is stored in a properties file. The final step is the copy of the files that refer to files which name was changed in the previous step. For this, we use the copy task:

<target name="-copy.php.files" depends="-copy.js.files">
    <mkdir dir="${build.dir}/php"/>
    <copy todir="${build.dir}/php">
        <fileset dir="site/php" includes="*.php"/>
        <filterset>
            <filtersfile file="${build.dir}/js.files.mapping.properties"/>
        </filterset>
    </copy>
</target>

You can download an archive containing a very simple Ant project illustrating this advanced technique.

Deploy your application

Deploying a web application is usually done by copying a set of files over to the production servers and running a list of commands on those servers (stop the services, run a few shell commands, and finally restart the services)

You have many options to copy files to remote servers with Apache Ant. You could use the copy task to copy files to a locally mounted file system, or you could use the optional FTP task. My personal preference is to use scp and rsync (both utilities are available on all major platforms) Here is a very simple example demonstrating the use of scp with Apache Ant:

<apply executable="scp" failonerror="true" parallel="true">
    <fileset dir="${build.dir}" includes="**/*"/>
    <srcfile/>
    <arg line="${live.server}:/var/www/html/"/>
</apply>

And here is an example showing how you can run remote commands:

<exec executable="ssh" failonerror="true">
    <arg line="${live.server}"/>
    <arg line="sudo webctl restart"/>
</exec>

Conclusion

I hope this article has given you some ideas of what’s possible to automate with modern build tools such as Apache Ant (other cool things include automatic generation of documentation, source control integration, packaging using rpm and such, etc.) My professional experience has taught me that the build process cannot be an afterthought. Just like performance, it should be part of your project from the very beginning. The second lesson is that all web developers need to be active participants in creating and maintaining the build process (with maybe one build engineer to centralize all the information) instead of relying on somebody else to build their component. Finally, keep your build process alive and maintain it actively as your needs will change during your project life cycle.