Executing Grails tests in parallel

As our Grails-backend continues to grow our test count increases, too. With the build times on our Hudson continuous integration (CI) server reaching around 13 minutes we decided to spend some resources on making the builds faster. Some speedup was achieved by upgrading the hardware and optimizing specific integration test and production code that slowed them down. Another big chunk of build time was cut off by upgrading Grails from 1.3.2 to 1.3.5 which provided a speedup of approx. 25%. All these measures together brought the build time down to around 9 minutes again. The speedup was nice but the feedback cycle still felt to long for us. The integration tests are the biggest factor still with around 6 minutes.

So we had a look into how to run the integration tests in parallel instead of leaving some cores of the build server idle. Ted Naleid provided a nice starting point for our efforts. But since our build is based on Apache Ant and we have a big runtime discrepancy between our unit and integration tests his approach was not enough for us.

We first translated his solution to our build.xml and then separated the integration tests in two groups with similar runtime. We start the unit tests, integration test group 1 and integration test group 2 in parallel. This currently means three compiles to different directories and parallel test execution. On our quad core server the build time is significantly improved despite the extra work. The integration tests are heavily CPU bound in our case. We moved some static code analysis and test coverage reports to a nightly build to improve the feedback cycle even further. The end result is a build time around 6-7 minutes which may not be lightning fast but still ways better than the 13 minutes before our efforts. See the relevant ant targets below.

<target name="-call-grails-with-testdir">
  <chmod file="${grails}" perm="u+x"/>
  <property name="target.dir" value="target/test-${testdir}" />
  <exec dir="${basedir}" executable="${grails}" failonerror="true">
    <arg value="-Dgrails.work.dir=${target.dir}"/>
    <arg value="-Dgrails.project.class.dir=${target.dir}/classes"/>
    <arg value="-Dgrails.project.test.class.dir=${target.dir}/test-classes"/>
    <arg value="-Dgrails.project.test.reports.dir=target/test-reports"/>
    <arg value="${grails.task}"/>
    <arg value="${task.param}"/>
    <env key="GRAILS_HOME" value="${grails.home}"/>
  </exec>
</target>

<target name="parallel_test" description="--> Run tests in parallel">
  <path id="slowtests.path">
    <fileset dir="${basedir}/test/integration">
      <include name="SlowTests.groovy" />
      <include name="AnotherSlowTest.groovy" />
      <include name="MoreSlowTests.groovy" />
   </fileset>
  </path>
  <path id="normal_tests.path">
    <fileset dir="${basedir}/test/integration">
      <include name="**/*Tests.groovy" />
      <include name="**/*Test.groovy" />
      <exclude name="SlowTests.groovy" />
      <exclude name="AnotherSlowTest.groovy" />
      <exclude name="MoreSlowTests.groovy" />
    </fileset>
  </path>
  <pathconvert property="slowtests" pathsep=" ">
    <path refid="slowtests.path" />
    <chainedmapper>
      <flattenmapper />
      <compositemapper>
        <globmapper from="*Test.groovy" to="*" />
        <globmapper from="*Tests.groovy" to="*" />
      </compositemapper>
    </chainedmapper>
  </pathconvert>
  <pathconvert property="normaltests" pathsep=" ">
    <path refid="normal_tests.path" />
    <chainedmapper>
      <flattenmapper />
      <compositemapper>
        <globmapper from="*Tests.groovy" to="*" />
        <globmapper from="*Test.groovy" to="*" />
      </compositemapper>
    </chainedmapper>
  </pathconvert>
  <parallel>
    <antcall target="-call-grails-with-testdir">
      <param name="grails.task" value="test-app"/>
      <param name="task.param" value="-unit -xml" />
      <param name="testdir" value="unit" />
    </antcall>
    <antcall target="-call-grails-with-testdir">
      <param name="grails.task" value="test-app"/>
      <param name="task.param" value="-integration -xml ${slowtests}" />
      <param name="testdir" value="integration" />
    </antcall>
    <antcall target="-call-grails-with-testdir">
      <param name="grails.task" value="test-app"/>
      <param name="task.param" value="-integration -xml ${normaltests}" />
      <param name="testdir" value="integration2" />
    </antcall>
  </parallel>
</target>

Conclusion
Although Grails does not provide native support for parallel execution of tests (especially integration tests) it can pay off to do it manually using your build system. In our case it chopped away a good part of our build time and brought down test execution duration from around 6 to 4 minutes in spite of extra compilation runs. We may refine our approach further but hope that grails will ultimately support parallel test execution natively sometime soon.

Advertisements

2 thoughts on “Executing Grails tests in parallel

  1. Hi,
    When integration tests are run in parallel, how did you deal with db concurrency exceptions ?
    I see a deadlock happening and tests fail to acquire lock if other test already has lock on it.

    So, we are thinking of using separate DB for each of the integration jobs.

    Any thought ?

    Thanks for the article. we are running into similar situation as our test code is growing.

    –Venkat.

  2. Hi,

    each of the parallel grails-instances should use its own in-memory hsqldb-instance so there shouldn’t be any problems regarding database concurrency. Maybe you use one central database server for the tests?

    But we’ve encountered some other problems with the solution described in the article: all instances write to one and the same test result file which leads to a corrupted xml. Up to now we haven’t found a workaround for this.

    Dirk

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s