Repairing a broken grub.conf in CentOS

Today I got an error during updating Hyper-V kernel modules on CentOS 6.3. The upgrading process removes the already loaded kernel modules in a running system. This results in a kernel oops and a broken init ramdisk for the currently running kernel. Sometimes it happens that a kernel oops occures during writing the new grub configuration. On the next boot grub cannot find any configuration files and falls back to the grub shell. You will get a prompt like this:

grub >

Experts will be able to boot the system typing the commands for grub directly in the prompt and boot the system manually. But if you entered the lines and the system is up and running again, you won’t find any grub configuration.

On CentOS you are not able to generate a grub configuration from scratch via any scripts or tools like the Debian update-grub. CentOS itself uses grubby to generate the kernel entries. But grubby needs a template from which it can generate the entries. Last but not least this template will be generated by reading an existing grub entry. This approach ends up in a worst case scenario if you have misconfigured your /boot/grub/grub.conf, /etc/grub.conf or /boot/grub/menu.lst (the both last entries are only symlinks to /boot/grub/grub.conf) and you now try to reset your configuration. And again big trouble, if generating the grub.conf failed – for example during a kernel crash. In my situation it was a kernel crash.

It is absolutely impossible to regenerate a grub.conf from scratch with any of the tools delivered by CentOS. My solution:

  1. boot your system via Install-Disk or by grub command line prompt
  2. create an empty new /boot/grub/grub.conf
  3. add the next code snippet to your grub.conf
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-279.22.1.el6.x86_64)
  root (hd0,0)
  kernel /vmlinuz-2.6.32-279.22.1.el6.x86_64 ro root=/dev/sda3
  initrd /initramfs-2.6.32-279.22.1.el6.x86_64.img

N O T E:
I have a separated /boot partition on my systems. In standard configuration delivered by CentOS /boot and / will be on the same partition. In this case, the path to kernel and initrd will start with /boot/vmlinuz... and /boot/initramfs... . The root partition mostly will be root=/dev/sda1.

Try to boot your system with your manually built grub.conf. If anything works fine you can add new boot entries by CentOS’ tool grubby. For example:

root@host:~ $ grubby --add-kernel="/boot/vmlinuz-2.6.32-279.22.1.el6.x86_64"\
--initrd="/boot/initramfs-2.6.32-279.22.1.el6.x86_64.img"\
--title="CentOS (2.6.32-279.22.1.el6.x86_64)" --copy-default --make-default

The tool grubby will replace the /dev/sda? device file with the UUID string of the partition.
You can use the next line to generate an entry for each kernel image in /boot/:

for kernel in /boot/vmlinuz-*; do \
version=`echo $kernel | awk -F'vmlinuz-' '{print $NF}'`; \
grubby --add-kernel="/boot/vmlinuz-${version}" \
--initrd="/boot/initramfs-${version}.img" \
--title="CentOS (${version})" \
--copy-default --make-default; \
done

You should check the /etc/grub.conf for duplicate entries or maybe you will resort the boot order. Reboot your system to check if anything works fine again.

Flash Container Height in IE10

Recently we had a problem with our flash client container within Internet Explorer 10. Within the CSS the height and width of the container is set to 100% to consume all available space. Firefox, Chrome, Opera and IE (pre version 10) have no problem to size the flash container correctly. However, IE 10 shows the flash container about 30% of the height while consuming 100% of the width. The div container has 100% in height, but the included embed object has not.

Our html source looks like:

<div style="height: 100%; width: 100%;">
  < embed id="RewooClient" width="100%" height="100%"
    type="application/x-shockwave-flash" name="Rewoo" src="Rewoo.swf"
    allowfullscreeninteractive="true" allowscriptaccess="sameDomain" >
    <!-- ... -->
  </embed>
</div>

Internet searches did not help. It seems that nobody else have this problem. The pragmatic solution was to use jQuery to set the height manually on page load and on page resizes:

(function($) {
$(document).ready(function() {
  if ($.browser.msie && $.browser.version >= 10) {
    $('#RewooClient').attr('height', $(window).height());
    $(window).resize(function() {
      $('#RewooClient').attr('height', $(window).height());
    });
  }
})(jQuery);

[update 2013-05-13] Starting with jQuery 1.9 the $.browser feature was removed and you need to add the jQuery Migration plugin to get the code working again (Thanks to Ed Graham)[/update]

Maybe this might help you… And if you know a better solution, please drop a line.

Grails Webdav Plugin with Apache Shiro causes Hibernate LazyInitializationException

We use Grails 2.1 with Apache Shiro 1.1.3 as security layer which handles the user authentication such as login via web app or webdav. To provide an easy and flexible file service we use webdav plugin 3.0.1. It hides the complexity of the HTTP file protocol webdav and gives a simple interface to work with virtual network filesystem structure.

When a user tries to login, our SystemDBRealm authenticates the given user with its password against the database and does other checks as well. These checks include the validation of the user role. The user role is modeled via SystemUserRoleRel within our grails domain model. SystemUserRolRel has two fields: user and role which links the SystemUser and SystemRole domain models together. Hibernate loads these models lazy in special hibernate proxies. These model proxies are loaded on demand to proper domain model instances which requires a valid hibernate session.

To check that given user has a valid role we execute following code (the code is simplified to depict the problem):

def user = SystemUser.findByUsername(username)
def standardRole = SystemRole.findByName('Standard')
def hasStandardRole = SystemUserRolRel.findAllByUser(user).find { it.role == standardRole } != null

On a normal login via browser SystemDBRealm has a hibernate session and can resolve the it.role hibernate proxy within the find closure find { it.role == standardRole }.

In case of a login via Webdav (e.g. through cadaver, a command line webdav client), the user is authenticated via BasicHttpAuthenticationFilter of shiro. As described in the Basic HTTP Auth with Shiro in Grails this basic authentication filter is configured in config.groovy like this:

security.shiro {
        authc.required = false
        filter.config = """\
[main]
authcBasic = org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter
authcBasic.applicationName = Rewoo World

[urls]
/webdav/** = authcBasic
"""
}

The BasicHttpAuthenticationFilter extracts the user password token and returns it to shiro. Shiro processes this authentication token in the SystemDBRealm like the browser based login. But now the hibernate session is missing and the find closure find { it.role == standardRole } throws a hibernate LazyInitializationException: it.role can not be resolved.

To solve this issue we use the withNewSession closure of an arbitrary domain model class to wrap the authentication code defined above (in our case we choose the SystemUser class but another class should be ok as well):

SystemUser.withNewSession {
  def user = SystemUser.findByUsername(username)
  def standardRole = SystemRole.findByName('Standard')
  def hasStandardRole = SystemUserRolRel.findAllByUser(user).find { it.role == standardRole } != null
}

Now a hibernate session is bound to the closure and it.role hibernate proxy can be resolved again.

PS: A ticket is filed at GPWEBDAV-18 for this issue. Comments are welcome.

Flash ReferenceError #1069: topMostIndex not found

Today, I struggled loading our flex application. I always got the following error

ReferenceError: Error #1069: Property http://www.adobe.com/2006/flex/mx/internal::topMostIndex not found on _Main_mx_managers_SystemManager and there is no default value.

It take me some time to find the solution: This error is caused by the new deployed version 4.57 of FlashFirebug from November 17, 2012. The newest version 4.58 fix this problem but it is not reviewed and therefore not yet deployed via auto-update. You can get this version from Mozilla and you can install it manually by moving your mouse over the version description and clicking “Add to Firefox” button.

The weak performance of the Groovy minus operator

During a recent hunt for performance problems within our grails application we’ve discovered that using the minus operator to subtract one list from another one seems to perform badly for list items which doesn’t implement the Comparable interface. This post will show the supposed reason for this bottleneck.

Have a look at the following simple microbenchmark which shows the problem:

We have a very simple pogo class which holds some data about books:

class Book {
    final String name
    final String author

    public Book(String  name, String author) {
        this.name = name
        this.author = author
    }

    public String toString() {
        return "Title: ${name} Author: ${author}"
    }
}

This Book class is used in an abstract unit-test to create one list of books, consisting of a configurable number of tech books and novels. The test cases will subtract the novels from the list: one time via an old school Java “removeAll” call, the second time via the Groovy minus-operator.

abstract class ListPerformanceTest extends TestCase {
    protected List<Book> novels
    protected List<Book> techBooks

    protected void setUp() {
        novels = createNovelsForFixture()
        techBooks = createTechBooksForFixture()
    }

    public void testGroovyMinus() {
        List<Book> allBooks = novels + techBooks
        measureTimeInMS("Subtracting ${novels.size()} novels from the list of ${allBooks.size()} books") {
            allBooks = allBooks - novels
        }
        assertEquals(techBooks.size(), allBooks.size())
    }

    public void testJavaRemoveAll() {
        List<Book> allBooks = novels + techBooks
        measureTimeInMS("Removing ${novels.size()} novels from the list of ${allBooks.size()} books") {
            allBooks.removeAll(novels)
        }
        assertEquals(techBooks.size(), allBooks.size())
    }

    public long measureTimeInMS(String description, Closure toMeasure) {
        long nanoTimeStart = System.nanoTime()
        toMeasure.call()
        long nanoTimeEnd = System.nanoTime()
        long result = (nanoTimeEnd - nanoTimeStart) * 1E-6
        println description + " took ${result} ms"
        return result
    }

    protected abstract List<Book> createNovelsForFixture()
    protected abstract List<Book> createTechBooksForFixture()
}

We created three subclasses: a SmallTestCase class (subtracting 1000 novels from a 2000 books’ list), a MediumTestCase (subtracting 2000 novels from a 4000 books’ list) and finally a LargeTestCase (subtracting 4000 novels from the list containing 8000 books).

Following you can find the results of the test runs:
Statistic Groovy minus

We performed the tests within the following environment:
Win-7 64 bit, JDK 1.6.0.33 (32 bit), Intel i5-2500 (4 cores, 3.3GHz per core), 8GB RAM

You could argue that it is not fair to compare both methods: the Groovy way will return a new list, the removeAll-method of Java uses an in-place-removement, so the original list will be lost. But just the fact that Groovy is creating a new list can not explain why the minus operator is so expensive: for the small list with 2000 books Groovy is about 1700 times slower than Javas’ removeAll, for the medium list even over 3000 times. And remember: all of this will only happen when your item class doesn’t implement Comparable.

That’s why we decided to do some more research to get an explanation. After some runs with JProfiler and the medium-sized test-case we found out that one of the biggest hotspots was the toString()-method of the Book class. According to the profiler it was called nearly 12 million times while performing the minus operation and burnt about 15% of the overall time. Because the profiler did not tell us anything about other hotspots we started with this trace and decided to take a look at the implementation of the minus operator within the Groovy source code under “Source release” of version 1.8.8.

We found the following implementation:

public class DefaultGroovyMethods extends DefaultGroovyMethodsSupport {
...
   public static <T> List<T> minus(List<T> self, Collection<?> removeMe) {
	...
        Comparator<T> numberComparator = new NumberAwareComparator<T>();
        if (nlgnSort && (self.get(0) instanceof Comparable)) {
		// Not interesting in our case, because Book doesn't implement Comparable
		...
        } else {
            //n*n version
            List<T> tmpAnswer = new LinkedList<T>(self);
            for (Iterator<T> iter = tmpAnswer.iterator(); iter.hasNext();) {
                T element = iter.next();
                boolean elementRemoved = false;
                for (Iterator<?> iterator = removeMe.iterator(); iterator.hasNext() && !elementRemoved;) {
                    Object elt = iterator.next();
                    if (numberComparator.compare(element, (T)elt) == 0) {
                        iter.remove();
                        elementRemoved = true;
                    }
                }
            }
            ...
        }
    }
...
}

At the first glance we did not find anything special about the code: it has a worst case scenario of O(n*m) and uses the so called “NumberAwareComparator” to decide, if two elements of the list are equal or not. So, let’s proceed with this Comparator:

public class NumberAwareComparator<T> implements Comparator<T> {
    public int compare(T o1, T o2) {
        try {
            return DefaultTypeTransformation.compareTo(o1, o2);
        } catch (ClassCastException cce) {
            /* ignore */
        } catch (GroovyRuntimeException gre) {
            /* ignore */
        }
        // since the object does not have a valid compareTo method
        // we compare using the hashcodes. null cases are handled by
        // DefaultTypeTransformation.compareTo
        int x1 = o1.hashCode();
        int x2 = o2.hashCode();
        if (x1 == x2) return 0;
        if (x1 < x2) return -1;
        return 1;
    }
}

You can find here a typical code smell in form of a misuse of the exception concept (see this refactoring catalogue for details): the comparator doesn’t check if the two instances can be compared at all by the DefaultTypeTransformation class before calling its compareTo method. Instead, the code relies upon the fact that DefaultTypeTransformation#compareTo will throw an exception whenever the list items can not be compared. In our case this will happen for nearly all Book items (except those pointing to one and the same reference). This leads to the following approximation of thrown exceptions:
Number of exceptions thrown by minus

So, in our example we will have about 6 million exceptions which will be thrown and afterwards just be ignored. Because the profiler ignores the overhead produced by the exceptions we did not have a chance to see this: the only trace for this problem were the 12 million calls of toString.

But everything we said up to now doesn’t explain them. To understand why we call toString so often, you just have to take a short look how the exception thrown by DefaultTypeTransformation#compareTo is constructed:

public class DefaultTypeTransformation {
...
    private static int compareToWithEqualityCheck(Object left, Object right, boolean equalityCheckOnly) {
        if (left == right) {
            return 0;
        }
	...
        throw new GroovyRuntimeException("Cannot compare " + left.getClass().getName() + " with value '" +
                left + "' and " + right.getClass().getName() + " with value '" + right + "'");
    }
}

As you can see here Groovys’ minus doesn’t only throw 6 million exceptions in our case but also use a string concatenation of the left and right element to construct the exception message, i.e. for each exception we will call toString() of the Book class twice.

Conclusion

You shouldn’t use the minus operator if you have lists with more than some hundred elements which don’t implement Comparable. We’ve tested this only with Groovy 1.8.8, because this is the version shipped with the current Grails version 2.1.1. But after a short look at the current Groovy 2.0.5 code it seems that the performance leak should still exist. And according to this jira issue we are not the only one having problems with the collection performance of Groovy. To avoid the minus operator just for performance reasons is a real pitty, because the minus syntax is not only much more readable than the method call but feels also “cleaner” in terms of having no side effects on the list it is called at.

How to use gorm after a http session timeout

In the previous part we’ve already discussed how to get informed about “regular” logout- and timeout-events in order to maintain a list of currently logged in users. But in our use-case we not only want to maintain a simple list, but also want to do some database cleanup queries whenever a user logs out (in our case: we have to remove some “locking” entries, identifiying resources exclusively locked by the current user).

We’ve already identified the two different places where we have to handle the logout event:

  1. The “onLogout” method of the RewooAuthenticationListener (informed by shiro when a regular logout occurres)
  2. The “sessionDestroyed” method of the RewooHttpSessionListener (informed by the servlet container when a timeout occurres)

The first scenario can be handled quite easily: because we are informed about the event by the shiro plugin we will have a fully initialized hibernate session which can be used to clean up your tables in the usual way:

public class RewooAuthenticationListener implements AuthenticationListener {
...
    public void onLogout(PrincipalCollection principalCollection) {
        // Maintain your user list as described in the previous post
        // ...
        // Do the clean up now
        SystemUser user = SystemUser.findByName(userName)
        ExclusiveLocks.findAllByUser(user) {
            it.delete()
        }
    }
...
}

The second one is the tricky part: because the method is called by the servlet container we will not have any valid hibernate session bound to the SessionHolder of the thread-context. So, the Gorm calls will break with a “No hibernate session bound to thread” exception The first solution which came into our minds was, to retrieve a new session by wrapping the routine in a simple withNewSession-closure call, like this:

public class RewooHttpSessionListener implements HttpSessionListener {
...
    public void sessionDestroyed(HttpSessionEvent event) {
        // Maintain your user list as described in the previous post
        // ...
        // Do the clean up now
        ExclusiveLocks.withNewSession {
            SystemUser user = SystemUser.findByName(userName)
            ExclusiveLocks.findAllByUser(user) {
                it.delete()
            }
        }
    }
    ...
}

In fact this solution will work for the FIRST timeout event. But afterwards you will get the following exception each time a new timeout occurres:

org.springframework.dao.DataAccessResourceFailureException: Could not open Hibernate Session; nested exception is org.hibernate.SessionException: Session is closed!

After some research and digging into the grails / hibernate code we discovered that after leaving the withNewSession closure (which resides within a grails class called “HibernatePluginSupport”) the session is closed but not removed from the SessionHolder. When the next timeout occurres the old session will be found by hibernate and the exception will be thrown. Interestingly enough the clean up code used by grails 1.3.5 is significantly different to the new cleanup algorithm used by HibernateGormEnhancer (which replaces HibernatePluginSupport in newer grails versions). Because of some compatibility issues we can not upgrade to a newer version of grails at the moment so we can not say if the problems described here will vanish with a newer grails release.

We’ve searched around for a while to find a possibility to remove the outdated session from the session holder, but we did not found any proper way to accomplish this:

  • you can not access the underlying sessionMap directly, because it is a private variable and for encapsulation purposes there is no getter
  • you can not use the removeSession method, because you have to pass in the key of the session to remove: unfortunately this key is not the session itself but a private object created by SessionHolder during instantiation of the holder and there is no way to retrieve it
  • even calling the clean method of the SessionHolder will not clean the internal map

Finally we found a solution: by calling the SessionHolder#getValidatedSession you will only get the current session, if the session is not already closed. In case it is closed, it will be removed from the internal map as a side effect. So, that is exactly what we want to have in our case but be aware that this is a very ugly workaround. It depends upon an implementation detail of a class which is not under your control.

Our solution leads us to the following method which can be used to clean up the hibernate session:

private void cleanUpHibernateSession() {
    SessionHolder sessionHolder = (SessionHolder) TransactionSynchronizationManager.getResource(SessionGrabber.getSessionFactory())
    if (sessionHolder != null &amp;&amp; !sessionHolder.isEmpty()) {
        // Calling this method will "remove" already closed sessions as a side-effect (so we don't care about the return-value)
        sessionHolder.getValidatedSession()
        if (sessionHolder.isEmpty()) {
            sessionHolder.clear()
        }
    }
}

The method will be called after the withNewSession closure call like this:

public class RewooHttpSessionListener implements HttpSessionListener {
...
    public void sessionDestroyed(HttpSessionEvent event) {
        // Maintain your user list as described in the previous post
        // ...
        // Do the clean up now
        ExclusiveLocks.withNewSession {
            SystemUser user = SystemUser.findByName(userName)
            ExclusiveLocks.findAllByUser(user) {
                it.delete()
            }
        }
        cleanUpHibernateSession()
    }
    ...
}

This will make sure that the next call of withNewSession will not be disturbed by an already closed session.

Conclusion

The solution we’ve presented here give you the possibility to clean up parts of your database each time a user logs out. We don’t feel very comfortable with our workaround for the SessionHolder problem: if you know a better solution we would be glad to hear about it!

Maintain a user list with grails

Preface

This is the first part of a blog entry about the problems we had with grails 1.3.5 in conjunction with shiro to retrieve and maintain a simple list with the currently logged in users. The first part will explain how to keep the list up to date. The second part will describe a solution we use to interact with the grails-environment in case of an automatic logout to remove some rows from the database.

How to take care of automatic timeouts for a active user list

As already mentioned we have the following (as we thought very simple) use-case for our web-application: We want to have an overview-page containing a list of all currently logged in users. To accomplish this, it should be sufficient to catch the following events:

  • Whenever a user logs in, add him to a internal list of logged in users
  • Whenever a user logs out, remove him from the list and release all resources locked by him (in our case: remove a row from a table. This will be the topic of part two)

Of course, we want the same behaviour for an automatic logout (which should take place after some minutes of inactivity).

To secure our app we use the shiro-plugin based on the apache shiro project. In fact, shiro already offers an easy-to-implement interface to inform our app about login and logout events called “AuthenticationListener”. So, we started with the following straight-forward implementation (based on this discussion), which has to be registiered in the resources.groovy file:

public class RewooAuthenticationListener implements AuthenticationListener {
    private static Collection activeUsers = Collections.synchronizedList(new ArrayList());

    @Override
    public void onSuccess(AuthenticationToken authenticationToken, AuthenticationInfo authenticationInfo) {
        PrincipalCollection collection = authenticationInfo.getPrincipals();
        activeUsers.add((String) collection.getPrimaryPrincipal());
    }

    @Override
    public void onFailure(AuthenticationToken authenticationToken, AuthenticationException e) { }

    @Override
    public void onLogout(PrincipalCollection principalCollection) {
        PrincipalCollection collection = authenticationInfo.getPrincipals();
        activeUsers.remove((String) collection.getPrimaryPrincipal());
    }

    public static Collection getActiveUsers() {
        return Collections.unmodifiableList(activeUsers);
    }
}

In case of a “normal” logout this solution works pretty well. But whenever a user just forgets to logout properly (like it is usual for hard working days) and the timeout mechanism has to take place our methods aren’t called at all. After carefully reading the javadoc we recognized the following statement: “This method will only be triggered when a Subject explicitly logs-out of the session. It will not be triggered if their Session times out.”

Therefore, the automatic timeout is neither triggered by your grails app nor by grails at all. Instead, it is your servlet-container which recognizes the connection timeout and cleans up the session. So we’ve searched for another possibility to get informed about the timeout (in this case through the servlet-container itself). The standard way to achieve this is via an own implementation of an HttpSessionListener which has to be registered within web.xml like the following:

...
    <listener>
        <listener-class>com.rewoo.listeners.RewooHttpSessionListener</listener-class>
    </listener>
...

The registered RewooHttpSessionListener looks like this:

public class RewooHttpSessionListener implements HttpSessionListener {
    private static List activeUsers = Collections.synchronizedList(new ArrayList());

    @Override
    public void sessionCreated(HttpSessionEvent event) { }

    @Override
    public void sessionDestroyed(HttpSessionEvent event) {
        String userName = getCurrentUserName(event)
        if (userName == null) {
            return
        }
        RewooHttpSessionListener.userLoggedOut(userName)
    }

    public static void userLoggedIn(String userName) {
        activeUsers.add(userName)
    }

    public static void userLoggedOut(String userName) {
        activeUsers.remove(userName)
    }

    public static List getCurrentUserNames() {
        return Collections.unmodifiableList(activeUsers);
    }

    private String getCurrentUserName(HttpSessionEvent event) {
        PrincipalCollection currentUser = (PrincipalCollection) event.getSession().getAttribute(DefaultSubjectContext.PRINCIPALS_SESSION_KEY);
        if (currentUser == null) {
            return null
        }
        return (String) collection.getPrimaryPrincipal();
    }
}

As you can see, we’ve moved the activeUsers list to the SessionListener and added public methods which can be used by the RewooAuthenticationListener to inform us about regular login / logout events recognized by shiro. We better don’t use the sessionCreated method here to update our user list, because it will be called each time some client connects to the server (and establishes a http session), no matter if the user has been logged in correctly or not. Thus, we better handle this within our RewooAuthenticationListener.

In case of a sessionDestroyed event, we have to differ between two situations:

  • a normal logout occurred: in this case shiro has already cleaned up the current session, so the HttpSessionEvent will not contain the name of the previously logged in user
  • a timeout occurred: in this case shiro hasn’t performed any clean up operation upon the session, so we can read out the name of the previously logged in user

This leads us to the following, new implementation of the RewooAuthenticationListener:

public class RewooAuthenticationListener implements AuthenticationListener {
    @Override
    public void onSuccess(AuthenticationToken authenticationToken, AuthenticationInfo authenticationInfo) {
        RewooHttpSessionListener.userLoggedIn((String) authenticationInfo.getPrincipals().getPrimaryPrincipal());
    }

    @Override
    public void onFailure(AuthenticationToken authenticationToken, AuthenticationException e) { }

    @Override
    public void onLogout(PrincipalCollection principalCollection) {
        RewooHttpSessionListener.userLoggedOut((String) principalCollection.getPrimaryPrincipal());
    }
}

Conclusion

To detect manual and automatic logouts, you have to combine the AuthenticationListener of shiro with the HttpSessionListener of your servlet api. Manual logouts will be handled by shiro, automatic logouts (due a timeout) by the session listener.

Improving performance of Jenkins builds

For building our software we use the continuous integration build server solution Jenkins CI. In the last week we got problems with the build duration. After analyzing the build progress we found some hints for improving the build performance.

Bottlenecks during compiling

The first hint was the duration time of compiling the source code. Before we added a new HDD, build time over all was 11 minutes. After them build time increased to 15 minutes! Another build job which only compiled an artifact without executing any tests needed 3 minutes more – in comparison with the older configuration.
So we analyzed the complete build progress step by step and found a bottleneck. Our software using grails during build progress. For compiling the source code into java bytecode, jenkins writes into .grails in the home directory. We moved the complete home onto a SSD and got back the lost 3 minutes.

Bottlenecks during testing

Another bottleneck is the working directory in Jenkins called “workspace”. Each job gets its own workspace. During generating the artifacts the complete source code repository will checked out into this workspace. Any test result, compiled class and the artifact itself will be buffered in the workspace directory. After build was successful the artifacts and the test results will be copied to jobs directory. If it is possible to mount the workspace of a build job on a SSD you will get any better performance.

The myth of SSD vs. tmpfs

Maybe in your opinion, it could be much better to mount the workspace and .grails directory and any other highly used directories on a tmpfs. We tried this option and the result is: Yes it will decrease the build time. But not as much as you think. Mounting the workspace and the .grails directory on a tmpfs decreased the build time at maximum 10 seconds in comparison to a SSD. The improvement to mount these directories on a tmpfs would be in reducing the write cycles on the SSD which increases the lifetime of a SSD.

Resume

It is possible to impove the duration time of a Jenkins build. The first approach to improve the duration time of a build is to get out in which directories will jenkins write/delete/copy files. In our example we improved the build time by 33% moving the workspace and all other directories on a SSD.
Another possibility to increase the build time could be a faster CPU. This option would need a build server (maybe a jenkins slave) with a desktop CPU. Because the Intel Xeons are much slower than the most high end desktop CPUs. We didn’t tested this, but one of our developer workstations with an Intel i7 CPU only needs 80% of the time for building the same artifact.

Solved problems updating perl-XML-SAX-0.96-7.el6.noarch on CentOS 6

Today yum chase its own tail during the package update of one of our servers.

Updating the packages needed the installation of any new packages because of dependencies. The transaction check breaks with:

Transaction Check Error:
  file /usr/share/man/man3/XML::SAX::Base.3pm.gz conflicts between attempted installs of perl-XML-SAX-0.96-7.el6.noarch and perl-XML-SAX-Base-1.04-1.el6.rf.noarch
  file /usr/share/man/man3/XML::SAX::Exception.3pm.gz conflicts between attempted installs of perl-XML-SAX-0.96-7.el6.noarch and perl-XML-SAX-Base-1.04-1.el6.rf.noarch

No way out. Google didn’t know anything about.

The solution of this problem is removing any packages and reinstall them manually. In my situation I removed both packages which made trouble: perl-XML-SAX-0.96-7.el6.noarch and perl-XML-SAX-Base-1.04-1.el6.rf.noarch. I reinstalled perl-XML-SAX-0.96-7.el6.noarch again and any transaction check works and yum installed the updates. I didn’t reinstalled the package perl-XML-SAX-Base-1.04.1.el6.rf.noarch because it seems to be no longer a dependency of any perl-SAX package.

Good font for coding

Some days ago Adobe announced a new monospaced OpenType font Source Code Pro designed for coding environments. I have used Menlo for years now and was curious if I can see any benefits from the new font for coding.

I find the described advantages of Source Code Pro in Menlo too: readability of potentially confusable characters like iI1 or O0 and readability of punctuation marks, brackets and mathematical symbols. But with the same font size the character height of Source Code Pro is smaller but the line height is larger. Take a look at a code snippet in 12pt, screenshot height 200px

Source Code Pro 12 pt

Menlo 12 pt

If you increase the real font size to get the same perceived font size the same code needs more screen width and height with Source Code Pro than with Menlo. And the additional line spacing is needless because you can configure the line spacing in most of the IDEs.

For me it’s very important to get as many information on the screen as possible. I don’t like to scroll to see the context or to read the line to its end. Sure, good code is short. A function has only twenty lines of code with no line longer than 120 characters and a class ends after two screens. But in real life you have to read some code from your colleagues. And be honest: not every line you wrote follows the rules of clean code…

I like the Menlo. It looks more elegant to me and I see more code. I will continue using it for coding.