Easing The Pain Of The Big Upgrade

We recently released a new major version our internal microservice framework (Crampon) that included lots of major changes. In this release we made the jump to Java 11 (from 8) which meant upgrading Spring Boot to 2.x (from 1.5), Spring to 5.x (from 4.x) and a whole bunch of downstream libraries all at the same time because of Java 9+ and Spring 5 compatibility.

Normally we like to release changes in smaller pieces but unfortunately the amount of interdependencies on various versions made that more or less impossible. Since we were already releasing a “Major” version we took the opportunity to make several other breaking changes at the same time such as a restructuring of Crampon itself, switching security frameworks, and updating to JUnit 5.

Suffice to say application owners were in for a fair amount of work to do the upgrade. When you multiply the amount of work by the number of applications needing to upgrade (~150) that’s a lot of manpower, particularly when most of the changes are very simple and repetitive. In order to make the upgrade a lot easier we knew we needed to invest heavily in upgrade tooling.

Overview of Crampon Upgrade Tooling

We’ve had a system in place to automate upgrades as much as possible for several Crampon versions. For each new Crampon release we just have to write the transformers to upgrade the target application’s codebase (the hard part). The system works like this:

• Each version of Crampon contains transformer code in the “upgrade” CLI command
• Upgrader script runs in Jenkins:
  • Ran weekly or on-demand
  • Looks for latest released version of Crampon and runs CLI upgrade command
    • Updates pom versions to target Crampon version
    • Runs all upgrader transformers in order that apply to versions between starting version and new upgraded version
  • Commits changes to git branch which triggers a Jenkins build that generates a new dynamic environment for the branch and runs tests.
  • Opens a Pull Request in BitBucket with Application Maintainers as reviewers

When this all works smoothly it makes for a very painless upgrade process, when the job runs and does an upgrade you get a pull request in your inbox with 3 green checkmarks signifying a successful Build, Deploy, Test cycle on the branch and you can simply merge it and watch it as it makes it’s way through the Deployment Pipeline.

The Hard Part

Since we have that framework in place the only thing we have to worry about it is writing transformation code that will do its best to migrate the application to the new version. Unfortunately this is easier said than done, particularly for a change as big as we made.

Text Based Strategies

For simple changes such as updating imports we just use basic text replacement. In the case of class moves we generated a mapping for fully qualified names in the old version to the new name and String.replace. Our upgrade scripts use Groovy since it works well for “scripty” tasks like regex and text manipulation and integrates with all of our Java code.

def updateImports(Path path) {
    Files.walk(path)
            .filter { it.toString().endsWith('.java') }
            .forEach { f ->
        def text = f.text
        def referenced = importReplacements.findAll { text.contains(it.key) }
        if (!referenced.isEmpty()) {
            referenced.each { from, to ->         
                text = text.replace(from, to)
            }
            f.text = text
        }
    }
}

Manipulating POM files

To update the contents of a pom.xml file we take advantage of Groovy’s excellent xml support. In order to preserve all comments and whitespace we use the groovy.xml.DOMBuilder and groovy.xml.dom.DOMCategory which allows using GPath syntax to modify the xml contents. This is an example of some code that adds a property named java.memory to the properties section of a pom.xml if it doesn’t already exist.

def addDefaultHeapProp(Path pomFile) {
    PomUtils.updatePom(pomFile) { root ->
        if (!root.get('properties').isEmpty() && root.get('properties').'java.memory'.isEmpty()) {
            def propChildren = root.get('properties')[0].children()
            propChildren.item(propChildren.getLength() - 1).plus { 'java.memory'('-Xms1024m -Xmx1024m') }
        } else if(root.get('properties').isEmpty()) {
            root.children().item(root.children().getLength() - 1).plus() {
                'properties' {
                    'java.memory'('-Xms1024m -Xmx1024m')
                }
            }
        }
    }
}

Spoon to the Rescue!

In the past we wrote most of the transformations with regex or other text replacement techniques. It’s simple and works well for more basic transformations like renaming or moving classes. For more complicated transformations like changing

// JUnit4 syntax
@Test(expected=SomeException.class)
public void exceptionTest() {
  ...
}

to

// JUnit5 syntax
@Test
public void exceptionTest() {
   assertThrows(SomeException.class, () -> {
        ...
   });
}

using regex is very error prone and difficult to get right. Fortunately we discovered Spoon. In their own words:

Spoon is a library to analyze, transform, rewrite, transpile Java source code. It parses source files to build a well-designed AST with powerful analysis and transformation API.

It’s a very powerful library that allowed us to do very precise analysis and transformations but unfortunately has a steep learning curve. The transformation APIs in particular aren’t well documented, and it has a few quirks, but it enabled us to automate the upgrade to a degree we never would have been able to do without it.

To use Spoon you create a Launcher instance:

Launcher launcher = new MavenLauncher(projectDir, SOURCE_TYPE.ALL_SOURCE)

In this case it’s the MavenLauncher which parses the maven pom.xml files to get the classpath to resolve references against.

launcher.environment.autoImports = true
launcher.environment.prettyPrinterCreator = {
  new SniperJavaPrettyPrinter(launcher.environment)
}

Setting autoImports = true makes spoon automatically manage the import statements. The SniperJavaPrettyPrinter makes the output attempt to only write out the changed Java elements. The DefaultJavaPrettyPrinter on the other hand will write out the entire file completely disregarding any existing formatting and does things like excessive parenthesis usage, fully qualifying many references, and making every statement one line (300+ character lines sometimes!). Unfortunately the SniperJavaPrettyPrinter is a little buggy and seems to not handle certain more complicated codebases and things such as statically imported methods sometimes break it so we had to make our transformations attempt with the SniperJavaPrettyPrinter first, falling back on the default one if it fails.

Here’s an example of a processor we wrote that converts test methods to Junit5 from JUnit4 (this is just one of several required to do the conversion):

class TestMethodConverter extends MethodAnnotationReplacementProcessor {

      ...

      @Override
      void onMatch(CtMethod method, CtAnnotation originalAnnotation, CtAnnotation replacementAnnotation) {
          def wildcardImport = factory.createTypeMemberWildcardImportReference(factory.createCtTypeReference(Assertions.class))
          factory.CompilationUnit().getOrCreate(method.getDeclaringType()).getImports().add(factory.createImport(wildcardImport))
          def expectedException = originalAnnotation.getValue("expected")
          if (expectedException.toString() != 'Test.None.class') {
              def newBody = factory.createBlock()
              newBody.insertBegin(createAssert('assertThrows', method.getFactory(),
                                               [expectedException, factory.createLambda().setBody(method.body)]))
              method.setBody(newBody)
              println "  wrapping in assertThrows(${expectedException}, () -> ...)"
          }

          method.getElements(new TypeFilter(CtInvocation.class)).stream()
                .filter { isAssert(it) }
                .forEach { CtInvocation assertInvocation ->
              println "  converting $assertInvocation to junit5"

              def args = assertInvocation.arguments
              def assertName = assertInvocation.executable.simpleName
              if (assertName ==~ /assert(?:Not)?(?:Equals|Same)/ && args.size() == 3 && args[0].type.simpleName == 'String') {
                  args.add(args[0])
                  args.removeAt(0)
              }

              if (assertName == 'assertThat') {
                  def replacementAssert = SpoonUtils.createInvocation(factory, true, MatcherAssert.class, assertName, args)
                  assertInvocation.setExecutable(replacementAssert.executable)
                  assertInvocation.setTarget(replacementAssert.target)
              } else if(args == null || args.isEmpty()) {
                  def replacementAssert = createAssert(assertName, assertInvocation.getFactory())
                  assertInvocation.setExecutable(replacementAssert.executable)
                  assertInvocation.setTarget(replacementAssert.target)
              } else {
                  def replacementAssert = createAssert(assertName, assertInvocation.getFactory(), args)
                  assertInvocation.setExecutable(replacementAssert.executable)
                  assertInvocation.setTarget(replacementAssert.target)
                  assertInvocation.setArguments(replacementAssert.arguments)
              }
          }
      }

      ...

}

NOTE: This example is truncated, the whole processor and the others we used to upgrade to JUnit5 can be found here. It also references some helpers that can be found here: https://github.com/rei/spoon-utils.

Querying With Spoon

The other very powerful thing about Spoon is the querying ability. Because it parses the actual AST of the code you can query for specific patterns of code. It includes a powerful query API that allows you to chain together queries in a similar manner to the Java Streams API.

A simple example is querying for usage of an annotation:

if (!model.filterChildren(new AnnotationFilter<>(Audited)).list().isEmpty()) {
    requiredModules.add('auditing')
}  

You can also write custom Filter implementations such as this one that looks for invocations of three methods on a type:

class CacheConfigReferenceFilter implements Filter<CtExecutableReference<?>> {
    def methodNames = ['invalidationOnly', 'redisBackedInvalidation', 'asyncReplicated'] as Set

    @Override
    boolean matches(CtExecutableReference<?> element) {
       return element.getDeclaringType()?.getQualifiedName() == CacheConfigurationBuilder.class.name &&
               methodNames.contains(element.getSimpleName())
    }
}

The snippet of code that filter finds typically looks something like this:

@Bean
public CacheConfigurationCustomizer cacheConfig() {
    return cb -> cb.define("cacheName", cb.invalidationOnly().expiration().lifespan(30, SECONDS));
}

That method invocation would be very difficult to correctly find using regular expressions or other text based options which is the real power of using a library like Spoon.

Wrapping Up

By combining these techniques we were able to totally automate the upgrade for lots of the simpler services and significantly reduce the amount of manual work necessary for the rest. Because of the sheer number of applications we had to upgrade it was definitely worth investing in the automation. It has definitely made a big difference in the number of people who have done the upgrade already, despite the large number of changes.
While the initial investment can be a bit daunting, when you have 150+ microservices saving even an hour or two of engineer time per app adds up to quite a lot.