Wednesday, July 08, 2015

Integration Testing - Real World Challenges


Testing is an integral part of any software development project. While everyone understands and acknowledges the importance of Unit testing, Integration testing is also equally important in an Enterprise software development project. Integration testing is the kind of testing where multiple parts/modules of a software system is tested as a group. The multiple modules could be different layers of the system including the database or it could be the application together with external systems.

Enterprise applications often have lots of integration with external systems, systems that are often out of your control. These external systems may not be consistent when it comes to availability, performance or state, posing challenges to day to day running of an integration test suite, continuous integration/deployment set ups or maintenance of an integration test suite.

Whether it’s integration testing covering different parts of your own application or covering external systems, integration tests possess different challenges that is not seen in Unit testing.

I’d like to share with you some important learnings around testing in a large Enterprise application development project.

It’s not just a ‘test’

Testing is not all about making sure your application is in good order. In a big enterprise applications, tests are a good mechanism of understanding how different parts of your system work. It’s essentially a type of documentation for developers while they are developing it. Further it’s a great tool for developers and support personnel during the maintenance phase. This is especially important considering the likelihood of key personnel of the project moving on with time.

In addition to being a great source of documentation, the tests also act as a great design reviewing tool. If you ever struggle to write a test for a particular component, it’s a very reliable source of code smell. Give a moment to rethink the structure of your code and components to see whether it can be tested easily. This is especially true for Unit tests.


Continuous Integration / Build System

A major benefit of a continuous integration system is quick feedback on the state of your application so that developers can quickly fix whatever is broken. However with the increasing number of tests, the build will start taking a longer and longer time to complete. This is increasingly true for integration tests as in general they took more time. So although having more and more tests is generally accepted as good practice, there may be a point of time when the time taken to run all of these tests could be having a negative impact on the project.

Also the dependency of integration tests on external systems mean that your build will break for issues that is outside the scope of your project.

Considering all of these, after a certain point, it no longer becomes feasible to run the full suite of integration builds for each check-in.

Instead, a more pragmatic approach is to have 2 or multiple level of Builds. A low cost build comprising of all Unit Tests and possibly a selected subset of integration tests can be run for each check-in. The full test suite can be run either in the middle of the day or overnight just to give the team some sort of an indication on the overall state of the application. This strategy will still encourage the team to write more tests while keeping the productivity in check.

Also it would be useful to differentiate integration tests that depend on external systems vs integration tests that just talk to internal systems or databases. This could be useful when troubleshooting test failures. You don’t want the team to be consuming time to figure out what’s wrong with an integration test to later realise that it’s due to an external system not playing nicely. It should be plain obvious by just looking at the test name or the failure whether the issue is due to an external system or not.

Seeding Data

Another contentious topic regarding integration tests is how the whole setup of data works. In the real application any operation will require the application to be in a certain state. The seeding of data is intended to bring the application to this state by artificially pushing data into the database or in memory data structures.

One important aspect of seeding is that after every test method the application should be back to its original state. In the context of database integration tests this would mean seeding data at the top of the test and then deleting it at the end. This act could become quite costly quite soon with deep object hierarchies and complex relationships among entities. Also it could be quite a lot of work just to set up the seeds compared to the actual code that does the testing. In fact if you are spending a lot of time in your tests writing code that seeds data it could be an indication that you need to rethink the tests or your application code.


After all these challenges posed by integration tests, one can not argue the importance of an integration test suite in an enterprise application. However with careful strategies we can overcome these challenges.

  1. Write integration tests to check end-to-end traversal of your business process. Not the business logic itself
  2. Make integration tests faster by using in memory databases like Sql Lite
  3. After a certain point, stop running the full suite of tests for each check-in. Opt for a multi level build system where integration tests are run periodically independent of check ins.
  4. Categorize the integration tests so that it gets easier to identify root cause when they fail

Post a Comment