Sunday, August 20, 2017

Tales from the Enterprise - Backward Compatibility

Backward compatibility - this is something you will hear over and over in various forms during your life as an enterprise developer. The arguments become even more strong when you consider someone who’s developing framework libraries for the organization. These are libraries that gets shared across the organization providing a standardised set of services - both business specific common operations and business agnostic operations like data access, logging, security etc...I can certainly see why these libraries need to sit at the organization level providing common functionality. However after few years down the track one realizes that there are lot of business critical application that have taken dependencies on these libraries. Now you realise that a change to a framework library that should not take more than couple of days, suddenly becomes a saga and involves dances between multiple teams and can get quite nasty. The underlying reason for this is that ‘Backward compatibility’.  

Even if the changes to the libraries themselves are tiny, the complexities around how to manage that change in the context of many applications that depended on these libraries can become quite complex.

So how does one keep upgrading these framework libraries all the while being backward compatible? Most of the business systems still want to get the benefit of these upgrades, but they don’t want to deal with the risks associated with the changes. What could be some of the tips dealing with cases like this.

  1. Version control is your friend. With a modern version control like git, concepts like branches are no longer difficult - it’s second nature. Use them to your advantage when you want to test out upgrades in isolation to a product that’s live and is potentially going through it’s own business critical changes.
    1. Create an ‘Upgrade’ branch to isolate framework library changes (Also absorb changes to the main branches in a controlled manner)
  2. Automated builds - Assuming your business critical application already has a CI build, (Don’t be surprise if there aren’t - then you gotto make one immediately) make another CI build so that the upgrade branch is continuously tested automatically.
    1. Make a copy of the existing CI build (TeamCity is great at this) and point to the upgrade branch created earlier
  3. Pick a few tests as critical - Your business project may have thousands of tests, but it might not be 100% reliable. You have to pick a subset of tests (you may have to write a few) that directly relates to the changes you are making. This is your test suite - you must make sure that they remain green during your upgrade project.
  4. Dependency Management - If your legacy projects are not using a proper dependency management framework like Nuget, you are in for a mess. If it’s not, it’s worthwhile investing your time to do so. In my case most of the shared libraries did actually had nugets published, however some of  the legacy apps were happily doing direct lib references like they did 5-8 years ago. It was well worth the time and effort to make them use nugets instead.
    1. Make the project use nuget (or paket or maven) instead of lib dependencies
    2. Make use of nuget versions to isolate from potential breaking changes
  5. Adapt semantic versioning - What goes hand in hand with dependency management is semantic versioning. With semantic versioning you can take a more or less confident approach towards adapting newer versions of a given internal shared library.
  6. Break it if you really want to - There may be times, when the friction from the past is too much, that you can’t really progress without making breaking changes. In these instances be brave and make it. If you don’t do it, it won’t be tried for a few more years and by that time the sheer weight of technical debt will break the systems down anyway/ But the caveat is with techniques like semantic versioning in place, you can do these in a safer manner.
  7. Pick a good time window for testing - Testing by the framework library developers themselves might not be sufficient. May be the merge back to main branch can wait till the project has a meaty enough change planned, so that you could leverage from a full regression test cycle. Being a bit business savvy in these decisions could be critical, since the project it self might not want to spend money in the framework library upgrade.

Monday, November 21, 2016

Is innovation overrated?

Innovation is a popular theme going around in many circles. Innovation gets people elected to government or it can make people loose office. But is the future all about innovation and creativity?

For years Engineers and others have been spending majority of their time on maintaining the world order - making sure things continue to work amidst the chaos of the real world. However these days young engineers coming out of universities are always given the message of innovation and entrepreneurship. But the reality is that roughly 80% of them will end up working as maintainer in one shape or another. In that case aren’t we intentionally distorting the reality and making them feel bad about their career choices?  Aren’t we shining the light out of the a very important and a significant chunk of workers who are maintainers and putting it towards a small number of innovators or creators?

It’s true that the fetish of innovation is a lot more sexier than the glamour less tasks of maintenance. But do we want to establish our government policy and organisational agenda purely around glamour?

Main reason for this is that capitalism in its nature is fine tuned at encouraging innovation or growth while not putting the same emphasis on maintenance. Maintaining the status quo is deemed as a failure.

Under Investment

Another real danger of over glorification of innovation is the under investment in existing system. Under investment in maintenance will create a ever growing list of technical debt and risk that is passed onto future generations. A good example is the hype around building new infrastructure. Politically, building new infrastructure is a winner. No one can argue the benefits of it. However historically big cities have suffered from putting extra attention to big new infrastructure projects in the expense of putting funds into maintenance. Not to mention the countries and cities piling up debt that will be passed on from generation to generation.

Considerations around maintenance should start even before you build your systems. One thing that drives people out of maintenance is how difficult and costly it gets to maintain systems with time. This directly tie into the fact that how much thought went into maintenance considerations when the system was built in the first place.

Maintenance in Software

The software industry has been aware of this for years. That’s why we as software designers and architects put writing maintainable systems at a high pedestal.  My personal opinion is that  complex systems inherently lean towards putting emphasis on maintainability. If you have to tear up a system every time you want to fix something, automatically maintenance become far more expensive than it needs to be and eventually will be replaced by the new system that is the child of sexy innovation.

That’s why software engineering education should specifically dedicate some of its curricular towards teaching students the value of maintaining systems and building systems that are maintainable. Essentially how to keep improving the engine of a car while it’s been driven. Concepts like SOLID principles, live documentation, version control, unit testing may have a bigger role to play in the tool set of a maintainer. The biggest job is driving a positive message around the role of maintainers.

Final Thoughts

I’m in no way against innovation or de-valuing the important part it will play in our future. I just feel that lately we have been over glorifying innovation and creativity un intentionally forgetting the historically critical role of maintenance. Adequate maintenance together with relevant innovation should be the goal of a growing society.

Thursday, July 14, 2016

Chrome Dev Tools - Console Tricks

I’m old enough to remember debugging javascripts using alert statements littered all over the code base. However Firebug from Firefox changed the game. Nowadays I use Chrome developer tools.

Let me quickly share some less known tricks around the Chrome developer tools’ console.

1. console.log(‘message’) - This is straightforward log of a message string, nothing special.

2. How about c like string formatting of the log statement?
console.log(‘Number %d, a string %s an object %O went to a bar”, 123, “message”, { country: “Sri Lanka” })

3. Another useful way to log objects specially if it's a collection (array) is to use console.table(). Instead of having to expand and collapse individual objects to check them, you can see all of them directly in tabular form.

4. Different log levels. Just like any logging framework console has different levels of logging in addition to the console.log called info, warn and error
console.error includes a stack trace as well

5. How about printing the stack strace? This is specially useful when you want to track where an error/special condition originates from.

console.trace(“message”) will print a whole stack trace to the point at which console.trace is called.

6. Grouping logs together
It’s also possible to group a set of log statements together so you can collapse expand them and also distinguish this group from other statements.
Here’s how above works"Group starts");
console.log("Testing 1");
console.log("Testing 2");
console.log("Testing 3");

I found this to be really useful when logging inside iterative blocks.

7. Similar to group we can use console.time() and console.timeEnd() to log the time taken to execute a set of js statements

8. Again similar to above we can use console.profile() and console.profileEnd() to add a CPU profile for further investigation. It can be seen under the ‘Profiles’ tab in dev tools

Sunday, May 29, 2016

First Impressions at Work - Employers Point of View


As a consultant, I have been moving between jobs more regularly than a normal employee would. In the past I have moved between couple of jobs as an employee too. Along the way I have picked up a few tips regarding how to make the life of a new recruit better. Sadly, as excited you are on taking a new job, there’s always the inevitable disappointments associated during the first few days at the job. What I want to discuss is, as an employee how you may improve the situation. This is quite important, as our human nature is to put more weight towards how you feel at the first few days at the job. Unfortunately these first impressions might go a long way towards how you feel about the job overall.

So it makes sense from the employer's’ point of view to at least get the first few days correct. Try to welcome the new team member with enough professionalism and friendliness which will re-enforces that he has made the right decision.

I will start with the real fundamentals and list down some of the more exciting ones towards the end. (Some of the items may obviously biased towards software engineering)


  1. Please make sure they actually have valid access to the building. I have worked in places where for few weeks my building access was not sorted. You basically get a visitor pass and it’s not fun to ask for a colleague to escort you to the toilet every time because you may be locked out of the working floors if you walk out.
  2. Have their desk ready. No - you don’t need flowers and balloons, just a clean desk with a computer is fine. Yes, that includes a chair too.
  3. Having a desk and computer is a good start. But also make sure they can actually login to the network and have proper access rights. I have worked in big corporations and government organizations where it takes them few weeks to sort out Active Directory access.
  4. Please take the time to show the new guy the office facilities and how to use them. Toilets, printers, kitchen, vending machines...anything. Teams may overlook these simple things specially if the team is quite busy, but a few minutes of effort may go a long way in communicating that the team actually cares about the new employee.

  1. The computer should have all the necessary software installed. If not installed, the installation process should at least be properly documented (in a wiki page with links to each software package). This can be the difference between someone either spending days to set their machine up or realising after few weeks that they’ve set it up in a slightly different way so they have to start it over again.
  2. Should be given access to credentials required to login to different systems. If installation of security certificates or keys are required, this is the time to do so. It will certainly help to have a password/key management software like keepass or lastpass.
  3. Admin rights to the computer. I can’t stress this enough - especially if you are a software engineer. Not having admin rights to the box could easily add up to days and days of support calls and productivity issues down the line. Not to mention the sheer frustration of the user.
  4. May be have a starter pack documentation in wiki.
    1. Page with the team with photos and details
    2. Organization/Project terminology
    3. FAQ
  5. Organize a meeting with few team members they would work with and clearly provide details around what they will be working on and the expectations. This is not to scare the poor fellow, but to show that you have planned his work and is serious about the responsibility of each team member.
  6. Get an experienced team member to briefly demo some of the common workflows and tools used. It could be JIRA, IDEs, Version Control, Build Systems, Messaging etc…


Even if you get most items from the above two lists,  it’s imperative that the new recruit feels part of the team.
  1. Take them around the office floor and introduce to each team member - one by one. Doesn’t matter if he doesn’t remember any single one of them - it’s the act that matters. This should also cover off-shore team members (or clients) if that makes sense.
  2. Invite them for lunch. A new recruit is reluctant to suddenly ask strangers to join in for lunch. It’s not a great feeling to have lunch on your own at your first day.
  3. Invite them for coffee - specially if lunch doesn’t work out.
  4. Get a friendly, outgoing (but knowledgeable) team member to act as his buddy for few weeks. This can be set up informally before the arrival so that it doesn’t become too uncomfortable.

There are some additional actions you can take which can really make an impression.
  1. Organize a team lunch on behalf of the new recruit. This could very well be the regular weekly lunch the team goes out to - but it has a theme just for 1 week. To welcome the new recruit. If the team culture is different, you may replace lunch with drinks after work.
  2. The lunch could be sponsored by the company. This gives the existing team members a tangible benefit due to the arrival of the new recruit.
  3. Offer to sponsor a tool or a software that the new recruit would like to use for his own efficiency. This really drives homes the fact that the company cares about the time of the new recruit - not just his output.
  4. Encourage them to improve their skills and offer to sponsor for online courses that they are interested about. Important thing here is not to be restricted to purely work related courses. Could be anything. (Coursera offers good quality courses from reputable universities for affordable pricing)

Monday, November 02, 2015

The illusive mission statement

Can you remember the mission statement of your place of work? If you do, then you truly are a special one. When I started my career, mission statements stood out like this carefully stitched up collection of words that emits its grandeur nature to all around us. It didn’t take too much time for that grandeur mission statements to pale into useless set of empty words that doesn’t convey anything objective to anyone in the organization. Except of course for the for handful of people who actually come up with it, because that’s the sort of things they get paid for.

But how do you measure the value of such inspirational one liners? Can they be useful in lining up a team towards a certain goal? It’s well known fact that when you know what you are working towards, chances of you actually getting there significantly increase. So how do you as a leader of a team (Could be your team at work, your family or your sons’ football team) assist your teammates to collectively work towards the same goal, by getting these one liners into their subconscious?

In his book ‘Essentialism’ Greg Mckeown nicely put these kind of statements into a simple diagram of 4 quarters.

The first quarter is not worth discussing about. 

The mission statement you see in 99% of big enterprise falls into the second quarter of ‘Inspirational’ but way too vague to be of any use to anyone. 

Your day to day goals and KPIs could well fall into the fourth quarter where they are very very clear and concrete, but mundane and doesn’t carry a lot of inspiration. True - they have their place, but it’s not when you want to inspire a bunch of people towards a goal.

The third quarter is where the magic is. It’s where you get inspirational yet concrete enough goals. These inspire people as well as giving them a sense of how their everyday tasks can relate to it. One example Mckeown comes up in his book is a mission statement by a government agency in UK.  In a  project to improve the government IT initiative, after the lead of the project came up with a  mission statement that was both inspirational yet resonated with both the project team as well as the citizens in general. It was  “To Get everyone in UK online by 2017”. Clear, concrete yet still inspirational. These sort of statements can be an essential tool for project people to gauge their everyday tasks against the end goal of the project.

Now extending this idea to the software development world, I think any project substantial enough can actually benefit from having such a clear, concise yet inspirational statement. You don’t have to call it a mission statement. But it should be communicated to team members early and clearly, and may be repeatedly. There’s always the risk of ending up with a watered down statement which ends up in the second quarter - which can actually do more harm than having none at all. But it may well be worth the risk, especially if you know the pitfalls as we do now.

When you are stuck with making tough decision that doesn’t have the necessary data behind it to make a highly objective decision, the statement could be your guiding light. I remember once a customer (CEO of a small business having around 20 staff) telling me that the goal of the project was to get rid of 2-3 staff members each year. Not that I call it a truly inspirational goal (specially if you are one of his staff members), but for a shareholder or the external consultant like I was - it could well be. It gave me clear idea about what this customer is after and the feeling that this project could really make a big difference to this company. Those are the kind of feelings and thoughts that needs to be nurtured so that they fuel your team during the tough times.

Saturday, September 05, 2015

Making the best use of Architects in an Enterprise

I have worked with many types of Architects over the past 10 years. Most of my work experience is in the Enterprise world, which gave me a chance to work with more Architects than a non Enterprise developer would get. They come in different shapes and forms, Software Architects, Product Architects, Technical Architects, Solution Architects and drums roll….Enterprise Architects. Although you may not see this in formal job adverts there’s such a thing as Architect Astronauts as well.


Although we got architects coming in different shapes and sizes, unfortunately I have not seen many adding a lot of value to a project delivery. Yes, some of the above mentioned architects by definition of their role does not have anything to do with a delivery, at least not in a direct way. I won’t be stretching the truth if I say that architects are people who have been in the industry for a considerable amount of time and have done a lot of delivery work themselves - albeit prior to them being architects. Given that, it’s counter intuitive in a way, that Enterprises in general values architect input in more strategic and high level areas and ignore the value they could provide in shipping software.

First, I’d like to discuss whether it’s beneficial for an organization to encourage its architects to play a more involved role with the delivery teams. Then in the cases where it is, I’d like to discuss some techniques one can use to get the best out of the role ‘Architect’. During the process the I’ll also try and see how an architect him/her self could benefit by being more involved in project delivery.

Benefits of Architects involving in Delivery
In my past experience most architects engage with the delivery teams either at the start during elaboration or at the end during roll out (specially when things turn out to be nasty). In my opinion this is not the optimum use of an architect. Software Architecture by definition covers the aspects of Software Development that’s hard and costly to change. By dis-engaging with the delivery team after the initial phases you are losing valuable feedback that you get during the lifetime of project development. Feedback that is essential in identifying the pieces of your software that need to change, adapt to the ongoing changes and new findings. Most of the time architect is not to be found until the project gets close to delivery and at that point most of the decisions that have been made have become too difficult to change.

Another disadvantage of not being in touch with the teams is that the architect losses upto date information re: how the technology landscape is moving. How can a person who is suppose to make organizational level technological decision get himself disconnected with the best possible source of relevant technological information. A dis-engaged architect has no chance of keeping the product architecture respond to changing requirements while making sure it also aligns with the enterprise architecture vision as well.

So what would be the best possible way of getting the best out of the Architects. Assigning him delivery tickets from your sprints might not be the best possible tactic. Yes, there’s nothing like delivering cold hard code that gets you right in to it. But considering the varied aspects of an Architects’ role, burdening him with direct delivery artifacts might not be the most efficient use of his time. It might even be detrimental to the team. The team velocity might see weird up and down spikes based on how busy the Architect is with other commitments.  

Tactics for enhancing Architect Engagement


Daily Stand Ups
Daily stand up is an efficient method of getting to know the pulse of the team. Most importantly it would give the architect an opening to know the challenges the team is facing. These challenges / problems could be the spark for more significant architectural changes. Alternatively architect could intervene early to direct the team in case the problems are due to a mis-understanding of the requirement or technology. By assisting the team through these difficult tasks the Architect could also gain some trust and appreciation from the team members which is vital in the long run.

Code Review (of Architecturally significant pieces)
In general code reviews is a great way to intervene and understand what’s actually delivered by the team. But considering the various commitments an Architect might have, it’s important that only a selected set of code reviews are performed by the Architect. I think the selection should be driven by the Architect, to examine architecturally significant deliveries. An example is any framework code done in the early phases of the project. In-fact I’d prefer the Architect to write or at least heavily get involved in writing most of the framework code, let alone reviewing it. In addition to framework bits, some other significant areas could be integrations, complex business logic, high usage areas etc…It’s also important to let go certain areas of the code both for efficiency reasons and also to enhance the decision making and ownership of the team.

Pair programming
Similar to code review, pair programming is a great way for the architect to keep a close eye into any architectural feedback coming from the team. It also helps the team members to learn from a more experienced engineer. Even little things like development utilities and shortcut keys could assist the development of the junior team member. It can work the other way too, specially for a time poor architect, this is a good way to learn a few tricks of the trade from a developer.

In a large enough project, it’s inevitable that you need to some research and proof of concept work before you commit to a particular technology or method. An example would be to check which logging framework to use for the logging in your project, specially if there are specific requirements around logging in the project. Architects with their wider understanding of the projects’ landscape put together with their experience with different technologies and processes are ideal candidates to carry out these pieces of work commonly known as Spikes.

Some of the key technological and tool selections generally need to be backed up by a solid spike. As with everything else, this benefits both the project and the architect, not to mention the team members.
Tooling Research & Purchasing
I am firm believer of usage of proper tooling in any given software development project. Sometimes its unfortunate that some of our Enterprise clients don’t put enough emphasis on the tools, which can end up costing them thousands of dollars of programmer time and rework. One thing I’ve noticed is that when the source of a tool request is a developer, the chance of approval diminishes. But a knowledgeable architect can do these things and own a tool request or an upgrade. Not only an architect can really justify the availability of a tool among team members, he/she is in a much better position to justify the purchase of a tool to a customer as well.

By being engaged in this kind of productive role towards the team, makes the Architect a popular figure among the team - one of our own -, a key requirement for a healthy working relationship between the development team and higher ranks in the company.

Process Improvements
Since Architects are not fully engaged in the day to day activities of a project, once it goes into full swing, he/she is in a great position to provide valuable feedback re: the software development process. For one this architect should be able to convert daily practices into more measurable heuristics on which decisions can be taken. A good understanding of current trends in software development practices as well as hands on experience on delivering software are key to understanding the (sometimes implicit) process quirks within a team and modelling them to known patterns.

This modelling is quite important, as that disconnects the actions of individuals or team, from the individuals themselves. When providing feedback to somewhat intangible aspects like ‘process’, the experience of architects can be drawn in to model the day to day activities into known software development process models.

In an Enterprise environments, software architects in general play a more strategic and advisory role. While appreciating the value of that, it’s imperative that software development teams also utilize architects to improve their capabilities around delivering working software.

An important shift in thinking about this is to engage the architect in one of many contributory roles. Not only these tasks will help the team to deliver software, it will help the team to understand the overall picture external to their delivery, will also help junior members to look up to senior members of the community and improve their skills. On the other hand it will also assist the architects themselves to get in touch with what’s happening in the industry at ground level and valuable project level feedback that could then be sent up in to the organization hierarchy.

Do your architects play a role in delivering software in your organisation? If so, what techniques do you use?

Wednesday, July 08, 2015

Integration Testing - Real World Challenges

Testing is an integral part of any software development project. While everyone understands and acknowledges the importance of Unit testing, Integration testing is also equally important in an Enterprise software development project. Integration testing is the kind of testing where multiple parts/modules of a software system is tested as a group. The multiple modules could be different layers of the system including the database or it could be the application together with external systems.

Enterprise applications often have lots of integration with external systems, systems that are often out of your control. These external systems may not be consistent when it comes to availability, performance or state, posing challenges to day to day running of an integration test suite, continuous integration/deployment set ups or maintenance of an integration test suite.

Whether it’s integration testing covering different parts of your own application or covering external systems, integration tests possess different challenges that is not seen in Unit testing.

I’d like to share with you some important learnings around testing in a large Enterprise application development project.

It’s not just a ‘test’

Testing is not all about making sure your application is in good order. In a big enterprise applications, tests are a good mechanism of understanding how different parts of your system work. It’s essentially a type of documentation for developers while they are developing it. Further it’s a great tool for developers and support personnel during the maintenance phase. This is especially important considering the likelihood of key personnel of the project moving on with time.

In addition to being a great source of documentation, the tests also act as a great design reviewing tool. If you ever struggle to write a test for a particular component, it’s a very reliable source of code smell. Give a moment to rethink the structure of your code and components to see whether it can be tested easily. This is especially true for Unit tests.

Continuous Integration / Build System

A major benefit of a continuous integration system is quick feedback on the state of your application so that developers can quickly fix whatever is broken. However with the increasing number of tests, the build will start taking a longer and longer time to complete. This is increasingly true for integration tests as in general they took more time. So although having more and more tests is generally accepted as good practice, there may be a point of time when the time taken to run all of these tests could be having a negative impact on the project.

Also the dependency of integration tests on external systems mean that your build will break for issues that is outside the scope of your project.

Considering all of these, after a certain point, it no longer becomes feasible to run the full suite of integration builds for each check-in.

Instead, a more pragmatic approach is to have 2 or multiple level of Builds. A low cost build comprising of all Unit Tests and possibly a selected subset of integration tests can be run for each check-in. The full test suite can be run either in the middle of the day or overnight just to give the team some sort of an indication on the overall state of the application. This strategy will still encourage the team to write more tests while keeping the productivity in check.

Also it would be useful to differentiate integration tests that depend on external systems vs integration tests that just talk to internal systems or databases. This could be useful when troubleshooting test failures. You don’t want the team to be consuming time to figure out what’s wrong with an integration test to later realise that it’s due to an external system not playing nicely. It should be plain obvious by just looking at the test name or the failure whether the issue is due to an external system or not.

Seeding Data

Another contentious topic regarding integration tests is how the whole setup of data works. In the real application any operation will require the application to be in a certain state. The seeding of data is intended to bring the application to this state by artificially pushing data into the database or in memory data structures.

One important aspect of seeding is that after every test method the application should be back to its original state. In the context of database integration tests this would mean seeding data at the top of the test and then deleting it at the end. This act could become quite costly quite soon with deep object hierarchies and complex relationships among entities. Also it could be quite a lot of work just to set up the seeds compared to the actual code that does the testing. In fact if you are spending a lot of time in your tests writing code that seeds data it could be an indication that you need to rethink the tests or your application code.

After all these challenges posed by integration tests, one can not argue the importance of an integration test suite in an enterprise application. However with careful strategies we can overcome these challenges.

  1. Write integration tests to check end-to-end traversal of your business process. Not the business logic itself
  2. Make integration tests faster by using in memory databases like Sql Lite
  3. After a certain point, stop running the full suite of tests for each check-in. Opt for a multi level build system where integration tests are run periodically independent of check ins.
  4. Categorize the integration tests so that it gets easier to identify root cause when they fail