The converging future of Operating Systems

Recent News:

• Opera starts using WebKit as their rendering engine - Feb 2013
• Google folks WebKit, welcome Blink - March 2013
• Opera follows Google and adapts Blink - March 2013

During a short period of 1 month web has experienced a significant turn of events. A useful coalition between Apple and Google (to use WebKit as the rendering engine for their browsers) that accelerated the ever increasing growth of the web has been parted. What would this mean to the web and to main companies (Apple, Google, Mozilla and Microsoft) involved in this? Is this separation a signal of bigger things to come?

For Google this is all about gearing towards the next big computing revolution, i.e Operating System on the Web, namely Chrome OS. If you want to build an OS on the web, you have to have total control over your browser. Initially google tried to achieve this by evolving the web towards its goals by complementing existing technologies (e.g Developing V8 - Javascript Engine to support javascript). However it slowly realised that it has to take drastic measures and started creating the web tool stack from ground up. Ditching javascript in place of Dart (The all new client side language of the web developed by Chrome) was a significant step towards this direction. Now with forking WebKit it has taken another step towards the same direction. I think Google will come up trumps here. I mean who thought Chrome would make such an impact on the browser market, but they did it. Both politically and technically Google is well positioned to deliver. I’m pretty confident that people will be talking about ‘Chrome Extensions’ in the same way they talk about ‘Apps’ in the mobile world presently.

For Apple this not such a good thing. It’s true that Apple (Safari) will gain the same independance in WebKit development by not having to consider about Google (Chrome) anymore. But Apple is not invested in the web to the same degree that Google is, which means that they might not cash-in on that advantage. In fact Apple wants its iOS to trump the web as the ubiquitous platform across web, desktop and mobile. With the loss of the current contribution to WebKit from Google (it’s close to 50%), we’ll probably see rate of innovations in WebKit going down.

Opera counting on Blink is further proof of this assumption. A reasonable question is whether Blink would end up as another WebKit with Opera and Google locking their horns. I think this will not be the case. My expectation is justified considering the relationship dynamics between the two, where Opera clearly has to follow Google shadows.

Fragmentation is a necessary evil

Another advantage of this move is the necessary fragmentation of a given technology (i.e browsers / rendering engines in this case) to make it more conducive for innovation and progression. Although it’s accepted norm that convergence of standards is a good thing, in the case of a fairly fluid and fastly progressing field like the Web there needs to be a certain amount of duplication of tools and technologies to make sure that innovation is not suppressed by the need for compliance. To put it bluntly, making sure that politics doesn’t trump technology. Mainly compliance driven by politics rather than technology. In that light Google and Apple having their own way with the browsers is a good thing.

What’s with Mobile

With Android been the most common mobile platform, I’m hopeful that we’ll see some good innovation in the mobile browser market as well. With opera also on the same platform this can work well.

And now to OS

Another speculation is the convergence of Android and Chrome OS towards a single OS. Although Google has been criticised in the past for seemly supporting 2 OSs for different platforms, their intention is quite clear that they want these to converge into ‘The OS’ over time.. The move of appointing the Sundar Pichai as the head of both Android and Chrome OS has clearly shown Googles’ intention if it wasn’t evident already.

Somewhat similar to Google, Microsoft is hedging on a converging OS but on Desktop and Mobile (Instead of Web and Mobile). Windows8 already shares a common kernel between the 2. It’s true that Microsoft is already finding it difficult to penetrate the mobile market - specially the consumer market. With Google potentially strengthening its position in the Mobile with these moves Microsoft will find it even harder. However they will still consider themselves to have a good chance in the Enterprise Mobile Market, provided they can get the deployment model fixed.

So as it stands here how the big 3 has put their bets;

• Google - Web and Mobile (Google will claim that with desktop will transition into web with ‘Chrome OS’)
• Microsoft - Desktop and Mobile (Counting on Windows 8 to deliver)
• Apple - Mobile and Desktop (Counting on the consumer market penetration of the i devices)

This convergence is real good news for developers. We may not be too far away from developing a single app and running it in the web, mobile and hopefully in the desktop

Microsoft is quite close on getting a Windows 8 app to work in both mobile and desktop. I’d image Google could get a ‘Chrome OS’ app to work in both Web and Mobile soon. Aren’t we getting closer and closer to the single platform dream world. Remember ‘Write Once, run anywhere’ promised from the past?

Who will take us there this time?

Simplifying design with Domain Events

Following post notes down how I used the concept of Domain Events to refactor the message processing component of an Enterprise Web Application. The outcome was great where the main achievements was a code base which was easier to maintain. The 'All hell break loose' kind of classes were gone, and instead a compact, self-explanatory, rich set of new classes were born.
The inspiration for the refactoring is this MSDN article from Udi Dahan.

Problem : Managing a business process where multiple domain object updates are taken place. Depending on the new state of the domain, there are additional tasks to be performed.
E.g: Notifying stake holders about a milestone being reached.

Goals :
Encapsulate business logic to the Domain Model as much as possible while;

• Not increasing the complexity of the code
• Keeping the code testable
• Keeping the code robust for future changes

Existing Solution:

• All of the domain object updates (domain logic) sits safely within domain objects.
• How ever there were lots of other business logic leaking out of domain objects. These were mainly reactions to the new state of the domain. Most of them were cross cutting operation that can't be attributed to a domain object thus ended up in the Service layer. To be honest for simple cases this might not be a problem, however as your code base gets more and more complex this becomes the class everyone wants to hate. Have a look;

Although the domain logic is nicely encapsulated in the domain object (Order) there are few problems with this code.

• Complexity of the Service Layer will increase with time making it harder and harder to introduce changes
• Same business logic (i.e What to do when an order is confirmed) may be required in another part of the application paving way for redundancy

Solution with Domain Events:
Rather than orchestrating the event dispatch logic in the Service layer,  the logic is modeled with events inside the domain it self. Each event and it’s handlers are separated to its own class which minimizes and transfers the complexity from 1 single class/method to a collection. (Note: 1 event could have more than 1 handler). All of the handlers extend a marker interface which is used by the dependency injection framework (In my case, Unity) to resolve the classes that handle a particular event.

Here’s the base class for all handlers
The following set of classes show how we can decompose different events and their respective handlers into their own classes.
The domain entities (Or any other class for that matter) should be able to raise these events easily, responding to different business conditions. This is achieved by having a generic static class exposing a Raise method as follows.

Here's how the domain entity has regained control of the situation by raising events as responses to business conditions internal to it. At the same time have a look at how bare bone the service has become.

NOTE:Here’s how you configure multiple implementations for the same interface using Unity.

QUESTION: How can we extend this design to handle events that respond to business states that depend on multiple domain objects? (E.g. Order is cancelled and User is a premium customer) 

An alternative take on current software practices - Udi Dahan

(Note: Some points are more relevant in the context of Enterprise Application Development)

The famous software personality ‘Udi Dahan’ gave a talk in Perth. I was lucky enough to be one of the attendees. After working in an enterprise environment for a while, I must say I agree with most of his ideas.

I’d like to note down my key learning from this session.

Key takeaways

1. Software is hard to build. First developers need to accept that fact and then they need to work on convincing the other stake holders of the same.
2. The common frailties of the human brain will work against you when building software. Being aware of these irrational behaviours of the brain can help you get better
3. Do not accept or adapt technologies or methodologies because it’s popular (in-time) or proven. Being popular or proven is heavily contextual. You should choose whatever that works for your team at the given time and situation.
4. Understand and ride the organizations decision making wave. Being more aware of organizational politics/workings will enable you to successfully deliver key changes to the products/services that you developer/support.

Brain Tricks and Software Development

There are some inbuilt and predictable irrationality in our thinking process. By being aware of these will help us to identify them when they actually occur and then to over come them.

• Recency Bias

- This is a basic human condition where we are good at relating our current (recent) experiences with our recent (immediate past) decisions or incidents. But fail to identify relations between recent observations and past incidents / decisions. A software project can take years to finish and even more years to ‘fail’ or deemed as ‘successful’. By that time the decisions taken at the start of the project which has direct causality with the success of failure of the project is not identifiable. Even worse is when we attribute decisions taken late at the project as reasons for our success or failure.

- This hinders us from learning what worked and what didn’t in a software project which pushes us in the direction of committing the same mistakes over and over again. A strong ‘retrospective’ activity in your process can alleviate this bias to a certain degree.

• In Group Bias

- This is a situation where we take a decision because our ‘group’ (team, organization, competitor, industry) has taken the same decision. In a way this is a way of distancing yourself from  the decision. In an Enterprise environment this is a classic ball-passing mechanism.

- Group bias clouds the critical thinking process required at a decision making time because you start with a preconceived idea. Classic examples are the almost instinctive decision to go with N-Tier-Architecture (UI-Service-BL-DAL-Database) for many software system that we design or a present day Microsoft house going for MVC - WebApi - Entitiy Framework - SqlServer technology selection.

Technology or Methodology Selection

• Agile

- Agile is probably the most abused term in Software industry today. It’s projected as the single ‘pill’ that will save any given software project. However companies who are successful in delivering projects in Agile are the same companies who were successful in waterfall (RUP) days. On the other hand it’s amazing how much of similarities are there between RUP and Agile in the first place. So what does Agile bring to the table?

- The ceremonies and rituals surrounding ‘Agile’ (Most of them introduced and glorified by consultancy firms) are a distraction to the core values of ‘Agile’. Most organizations tend to get lost in the ceremony and ‘misses the train’ when it comes to delivering quality software. Instead they should focus on basics like improving team communication.

• Unit Testing and TDD

- Unit Testing is useful only if you have ‘Units’ of code that aligns to the ‘Single Responsibility Pattern’. If your code is a ‘bowl of spaghetti’ your tests will be the same...and it will be coupled with the code as well. So 2 bowls of spaghetti instead of one and one mixed with the other...just perfect :).

- Unit test can test what you already know. Most of the time bugs are caused by things that you don’t know. So in principal unit tests are not good at finding bugs. In fact research has shown that code reviews are much better at finding defects as oppose to testing (automated or manual). If you are required to take a decision on where to put your team effort in to, always consider code reviews before unit tests.

• Modelling the ‘Real’ world

- Software that is demanded by todays business is far more complex than 10-20 years ago. Software needs to handle various different scenarios that is not distinguishable in the real world. If your design is an attempt replicate the ‘real’ world you may be constraining your system.

- One example is the notion of a customer in the context of an online store. Who would you consider your customer out of the four;

1. A user who browses your store
2. A user who adds item to the cart but does not proceed
3. A user who fills in all his billing and shipping info but quits before confirming
4. A user who purchases items

A classic ‘Customer’ entity modeled after the real world ‘Customer’ might not be able to handle all these scenarios. As you can see the actual business process(es) are much more complex than the obvious ‘Customer buys a product’ scenario and it can be extremely useful to enrich the model beyond the real world entities.

• There’s no silver bullet

- This has been said over and over again in the industry but we seem to be making the same mistakes still. No new technology, pattern, architecture or methodology will solve all or even most of your problems building software. It’s a matter of you using the correct ‘ingredient’ in appropriate ‘quantities’ at the required ‘time’. Udi gave a nice metaphor for this where it is similar to cooking a dish. You need to choose correct ingredients in proper quantities and add them at the correct time in order to have good quality or at least edible meal.

Riding the ‘Organizational’ wave

• This has to be taken in a positive tone; As technical people who lack ‘Soft Skills’ to match business people, we at least need to be capable ‘sniffing’ the decision making process of the organization. (Note: This is inherently hard to access and understand for technical people)
• This knowledge can enable us to time our technological decisions so that we have a higher chance of getting the buy-in from the business. Udi mentioned a story of the 3-Envelopes. If you can time the opening of the second envelop correctly, that will be the time you pitch the ‘Big Re-write’. (Or ‘Resume driven development’ either way you like to put it)
• Also as ‘self aware smart people’, programmers (or technologists) tend to damage the ego of the people around them more often. You have to be patient and polite in an enterprise environment when you encounter some of the most ‘un-educated’ ideas. The basic rule is ‘Not to be a dick’ to others regardless of how smart or knowledgeable you are on a given problem or a solution.

Improving your skills

• Given the fact a given developer has sufficient technical skills to do the job, it’s much more productive to work on improving your ‘Social Engineering or Soft’ skills. This will enable you

- To convince people to listen to you more often

- To be more aware of people around you and what they think / feel

- To work as a team

• Start somewhere! Don’t wait or search for the perfect book to read or best training to attend to. Start somewhere...anywhere. There are enough quality material to keep you busy.

Improving usability with Telerik MVC Grid

The 'Grid' is a presentation element that you will come across quite often in the traditional LOB application development. Whether it's web or windows one can never avoid meeting the 'Grid'. In ASP.Net MVC development platform Telerik MVC grid provides a rich, consistent and powerful abstraction to handle grids.

What I'd like to discuss in this post is the 'Hierarchical' capabilities of Telerik MVC grid. Specifically how you could achieve different levels of usability and developer productivity by using the server and client data binding techniques that Telerik MVC grid provides.

NOTE: Prior experience of Telerik MVC grid usage is assumed.
Let’s take a 2 level grid and identify different kinds of data binding techniques we have.

Master Data binding	Detail data binding	Details
Server	Server	Suits small scale grids with low. Quick to implement
Client	Client	Fairly easy to implement and can be used for a smoother user experience when larger loads are involved.
Server	Client - Automatic	Detail level is bound as and when the master is rendered on the browser. All the detail levels are bound regardless of whether they are expanded or not.
Server	Client - On Demand	Detail levels are bound only when user expands a level. This scenario scales the best.

The first 2 scenarios are the easiest to implement and you will find several online references. But for the rest of the two, where you got mixed data binding techniques going on between master and detail levels, the examples are not so common.

Server-Client (Automatic)

MasterGrid

DetailGrid

The master grid is pretty straight forward. Only thing to note is that we use ‘Server Templates’ to render the detail view.

In the detail grid a client data binding mechanism is used. This will force the detail grid to perform deferred data binding after the master grid is rendered on the client. However all the detail levels will be data bound regardless of whether they are expanded. This works OK in ‘Chrome’ where the detail level binding happens behind the scene after the master level is rendered. The page will silently bind and render detail levels without page flickering. But in ‘IE’ (7.0) the page gets hold up until all the detail levels are rendered. 

This technique could be preferred if the usage of the grid demands high responsiveness at detail levels but does not care so much about the initial page load time. Since all the detail levels are bound at page load, they become highly responsive afterwards. Good choice for a business case where the user will spend a considerable amount of time in the same page/grid.

Server - Client (On Demand)
In this scenario we attempt to bind the detail level only if the user decides to expand a particular level. 

MasterGrid

DetailGrid

The interesting thing to note here is that although the Detail grid is marked for client data binding it does not have a data binding client event handler like in the previous instance. This will render an empty detail grid to the browser when the master grid is rendered. To trigger the data bind of the detail grid when a particular level is expanded in the master, we use the ClientEvent handler.
.ClientEvents(e=>  e.OnDetailViewExpand("showDetail"))

This will trigger a javascript function ‘showDetail’ which will do the necessary data binding.

This is an excellent choice if you value the initial response time of the page. The detail levels are shown only when required thus this does not put unnecessary load on the network and server thus scales the best.

Few tips on date handling

Handling of Dates have always been an interesting aspect in any programming task irrespective of the environment or language used. When you are developing software in an enterprise environment where multiple disparate systems try to communicate with each other, it becomes even more interesting.

The root cause is generally the lack of standard among the parties. However in an enterprise environment you don't get to establish standards across the board as most of the time you are integrating in to existing systems or standardization simple does not exist. The problems that may occur due to this may seem trivial once you find them, but could be hard to pinpoint when you are dealing with complex integrations.

Some common pitfalls I’ve come across are;

1. Datetimes are either stored or sent without timezone information - Different systems interpret them differently.
E.g : The date 01/01/2012 could be interpreted by 1 system as 01/01/2012 - UTC or another as 01/01/2012 GMT+8 leading to disaster.

2. Sensitivity of the date times. Different systems could interpret a date differently based on how they react to the accuracy level of a date object.
E.g Some could be accurate to the second while another system expects dates to be accurate to the millisecond.

3. Ambiguous display of dates in the user interface - I always prefer to use the format dd-MMM-yyyy (12-Sep-2012) when displaying dates in the UI. This is a far better format than dd-mm-yyyy (12-10-2012) which can easily be misinterpreted to mm/dd/yyyy (10-12-2012).

2. Maintenance of servers to make sure that they are upto date with latest daylight saving patches.
E.g : I’ve seen programs suddenly misbehaving because a certain DST patch is not properly installed in the server they are running. This is especially true in dev and test environments as the SLAs for maintaining them are usually more lenient.

3. Making sure that all integrated servers are synched by the same clock.
Generally this just works, but in an environment where you have different platforms (say unix servers in a primarily windows environment) this can happen.

Best method to solve these issues (at least the ones directly related to software) is not to rely on developers to solve them on a day to day basis. The design of the system should force developers to always do the right thing.

Some of the guidelines that I follow are;
1. Make sure that the data store always have date fields with timezone - Choose the correct data type
2. Have extension methods to specify timezone (I always prefer to use UTC) by default and use them in your outward communicating facades like the DAL layer or web services layer. e.g (.Net)

public static DateTime UtcDate(this DateTime value)
       {
           return DateTime.SpecifyKind(value, DateTimeKind.Utc);
       }

Another extension to this is to establish standards both within and across layers as to the type of dates that the layer supports. As an example;

- Presentation layer is Local Time always

- Business logic layer works in UTC

- Persistence and integrations works in UTC

3. Use a custom user control with dd-MMM-yyyy format to always render dates to the UI

Massaging the Messages - The Right Way

Message processing is an unavoidable piece of work in any integration project. When software systems talk to each other they use messages and generally the messaging interface is tightly managed. Message processing can have different priorities depending on the what the messages are trying to achieve and under what circumstances. In this instance I’m interested on event messaging in the context of system more concerned with reliability and less so with efficiency. Let’s just say it’s a non real time, critical system.  

In general, integration projects take time. They also have lot of complexity and interdependencies. Generally the natural way to build them is phase by phase. In each phase the messaging interface will grow in complexity and depending on the learnings from the previous phase it will change a fair bit as well.

That’s why a well designed message processing component should be robust - i.e respond to changes in the message interface (both semantic and syntactic) without heavy costs.

In addition a critical feature of these kind of message processing component is to be fail proof - i.e respond gracefully in case of failure and carry on the work. Sometimes it’s also required that the failures are notified and corrective procedures are facilitated.

In this post I’d like to share some simple patterns that can be followed to achieve above.

Syntactic Robustness

What I mean here is that the core of your message processing engine should not be worried about the actual format of the message being changed over time. You need to separate out the message parsing (deserializing) part from the message processing part.

In an Enterprise environment this could be achieved by delegating these aspects to a middleware. Middleware will manage the transformation of messages from one system to another. In the absence of such, you need to make sure that you have a separate mapping component which maps messages to your internal entities. Messages could be in the form of XML, DB records or SOAP objects or anything.  

Semantic Robustness with message processing

Here we are more concerned about the actual meaning of the message content.

Most of the time when integration projects emerge, it’s quite simple and straigtforward integration.  But with time they become quite complex and if the initial design is not resilient enough, the code base could end up a mess quite quickly.  

An example is a distributed workflow implementation system. The actual workflow might be happening in a different system (say a control system) and the downstream systems are required to perform different actions based on different workflow events. The first phase of the integration would be to support couple of critical workflow events but with time there will be more and more integration between the two.

That’s why I feel it’s a good idea to start with some form of software pattern which puts you in a good place once the changes start rolling in. A good candidate would be ‘Strategy Pattern’ or some derivative of it.

You can start with defining an interface which will capture the message processor responsibility and a data structure to store instances of these interfaces.


interface IMessageProcessor
{
void Process(Message message);
}

Hashtable<IMessageProcessor> Processors

In phase 1 you might have only 2 implementations like

class ProcessorA : IMessageProcessor
class ProcessorB : IMessageProcessor

Processors.Add(new ProcessorA())
Processors.Add(new ProcessorB())


In addition you need to have a manager class which diverts each message to its proper message processor. In order to do this it will have to do some preprocessing on the message.
It would be great if we could delegate the pre-processing to the interface itself. i.e Each processor is responsible of figuring out whether a message should be processed by it. We will need to enhance the interface as follows;


interface IMessageProcessor
{
void Process(Message message);
bool IsValid(Message message);
}


And the manager class might have a routine like;


foreach (IMessageProcessor p in Processors)
{
if (p.IsValid(message)) {
p.Process(message);

}

}

However in most instances the preprocessing has to happen within a context.
E.g A document edit message/event need to be processed in the context of the current state of the document

So we can enhance our interface like;

interface IMessageProcessor
{
void Process(Message message, Context context);
bool IsValid(Message message, Context context);
}

What we have here is a very simple, yet robust message processing engine with
1. Extensibility (Open-Close Principle)

- Extending support for new scenarios (class ProcessorC : IMessageProcessor)

2. Modifiability

- Localizing the changes to particular scenarios (If you want to send an SMS to the document owner as part of ProcessorB, you will only touch ProcessorB class)

3. Testability
- It’s fairly easy to write a test suite against the different Processors and maintain it.

Fail Proof

Here what I’m concerned about is what happens when you can’t process a message. There are endless reasons why your well intentioned message processing engine fails to process a message. Rather than dwell on why, let’s see how we should design it, so that life goes on even if a message (or two) fails.

One decision to be taken here is whether to hand over the responsibility of failure handling to the manager class or to each processor. If your failure handling mechanism end up being more complex and different (even in a few cases),  you will end up enhancing the interface as follows.


interface IMessageProcessor
{
bool Process(Message message, Context context);
bool IsValid(Message message, Context context);
void HandleFailure(Message message);
}

foreach (IMessageProcessor p in Processors)
{
var processed = false;

if (p.IsValid(message)) {

processed = p.Process(message);

}

if (!processed) {

p.HandleFailure(message);

}

}


If the failure handling is generic, there’s no need to change the interface and the manager class will look like; The requirement here is that every processor class won’t swallow errors internally.


foreach (IMessageProcessor p in Processors)
{
try {

if (p.IsValid(message)) {

p.Process(message);

}

}catch (Exception ex) {

//Do whatever

}

}


In either case, one specific requirement of failure handling is preventing subsequent messages from failing.

Suppose you fetch messages for processing as follows;

locator.FetchAll<Message>.Where(m => !m.Processed).OrderBy(m=> m.CreatedDate).First();


In a failure situation, if you don’t mark a message as processed, the problematic message could hi-jack your message processor. i.e Message processor will always try to process this message in each cycle. Thus we should have something like;


foreach (IMessageProcessor p in Processors)
{
try {

if (p.IsValid(message)) {

p.Process(message);

}

}catch (Exception ex) {

//Do whatever

}finally {

m.MarkAsProcessed();

}

}