Help Turn the “Jersey Shore” into “Silicone Shore”

SiliconeShoreWant to work with the latest technology on a self-directed team? Think you need to be in Silicone Valley to be part of a company setting the standard in cloud and mobile computing? Read on.

Marathon Data Systems, on the beautiful Jersey Shore, through our various vertical-focused brands, serves the people who provide the services that we all count on: HVAC, plumbing, lawn care, pest control, maid services, carpet cleaning, and more. We are in the process of building a mobility solution that will set the industry standard for field service worker and salesperson productivity.

Cross-Platform Mobile

Our new cross-platform mobile client is HTML5-powered and will initially run on iOS and Android devices. The technician or salesperson is continually operating on the latest information (no more periodic synchronization) and yet can be fully functional while out of network coverage.

SenchaPlusCoffee

The new mobile client is built in CoffeeScript on the Sencha Touch 2 platform. The combination of Sencha Touch and CoffeeScript making building off-line capable, singe-page applications actually pretty fun. Sencha takes care of the tedium of rendering just the right HTML to the screen for each device, while we focus  on building great user experiences.

Since WebSockets are not yet a reality (and pull-based applications are lame), we’re using SignalR to simulate server push from ASP.NET MVC (think Node.js scalability in ASP.NET MVC).

Message-Driven CQRS Back-End

DomainDrivenDesignOn the back-end we are are building out a scalable, event-driven CQRS system built around NServiceBus, following the principles of Domain-Driven Design (DDD).

In Domain-Driven Design, we recognize that the heart of the software is the domain-specific behavior that allows its users to solve their problems. We establish a vocabulary that spans across technical and nontechnical people. We iteratively tweak and adjust our domain model to better map to the problem space. We focus on building an effective model of our domain, while minimizing entanglement with infrastructure concerns.

nServiceBus_LogoNServiceBus (NSB) allows us to focus on the semantic meaning of our commands and events and what the system’s behavior should be. NSB handles interaction with queues, message routing, retries, and other plumbing-level concerns.

ravendb

Data persistence is achieved through the joy that is RavenDB. If you think ORMs like NHibernate are good, just wait until you have built a C# application with RavenDB.

RavenDB is a high-performance, second generation document database, native to C#. Queries are made through LINQ. Joins and transforms are handled through map/reduce functions also written in LINQ. The impedance mismatch between object-oriented code and a relational database is not merely bridged, but is eliminated.

Agile Team

Agile can mean almost anything you want it to mean, but at Marathon, it means…

Agile at Marathon

A belief in the core values of the agile manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The Scrum process, where a self-organizing team works together to plan and deliver iteratively and incrementally.

The Extreme Programming (XP) practices of test-driven development, pair-programming, continuous integration, collective code ownership.

A commitment to software craftsmanship. Being a professional means a commitment to quality work and continual improvement. Great software comes less from particular technologies that come and go, but from fundamentals like the SOLID principles and object-oriented design principles and patterns that stand the test of time.

This job may be for you if…

  • You are good, but you expect to keep getting better.
  • You can do it yourself, but you’d much rather work together with a team of dedicated developers.
  • You love cool technologies, but you care about user loving and using your software more.
  • You love coding in C#.
  • If you are not currently test-driven, you want to learn to be.
  • If you are not currently pair-programming, you are willing to give it a try.

If that sounds like you, please send your resume to jobs@marathondata.com. Come join us as we create great software to serve the people who serve the world!

Review of Advanced Distributed Systems Design using SOA & DDD on Video

As my team was getting ready to embark on a significant new project built around asynchronous messaging and NServiceBus, I would have really liked to send the entire team to Udi Dahan‘s  five-day Advanced Distributed Systems Design using SOA & DDD course. Distributed, service-oriented systems have many advantages over traditional centralized solutions. However, the change in thinking that the development team must go through is daunting. However, sending the entire team offer a week would’ve been a challenge. As an alternative, I purchased the course videos and we work through them together over the course of three weeks. This is my review of that course (on video).

Why should you even care?

Before getting into the review, why would you want to build a distributed, service-oriented system? In a word, scale.

Scaling the software: Asynchronous messaging and eventual consistency allows the system to scale out to redundant, cheap hardware. Brewer’s CAP theorem shows, centralized systems that rely on two-phase commits for all updates sacrifice reliability/availability in favor of an immediate consistency that isn’t always necessary.

Scaling the development team: A well-factored service-oriented system avoids the problem of a single monolithic system that becomes increasingly difficult to enhance and maintain and requires scale up of a single development team and resists scale out to multiple teams.

Review

Advanced_Distributed_Systems_Design_-_Day_1_Session_1The production quality leaves a lot to be desired. There is an annoying amount of background noise through all of the videos. The video is shot with a single camera that pans inconsistently between Udi, the whiteboard, and the projector screen. Honestly, you have to really want the content to make it through all 40 hours. However, you will be well rewarded.

The best way to describe the course is to think of it as a five day walk through all of the factors that led Udi to develop NServiceBus. I’ve followed Udi since way before NServiceBus. I listened to the Ask Udi podcast and attempted to grasp the essence of SOA. No matter how hard I tried, I was never able to implement the tenants of SOA with the traditional Microsoft tools and WCF. It wasn’t until I built a system with NServiceBus that I started to make the mental transition. Every time something I was trying to do was difficult and I was feeling that NServiceBus was overly constraining, I eventually realized it’s not you, it’s me. When you are doing it right, NServiceBus feels natural. When you violate good design for distributed, service-oriented systems, NServiceBus nudges you back by becoming difficult, causing you to go looking for a work-around, and eventually finding a post by Udi that gently explains why what you are trying to do would eventually bite you in the rear end and what you should be doing instead. This course lays the groundwork that will help you avoid the wrong turns and false starts you are bound to make otherwise if you are moving from the familiar world of centralized, monolithic systems.

He spends a significant portion of the course methodically and deliberately dislodging ideas that most of us have held for so long that we no longer question. He also works to temper and ground some of the techniques that have caught on with a small but vocal minority of.net developers, such as CQRS and event sourcing.

I recommend this course if any of these apply to you:

  • I don’t know how to model autonomous services. Everything is related to everything else. My domain is too hard to tease apart into independent components. (Hint: this is hard at first, but Udi will take you through enough examples to get you headed in the right direction. Oh… and your domain is not too hard to do this.)
  • I love Domain-Driven Design, but I don’t know how to model the domain and application layers. I recognize that I have fallen into the anemic domain model and all of my logic is in procedural application layer functions. (Hint: There is a place for simple CRUD and the rest cannot be modeled without sagas.)
  • I like the idea of CQRS, but I don’t know how to approach it, or I tried and it didn’t work out as well as I expected. (Hint: Event sourcing is not the silver bullet of CQRS.)
  • I am using NServiceBus, but I don’t understand sagas; so, I’m not using them. (Hint: you’re doing it wrong.)

All of these were true for me at some level. Getting these issues addressed made the course extremely valuable to me.

Conclusion

The entire team feels the course was a great investment and is entering this new project with anticipation and confidence that we will avoid many mistakes that we would have otherwise made if we had not taken the virtual course. I believe we will save the money we spent on the course and the time away from coding many times over on this project and beyond. If you are building a moderately complex collaborative system, you owe it to yourself to check out this course. If you can attend a live offering, I recommend that. But if you cannot, then the videos are well worth the investment as the next best thing.

DDD Anti-Pattern #1: Not Accounting for Commands and Queries as Separate Concerns

As a long time object-oriented programmer, I was immediately drawn to the DDD concept of Persistence Ignorance. My initial idea of DDD was to simply express the domain concepts in a C# object model, then one one side, map those objects to a persistent store with a technology like NHibernate and on the other side, build a UI without any business logic in it.

Naive DDD Initial Transition

Figure 1: The Naïve Transition to DDD

This was a big improvement over the traditional techniques on the Microsoft stack. Databases are inadequate for expressing domain models and business logic in UIs or databases makes unit testing nearly impossible.

How the Naïve DDD Approach Fails

What I didn’t see was that this naïve solution traded one set of problems for another. The new solution looks like Figure 2 below.

Naive DDD

Figure 2: The Naïve DDD Approach

This looks OK until you try to build anything remotely complex with this model. The first bit of friction we ran into involved populating some of the screens on the UI. Many screens don’t map smoothly to our rich object model. We get by with technologies like LINQ and lazy loading, but it still feels awkward to spin up a rich object model to simply copy properties to a view.

Things really turn south when we start to have multiple interfaces to the application. Imagine we have our UI and add a SOAP-based API to allow other programs to drive the application.

Naive DDD Multiple Interfaces

Figure 3: The Naïve DDD Approach with Multiple Applications

Now the CRUD layer really starts causing trouble. First, notice that because our domain model is defined in an assembly (or package or library), it gets deployed to each separate application that needs it. We have a layered architecture, however, all but the database layer get replicated to multiple applications.

CRUD-Based Models and Meaning

Imagine that a user makes a update to the comments on order 0123 through the UI at the same time that an external program calls GetWorkOrder(“0123”), updates the order description, then passes the updated order into SaveWorkOrder. Now both components are instantiating order 0123, copying the updated property values and saving it back to the database.

We lose the intent of each of those two actors. We are left to sort out any conflicts on simply timing and the intelligence of our ORM tool.

You could avoid the replication in Figure 3 by creating a application server, but you would still have the loss of semantically meaningful information that comes with a CRUD-based object model.

Request/Response: Good for Some Things Not for Others

Our Figure 3 architecture is inherently synchronous request/response. Consider the GetWorkOrder Web service method. Our external system calls that method and gets back the specified work order. This is what we want. The calling code cannot continue until the request comes back.

On the other hand, imagine any method that makes any kind of change to the system (like SaveWorkOrder). We would like to be able to queue these changes and allow the system to process them as compute capacity is available. We also don’t want to have to write code to handle the case where the Web service is temporarily not available. 

The Old Way Had Some Things Right

Notice that the bad old traditional system does not exhibit some of these problems:Smart UI

Figure 4: The Old Smart UI Approach

What the traditional system did right was account for updates being different from queries. Reads used SQL queries or stored procedures and views to get exactly what the screen needs—no more, no less. The write side used SQL inserts and updates to make just the changes necessary to accommodate the user’s actions in the UI. You couldn’t test it and you wouldn’t want to be the one to have to maintain it or enhance it, but it performed well and got the job done.

It did, however, have all the same request/response limitations.

Getting it Right with DDD

Fixing this problem meant an overhaul of the architecture, but the result was well worth the effort. In a nutshell, we redesigned the system to follow the Command Query Responsibility Segregation (CQRS) pattern.

Now all changes to the system are expressed as commands. Anytime the system changes, one or more events are published. Everything is commands in and events out. For reads, we added an Open Data Protocol feed that allows any application to query the data.

CQRS

Figure 5: CQRS Design

Each of the applications and integration points is now sending commands to a central processing module (which may be dispatching to several machines). The commands express the precise intent of the sender of the command. So, UpdateWorkOrder becomes SetWorkOrderComment or SetWorkOrderDescription. Sending the command allows semantic meaning of the action to follow all the way through. We now log both the commands and the resulting events, creating a detailed audit trail of what happened within our system. This also allows us to better avoid merge conflicts because changes are more targeted with the specific commands.

Asynchronous Messaging versus Request-Response

Another huge win for this design improvement is in how it can scale. Building out the “write” side of the system with NServiceBus means that we can ditch all of the home-grown queuing, retry, and pub/sub logic between components. Now, all of our writes are async, forcing us to learn to deal without getting return codes or statuses from our calls. It took some getting used to, but is liberating in the end.

Our load testing shows that the system is now resilient to peak loads. The message queues swell temporarily, but everything continues to function.