Rescuing a Product from Itself

A small application development team that serves a niche market that was quite profitable due to high license fees, was experiencing shockingly bad developer productivity with low volume of new features released in the product. The team had been around since 10 years developing a complex ERP system that had commitment from customers for licenses despite the weak performance of the software due to lack of feature richness that was a result of the failure of the leadership team to get a grip on the drivers of developer productivity and over the delivery process.

The development team was 10 man and with that capacity they could only release 5 -6 medium priority features per year in the last 5 years. Commercially it was unsustainable as due to the capital structure of the business and the cost base, which was high given the output of the team, it required 3 deals (license sales) just to deliver the 5-6 medium weighted features. The issue was that the code quality was so poor, each time a new feature was added, it created extensive regressions that prevented the product from shipping at all; there was no capacity to develop and deliver features that would open new markets or sustain the and expand existing clients.

The team was challenged by a poor reputation on software product quality that was an unstable product, adding a new feature broke existing functionality elsewhere, the team were unable to deliver as promised and that had consequences for the team.

The management put together the first improvement plan that focused on refactoring, hardware for testing environments and build automation with the following goals prioritised:

  • Lower velocity
  • More unpredictable planning
  • Lower output compared to prior years
  • Unwanted attrition in development
  • Low motivated team across the company
  • Team behavior / commitment issues
  • No visibility from IT leadership on the plan to improve development

A root cause analysis was done with the following findings:

  1. The product had after 10 years of development become unmanageable, a veritable rock that would be impossible to refactor without significant investment
  2. The team makeup and balance were wrong, without the needed skills and experience to deliver a product based on standards
  3. The executive leadership of the business were across purposes; constantly bickering with turf wars exploding that created high levels of stress and tension in the organization. There was a lack of leadership top down with nobody being accountable but wanting to be responsible

Any solution had to work within the following constraints:

  • Cost – had to be equal to the current run rate per month
  • Process maturity – the velocity had to become stable and linear

Refactoring the rock – as the product was deemed impossible to rewrite given the levels of cyclomatic complexity at its core, the solution had to posit a product that would be viable for existing clients enabling operations while in parallel, building an improved product that was satisfying the requirements of the business in that it had to deliver:

  • A simplified development process
  • A product (development) road map aligned with business (Sales) needs
  • Stable velocity for forward planning
  • A comprehensive product testing plan
  • A high-level architecture and a product roadmap
  • A plan and process to improve product quality

Tangible evidence of success would be:

  • 2 large high priority features
  • 10 medium ones
  • 1 or 2 extra progressive developments

Baking a layered cake

We were asked to assess this failing development studio and make proposal on transforming it towards a commercially viable entity, at peace with itself. Parts of this brief sounded like “mission: impossible” as there were critical dependencies which had to be addressed if any plan were to succeed. That was organizational alignment. The real problem in most organizations is that functions become silos that are fenced off with no transparency on how they work. The leaders of such silos resist visibility and only accept help with condition that the silo remains in place. That is counter intuitive and doomed to fail but then, that itself exposes another issue. Lack of top down leadership. If a CEO has responsibility for the organization then she needs to exercise that responsibility and align her one downs according to a common and consistent goal. This company lacked that. It was complicated by the fact the owners were in the business operations and delegated leadership that could not be exercised. Anyway, for the purpose of this case study, we can assume that the problem was overcome, and leadership was vertically integrated. With that, gaining commitment should be easy and then the next parts of the plan can execute.

Clearly there would need to be a parallel delivery model in place, one that focused on support for existing functionality that would be in the form of the service desk, and then the product backlog which would create the new product that would take elements from the old and extend them. The product roadmap would be key as that was a requirement to provide visibility to the business for new features and would be a framework on which to hang the detailed delivery plan for the development teams.

A significant issue was the lack of architecture in the existing product. This would have to be addressed. The product was a monolithic single process running on a server where all the logic, UI and persistence data ran. To scale it you would duplicate the server processes using virtualization but that would bring its own issues on stability and performance.  Trying to support such a product, even when it ran correctly (which it hardly did) would challenge the finite resources of the team, so investing in a new architecture that would be scalable horizontally, modular in delivery, maintainable in the code base, easier to understand, easier to deploy, less dependent on tightly coupled components or technology stack would be strong considerations for the new reference architecture.

We felt strongly that whatever the standard adopted; the following patterns had to be respected.

  1. Componentization – of single application into multiple serviceable components is the most important activity. Best would be to start by defining a RESTful API to access this service, then plan and create an implementation using comfortable development language and platform
  2. Collaboration – Communication among bigger teams becomes complex, which results in numerous mistakes and curbing the speed of development. Collaboration among teams focuses on API contracts and Technology standardization.
  3. Connection – The final delivery of an application requires more than the development of the constituent components. These components must be connected, presentation layer and additional services must be layered in, then the completed application must be delivered to users. The use of APIs is considered necessary.

Taking the above into account, we came up with following goals that from a product management perspective, was to center on a strategy which delivered the following benefits:

  1. Lowered learning curve for the business and the technology teams – a common domain model would provide this
  2. Business alignment – the objective was to deliver more features and open new markets at pace this would be delivered by the choice of architecture and the delivery model
  3. Current bottlenecks would need to be scaled – that is the capacity in development and the velocity. This had to be achieved whilst satisfying the cost constraint and we and we proposed outsourcing development to a dedicated delivery team where the following benefits would be delivered
    • Economy of scale using low cost location – developers with specific skill sets could be quickly onboarded and ramped up as the project matured
    • The term length of project was long run and the requirements were scoped and bounded
  4. Faster user acceptance testing – by making the product modular, less monolithic, and more distributed, changes are done in isolation and can be independently verified and validated with less dependencies
  5. By adopting standardized CI/CD tooling and expertise that included removing bottlenecks in environment provisioning, elimination of rework due to mis aligned integrations, elimination of stalled tests, reducing delays due to error prone manual tasks and misaligned requirements that were never implemented and eliminating cycle times due to unavailability of shared resources, people, HW/SW, the cumulative benefits would be in the order of  going from 10 releases per month to above 140, with a deployment time of under 0.5 hours whereas before it was more than 4 hours. The release window was estimated at between 24 hours to 3 days where we now would see 4 hours and defect leakage which was above 40% would be at 1%! All due to increasing the level of automation in the build pipeline from under 20% to just under 80%
  6. Quick to market changes – we saw this as a freebie, an emergent property of the preceding goals.

Due to non-disclosure covenants we cannot go further in the case study, but the point is clear; building a successful product is not about selling a license a couple of times, that could be just luck. There needs to be rigor in the process of delivery of the product and there are many organizations out there who have grown organically with a product that fulfils the mission for a niche set of clients but the team have not scaled themselves to broaden the appeal. Being satisfied with a niche maybe fine but the legacy of the product will die with the team unless you take action to reduce technical debt, strengthen the thought leadership around the product and have a vision for its lifecycle beyond you.

This is what we came up with

About Codacity Informatica Group

We are at our heart, engineers who love change and delivery. That what makes us stand out. We recognize that systems must always start and end with clean code. CIG can power your software development transformation from ideation to execution and benefit delivery. We have more than 18 years of experience driving business change in many settings. We are ready to enable you today.