Throughout my career I have observed that most companies that create their own software are dissatisfied with the quality and the time and money it takes to create. In this post I explore 10 Principles which can help companies achieve better outcomes.

Lets start by defining a couple of terms:

Software Quality

Is a measure of:

• How well software that performs its intended function

• How quick and easy is making changes to the software

Complex versus Complicated Systems

Complicated systems have many steps that generally remain unchanging and can be predicted in advance. A good example is preparing your tax return.

Complicated systems are composed of many dynamic and changing interconnected parts.

Due to the dynamic nature of complex systems, they cannot be solved permanently. They therefore require continuous attention and change if they are not to degrade in quality (see above).

Why Software Delivery is a Team Sport

"If you want to go fast, go alone; if you want to go far, go together."

If the goal is to create high quality software which is a complex system, its going to need more than one person working on it, its going to need a team and in the majority of cases, more than one team working on the same codebase.

How to best organize individuals into software a delivery teams is a question that all companies creating software ask themselves to some degree. While there is no single answer that fits all the different contexts there are key principles that are broadly applicable.

Core Assumptions:

• The goal is to create software as quickly as possible over the medium and long term

• The software is changing both before and after release, therefore it needs to be extended, improved and remediated quickly and cheaply on an ongoing basis

• The software needs to be intuitive, reliable, performant and secure

The Quality / Cost Tradeoff

Common wisdom assumes a tradeoff between quality (see definition above) and the cost and time required for delivery. While this may be true in the short term i.e. delivering low quality software that is hard to change and extend may be cheaper, in the medium and long term the additional cost of making changes in a low quality system quickly grows to exceed any short term gain.

Given the consideration outlined above, that software will be subject to ongoing change, maintaining a quality codebase i.e. one that can be changed and extended with relative ease, is critical to managing the cost and time required for development.

In summary, delivery needs to be organized for a group of people and needs to deliver quality software.

10 Principles for Successful Delivery

1. Small, Cross-functional Teams with Ownership and Accountability Teams should be optimized to deliver business value across a slice of the application quickly with minimal dependency on external individuals or other teams, who may well have different goals and priorities. Teams should have ownership of the this slice of capability and be accountable to its success and performance based on real-world metrics. Strive to give teams ownership of a vertical slice of application capability as opposed to a horizontal layer of the codebase. For example a team should own payments across, backend, frontend UI, third-party integration, monitoring, scaling etc. Rather than having a horizontal ownership such as backend team and a frontend team, as in this case neither team would be able to deliver without work being performed by the other. Cross functional, i.e. the team should contain all the skills and capabilities required to deliver value to the customer. This commonly includes Engineering, Data, Product, Design, Testing, etc. By minimizing dependency outside the team, the team is able to optimize and organize for rapid delivery.
   

2. Fast Feedback Loops Complex systems require rapid feedback to achieve high performance.

   When making a change to the system, which encompasses everything from small incremental changes to code, to the release to customers of a major new feature, the following needs to be understood:
   

   1. Does the change work as intended?

   2. Did the change break any existing capability?

   3. Does the customer like the change and understand how to use it?

   
   

   In order to quickly validate that change meet the above criteria it is critical that Manual Testing which is time consuming, expensive and provides slow feedback, is replaced by Automated Testing and Monitoring. Automated Testing and Monitoring  Depending on Manual Testing to validate changes is not just time consuming and expensive but it also makes feedback to developers very slow. Automated testing has the potential to deliver better outcomes:

   
   

   1. Unit Testing to validate that the a unit of code under development works as intended.
      

   2. Integration Testing to validate that the application is working as intended with components outside the application codebase, for example the Database.
      

   3. Contract Testing to validate that changes to a application service does not break the integration with other services within the application. For example, does a change to the 'user creation service' break the 'customer billing service'.
      

   4. UI Testing to validate that the user interface is working as intended.
      

   5. End-to-End Testing to validate that critical workflows spanning multiple services and integrations are working as intended.
      

   6. Acceptance Testing to validate that the system is and continues to meet business and customer needs.
      

   7. Exploratory Testing is a manual testing approach that attempts to identify quality issues and edge cases that have not been caught by more automated approaches.
      

   Certain tests can be time consuming to create and prone to breaking, such as End to End Tests, this can contribute to teams spending a lot of time maintaining the test suite. Its therefore important to ensure that the right mix of the right kind of tests in the right quantity are being used to meet the goal of fast feedback when problems are introduced.
   

   Finally effective automated application monitoring and alerting systems mean that problems can be identified and anticipated before they impact customers.
   

3. Small, Frequent Deployments to Production-like Environments Production-like Environments Teams should have access to environments which as close as possible to the live Production environment. Ideally Teams should be able to create and destroy these environments on demand but as a minimum they should have a full end-to-end environment that uses production-like data that is dedicated to and owned by that team. Push-button Deployments Deploying to environments, including Production, should be trivial i.e. a single push-button step. When deploying code comes with cost it has a direct impact on the frequency the deployment is performed therefore the steps required to deploy code should be fully automated. This also prevents manually introduced errors into the deployment process a common cause of deployment failure and downtime. Decouple Deployments and Releases in Production using Feature Toggles Decoupling Production Deployments, the introduction of new code into the live Production environment from Release, making changes available to users can be easily achieved by using Feature Toggles. Feature Toggles enable controlling different versions of code for different customers. For example staff members might could see a version of the code that is not yet available to the public. This has a number of benefits:
   

   1. Test on Production Able to validate features and capabilities function as intended within the Production Environment. Particularly useful for Acceptance Testing as it can use Production data.
      

   2. A/B Testing Enables the testing of two or more implementations with customers to see which is the most popular.

   3. Rollback Rapidly Can make a new version of the application available to the customer or roll-back to a previous version with a configuration change only. No deployment required.
      

   Small, Frequent Deployments

   Many types of testing are only possible once changes are deployed to an environment that contains the full application stack. Deploying small changes frequently has two main benefits when problems are identified:
   

   1. Problems are identified quickly which enables the Engineer that introduced the change to revisit the code while it is still fresh in their mind. Attempting to recall and debug code that was created days, weeks or months ago incurs a significant cost in time that should not be underestimated.
      

   2. Small changes mean that the delta of change is small. This makes identifying and debugging a problem significantly easier as it becomes exponentially harder to diagnose the cause of a problem given a larger delta of change.
      

4. Accessible Codebase An accessible codebase is one that is easy to read, understand and change. A highly accessible codebase has a consistent style, is well structured and well documented. Coding Standards A coding standard is a set of rules which defines who code should be written and formatted in a particular codebase. There are common standards or companies can create their own but the value comes from a consistent approach that is followed by the entire Engineering group. Standards can be easily enforced with code commits and deployments failing if they don't meet the Linting rules or similar. Application Architecture The architecture of the application should be well defined and well understood by all Engineers. Ideally the architecture will be characterised by a seperation of concerns and loose coupling of services. The architecture will change and evolve over time and keeping abreast of the current state should be an explicit part of an Engineers' role. Code Review Code Review is a process where changes made are reviewed by another Engineer, often one with more experience or seniority, before being accepted into the codebase proper. While an effective 'gate' to ensure poor quality code does not creep into the codebase in can also slow momentum, consume the time of most productive people and delay delivery of business value. More advanced approaches such as non-blocking code review should be explored once a disciplined practice is in place. Technical Documentation Documentation should be high level, have clear responsibility and ownership and be well maintained. It should also be minimal and exist to enable Engineers to understand and contribute to the codebase, it is not an application manual for non-Engineers. Onboarding Any codebase that has been worked for more than a few months will have unique characteristics, non-obvious decisions, quirks and gotcha's. New Engineers can create unintentionally create a lot of problems and additional complexity if they start making contributions without understand the rules and history. Its therefore vital to the long-term health of the codebase that new Engineers are properly onboarded and have read and understood the Coding Standards, Technical Architecture and Technical Documentation before making any any changes.
   

5. Alignment and Transparency One of the most common challenges with software delivery is building something that customers don't need. As this document is concerned with successful delivery teams lets assume that the product and business strategy is based on solving customer problems. Given that, there are several primary causes of poor alignment between the business and the teams performing the technical delivery work. Disconnected with Business Goals Effective delivery directs its efforts at the same problems that the business as a whole is tackling. In practice there is often a disconnect between the started goals of the business and the technical activities performed by teams on the ground. Working to ensure there is a well understood connection between OKR's or KPI's and roadmap for each team means that even the most junior developer understands how they work they are performing relates to a key goal. There are formal and informal approaches to ensuring this connection is well understood. Once common example is, given a common structure of Initiatives on the Roadmap that then break down into Epics and Stories on a Team delivery board, each Initiative should have a clearly documented rationale for how it contributes to an OKR. Epics should always be tied to an Initiative and User Stories or Tasks should directly relate to an Epic. Poor Visibility of Delivery Team Activity An additional cause of poor alignment is a lack of understanding of what teams are working on. The day-to-day of delivery is full of competing priorities, changing requirements and urgent fixes. This productive chaos often leads to teams working on what they believe to be most important while external stakeholders assume that progress is being made elsewhere. When the discrepancy inevitably becomes apparent, there is often frustration and an undermining of trust on both sides. Teams should therefore aim to broadcast plans and current activity on a regular basis in a high level and easy to consume format. Useful context includes, the current roadmap, progress on current initiatives, known risks, and any major changes to plans or priorities.
   

6. Long Standing, Stable Teams Although software delivery needs to be able to adapt to a dynamic environment and requirements this is best achieved with stable long-standing teams. It takes time for individuals to get to know each others capabilities to understand how to best communicate and collaborate. Changes to a team are highly disruptive and should be kept to a minimum. New teams will take time to go through the forming, storming, norming, performing process. While there is a tendency to associate a team with a specific piece of work and to create and dismantle teams as work starts and finishes this risks losing all the benefits of long standing teams just as they are starting to be realized. Better to consider teams as high quality units of capability that are able to adapt to changing circumstances (within reason). When a piece of work ends then plan to bring a new stream of work to the existing team with minimal changes.
   

7. Disciplined Technical Practices The quality and consistency of the regular technical practices that a team performs are a key determinant of the quality and medium to long term quantity of output. It sounds obvious but poor discipline and a lack of consistency are the extremely common. Some examples include:

   1. Code Review process

   2. Solution Design

   3. Technical Decision Making

   
   

   Technical Leadership is essential to build a habit of deliberate and consistent technical practice. The exact nature of the practices and how they are applied will vary from team to team and company to company and are subject to change as the maturity of delivery grows and the work changes. Teams should be encouraged to experiment innovate to find the most effective ways of working but even during a trial of a potential new approach, discipline and consistency are important if the value of the practice is to be effectively evaluated. While some practices may exist at the team level there are likely practices which need to be consistently applied across all teams that share a codebase.
   

8. Control the Incoming Flow of Work One of the most common challenges that delivery teams face is becoming overwhelmed with work to the degree that quality is sacrificed, timelines blow out and productivity falls. This inevitably creates even more pressure to deliver and teams become caught in a vicious cycle. To prevent this happening teams should maintain tight control over the way in which new work enters the system. Consider the following truths:

   • Estimation of time and effort within complex systems is impossible to do accurately without a major time investment before work starts.
     

   • The more different things a team needs to simultaneously work on, the greater the amount of context switching is required. This has a disproportionate impact on delivery. I.e. a team that is working on 5 problems simultaneously will take much more than 5 times as long to complete as a single problem.

   • Unplanned work does and will continue to happen.

   
   

   Therefore, to keep productivity high and to complete work quickly teams should be focused on solving a small number problems. New work should not be started until the proceeding work has been completed. Stop starting and start finishing.
   

9. Know Your Architecture The Architecture of a system has a profound impact on successful delivery across a number of factors. Firstly, the system architecture is not static, it will grow and evolve over time. If this is not planned for then over time then it becomes less of a planned system and more of a collection of patches, fixes, extensions and revisions. To avoid this its important to be explicit about the desired properties so that they are maintained. There are many different kinds of architecture but some valuable properties include:

   Modularity The code is organized into distinct modules or services, each with a well defined purpose and boundaries. This facilitates ownership of parts of the codebase teams easier and makes deciding where to add new code and capabilities more logical. Integration Patterns Modules and services should pass information between themselves in a consistent manner. Define the Layers The UI Layer, the business logic layer and the data layer are common examples but whatever is used is a specific context its important that these are well defined and documented. Separation of Concerns Whether monolithic or distributed by adopting and following a clear separation of concerns within the codebase the system is organized in a way that each part addresses a single concern or aspect of functionality, rather than mixing multiple concerns together. This approach enhances modularity, maintainability, and scalability of software systems.
   

10. Have an Engineering Culture By explicitly defining the values and behaviors that are desired and fostered within your technical team you are more likely to attract people who share those values. Culture can be powerful force in software delivery particularly around the intangible, hard to measure decisions and interactions. Some examples of values that inform Engineering Culture could include: "We believe long term productivity comes from creating quality, maintainable software" "We believe in working at a long-term sustainable pace" "We believe that innovation comes from experimentation and learning from failure is an essential step" "We strive for excellence in all that we do through a process of continual learning"

Software delivery remains highly complex and achieving a high standard of performance has many factors. If you are frustrated with the outcomes of your current delivery the 10 points above are a good place to start thinking about how to improve.