Are You Agile? - Doug Durham

Have you ever been asked a question along the lines of … “Are you an Agile dev shop?”

In my experience, this question is asking, “Are you following an Agile methodology?” and not, “Are you agile?”

I think the distinction is important. It seems we have come to a point where whether you are following an Agile methodology is more important than whether you are agile.

I want to take a step back and reflect on what it means to be agile (i.e., to have agility) independent of what methodology you may or may not be following. Spoiler alert: doing Agile does not, by itself, mean we become agile.

What Does It Mean to Be Agile?

The “agile” I care about is an adjective, not a noun. In the context of a software development team, agility is a characteristic that reflects their ability. Being agile to me means that we can quickly and effectively respond to change and opportunity – whatever the source of that change and opportunity may be. The level of agility correlates with the ability of a business to seize opportunities and differentiate itself within the market. Correspondingly, a lack of agility becomes a liability for a business.

What Does Agile Look Like in Practice?

I think one of the things that is becoming lost in all the talk about the Agile movement is the outcomes we are looking for and what it takes to achieve these outcomes. It’s as if the industry is looking for Agile methods to solve all of their problems. I have written before that I don’t think Agile methods should be viewed as a silver bullet. The factors impacting our agility in the complex world of software development are multi-dimensional. Agile methods are a tool, not the entire toolbox.

The multi-dimensionality of this agility “problem” makes it hard to define specifics of what it means to be agile. That said, I know agility when I see it. Below is a list of capabilities that I feel would often be present within teams that are agile. I do not suggest the list is all-inclusive or complete. My point is to get you thinking about what it means to be agile.

Being able to test ideas with minimal investment
Recognizing risk early and taking steps to mitigate
Moving developers within and between projects with ease, minimal friction, and efficiency
Being able to quickly and confidently tell someone how much effort and what changes are required to implement a new feature
Being able to make changes in a system without having to understand the whole system
Not spending weeks of stabilization after every “feature complete” milestone
Not rolling back releases due to errors found in production
Minimizing unnecessary rework by efficiently identifying known-knowns and unknown knowns before development occurs.
Easily understanding and visualizing progress and effort throughout a project
Being able to confidently and accurately communicate expected timelines
Delivering on time and on budget
Detecting 90% of defects before customers find them
Never having to say “we should rebuild everything and do it right this time”
Being able to extend a product with minimal changes to the conceptual design
Ability to push decision making down to the developer with the confidence that their decisions will be consistent with values and design principles of the organization
Having a consistent design methodology that is understood by all and applied to all projects
Identifying, adapting, and adopting new tools, techniques, or processes that improve our efficiency and the quality of our efforts and outcomes

What Does It Take to Become Agile?

One of our goals at Don’t Panic Labs is to become a shining example of what professional software engineering looks like. This is a lofty goal. I define “professional” by comparing the expectations people have for software developers to those of other recognized professions such as doctors, lawyers, pilots, civil/mechanical/electrical engineers, etc. I think we would all agree that our industry has a long way to go before the general public has the same faith in us as they do with doctors or airline pilots. But this should be our goal, and it is our goal at Don’t Panic Labs. It is my firm belief that our ability to achieve the type of agility I described above will correspond directly with our ability to deliver “professional” expectations.

So how do we get there? It is certainly a journey, as opposed to a destination. In the development teams I have been a part of, the one thing I have valued is their ability to see processes and methodologies as tools to achieve desired outcomes. At Don’t Panic Labs, we have assimilated many disparate techniques, but have done so in a lean, lightweight way. We prove the value of a process and then determine where the point of diminishing return is and don’t go beyond it.

So where do I start? The advice I would give folks is to sit down and enumerate the outcomes they desire, prioritize these, and then intentionally and deliberately adopt and adapt processes to achieve these outcomes. You can use the list above as a starting point. Keep in mind that you may not see profound results right away (remember there are multiple dimensions to agility and no silver bullet). The key is to continue to build upon these efforts and layer in new tools and techniques while reviewing the existing tools and techniques.

Case Study: Quality and Agility

As an example, let me walk you through the path we have taken related to improving and maintaining our software quality objectives and creating a culture of quality accountability. It is quite clear to me that the velocity (and morale) of a development team is strongly correlated with how much troubleshooting of quality issues in production they are experiencing. I know of no developer that enjoys constantly revisiting unstable or poorly-written software to resolve some problem.

Research into expected defect detection rates for leading organizations revealed that we should be targeting detection rates greater than 85% (i.e., 85% of defects found prior to shipping code to production). I also know from experience that quality doesn’t “just happen”. You need to be deliberate about your processes and establish a quality culture if you are going to achieve any success.

At a previous company (eSellerate, now MyCommerce), we were able to achieve this detection rate through what I would call “brute force”. We had an army of Quality Assurance (QA) people who went through the software in great detail, over, and over, and over again until it was very stable. While this worked, I was still not satisfied with the result.

I felt that the development team was not taking the proper ownership over the quality of their work. The code that was handed over to QA was often not ready for testing. This created the classic ping-pong scenario where QA and development would go back and forth trying to resolve issues. These quality and stabilization periods delayed releases. When the software was eventually released, it was solid. However, there was very little that was “agile” about our team. This is why I called this the “brute force” quality period. If not for the patience and dedication of our QA team, we would have never reached our quality numbers. I knew there had to be a better way.

When we started Don’t Panic Labs in 2010, I decided not to hire any QA people for the team. I wanted to establish a mindset and culture that placed quality accountability on the developers. Taking this approach naturally increased the amount of scrutiny and desk-checking that our developers were doing before calling their work “complete.” Along with this, we established a consistent design methodology that involved decomposing systems into service-based architectures that are composed of loosely-coupled, stateless/state-aware services. This architecture pattern enabled a high degree of testability, and it made unit and integration testing a core component of our quality processes. Since then we have hired some QA folks, as I believe the QA role is a vital part of any mature development team. However, the quality must begin with the developer.

We also adopted a software estimation process that required decomposing requirements/stories until the estimated level of effort for the story was one week or less. The goal was to reveal hidden assumptions early on so that what was built by the developers was what the product owner was envisioning – another aspect of quality. The combination of these two processes got us to our defect detection rate objectives, but it fell short of ensuring long-term velocity of the developers. We needed something else.

Our main issue with velocity at this point was maintaining the conceptual integrity of our designs. Software entropy is a real phenomenon. If you do not take active steps to overcome it, your systems will degrade and velocity (i.e., agility) will be compromised. A few senior developers at Don’t Panic Labs were doing their best to ensure the conceptual integrity of our systems was maintained – but it was tough.

In 2012, I met with people at Hudl who were using GitHub and their pull request workflow. From the very moment I heard about this, I knew it could be an important part of our quality processes. We immediately began transitioning to GitHub for source control management. We then instituted a pull request process whereby every code change was reviewed (as a minimum) by the lead developer who was responsible for ensuring changes were consistent with our architecture and software design standards. Only the lead developer could merge changes into the master branch. We were now actively maintaining conceptual integrity.

Today all of the above practices give us a layered approach to quality, helping us reach our defect-detection goals. In addition, it enables the long-term velocity and agility we seek in our software products. It did not happen overnight, and there was not some pre-defined recipe for us to follow. Now we are working on developing lead engineers and architects so that we can effectively scale these processes as we continue to grow.

Leaders in our organization are always looking for tools, techniques, and processes that will move us toward our ultimate goal – true, sustainable, business agility.

Looking Ahead

I believe software engineering will only be elevated to the “professional” level of other industries after our industry prioritizes the activities that increase the creation of quality software, and realizes that a single methodology alone will not solve our problems.

Teams should always be on the lookout for new ways to improve their effectiveness (a few of which I’ve included in this post). There’s not always a “one-size-fits-all” approach to enhance a team’s process — it’s up to each one to make these determinations. However, keeping a lookout for these new processes will make a more significant impact than any pre-packaged methodology ever could.