Agile: Creator and Destroyer of Value – part 2
The second part of this blog on Agile covers the transformation of Agile as a valuable set of development practices into a money printing machine for trainers, consultants, tool companies and certification bodies. It reduced Agile to a PowerPoint presentation, a set of bullets points deemed applicable to every situation. Those who actually work in the trenches know reality is far more complex than that.
You can find the first part of this blog here.
Lets starts with a positive note. More companies engage external service providers to build custom software. According to a research report by Forrester, the market grew from $43 billion to $136 billion between 2011 and 2015, a CAGR of 33 percent. Due to the widespread adoption of Software as a Service (SaaS), the amounts may look counter-intuitive.
But consider this: with the digitization of business models, companies inevitably start looking for ways to digitalize their strategic differentiators. SaaS solutions are optimized to automate standardized business processes. The business model of SaaS vendors is based on ‘build once, sell many times’. Hence, a company that wants to strengthen its unique strategic differentiators by fusing certain business and IT capabilities will be on the lookout for a custom solution. The high upfront investment is offset by the ability to tailor the solution to the specific needs of the company and ability to own the intellectual property rights.
If I were to believe the stories of Agile Evangelists, the full $136 billion invested in custom software would have resulted in business value if all those projects had adopted Agile development practices. However, according to an article by Vik Bogdanov based on research by Wrike:
“38% of organizations globally are using Agile and only 64% of all Agile projects meet their goals. As many as 70% of companies admit having at least one failed project in 2015. In addition, a staggering 43% of all Agile projects are challenged (late, over budget, and/or with fewer than the required features and functions) and 18% fail completely (either cancelled prior to completion or delivered and never used).”
Similar figures in the Standish Group 2015 Chaos Report. Of the 10,000 projects analyzed for the report, the average Agile project had a 39 percent change of success. For large projects, the success rate was 18 percent and 58 percent of the small projects reported success. While waterfall projects reported even lower success rates, Agile fails to materialize as the knight in shining armor.
Several founding authors of the Agile Manifesto declared Agile dead. Not literally, but they expressed their frustration with the commoditization of their brainchild. Quoting Andrew Hunt:
” What was supposed to be an adaptive, collaborative, and customer-centric approach to development has instead become an over-hyped, buzzword-ridden exercise that delivers almost no value to the organizations that have embraced it.”
Dave Thomas, another founding author is even more pessimistic:
” The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products.
Once the Manifesto became popular, the word agile became a magnet for anyone with points to espouse, hours to bill, or products to sell. It became a marketing term, coopted to improve sales in the same way that words such as eco and natural are.”
One can hire Agile Developers and Agile Testers, buy Agile Tools and become a Agile Certified Practitioner. Apart from the fact that agile is an adjective, not a noun, the Agile Manifesto is “a set of values based on trust and respect for each other and promoting organizational models based on people, collaboration, and building the types of organizational communities in which we [the 13 founding authors] would want to work.” It is not a set of strict rules or methodology. The founding authors expected us practitioners to be pragmatic and use our common sense.
But ‘mushy’ stuff is difficult to sell, let alone certify against. Trainers, consultants and tool vendors need easy to convey bullet lists in their PowerPoint presentations.
The same trainers, consultants and tool vendor also have to be mindful to catch the next wave. In this particular case, expect your consultant, education company or tool vendor to start talking more about DevOps or Continuous Delivery. According to Matthew Kern, DevOps
“is the “heir apparent”. If you buy software development services at a medium or large scale then someone will be in your office selling you DevOps this year. (When they do, they will say it is better than mere Agile. You have to buy it.)”
Miko puts his money on Continuous Delivery as the successor of agile.
“The paradigm of “Continuous Delivery (CD)” seems to be the logical successor to Agile. Continuous Delivery provides an umbrella term that does not specify methodology–and doesn’t require much of a manifesto. Everything you need to know is in the title–you just deliver shippable software in as continuous of a way as possible.”
Or maybe Lean Agile? Regardless of the label, the next wave will come, including the accompanying Peak of Inflated Expectations, Trough of Disillusionment and Slope of Enlightenment. When reaching the last one, common sense kicks in again, a precondition to reach the Plateau of Productivity.
But there are other, more relevant, reasons why Standish Group reports an average success rate of 39 percent for Agile projects.
Zealots and Theorists
William Edmondson suggests that “developers should stop focusing solely on standups, burn-down charts, refinement meetings, or sprint velocity, and start delivering true value to clients, i.e. quality products that meet their budgets and expectations, and are able to provide an appropriate ROI.” While I agree with the message the statement wants to convey, I think the core of the problem is not the developer, but the trainer who taught the developer.
I used to be a consultant and trainer myself so I understand the importance to grab a wave and make money until the wave dies out. At that point, you or your colleague has identified the next wave and the cycle starts again. The first wave I surfed was implementing ITIL V2 processes, followed by BSi15000 and ISO20000. I switched career before the bureaucratic nightmare ITIL V3 arrived (more on that in a future blog).
With every implementation project, I slowly learned to translate theory into practice. Culture, leadership style, business model, organizational complexity and the maturity of both the business and IT all shaped the implementation. So now and then, my pragmatism collided with the views of colleagues who took the term ‘best practice’ literally. These ITIL Evangelists or zealots ignored the organizational context and expected the IT department to strictly follow and execute everything according to the Book (actually ten books in case of ITIL v2, one for every process).
Zealots ignoring context is only part of the problem. Trying to capitalize as quickly as possible on the next wave, consultancy firms and training companies send their employees on a week-long training and certification crash course; and sell them as experts the next Monday. As a result, a trainer proficient in for example ITIL, PRINCE2 or ISO2700 suddenly has to train a group of developers in Agile Scrum. The trainer has no other choice but to stick closely to the standard slide deck. All practical questions from the attendants are answered with theoretical answers. The focus of the certification exam on regurgitating theory only reinforces the mismatch between an imaginary perfect world and the real world.
Similarly, consultants with years of experience in professionalizing support and operations teams are suddenly expected to transition a development team used to waterfall to Agile Scrum. If the experience of the consultant with software development is restricted to being told that ‘waterfall is bad, Agile is good’, the client company is in for a treat. Even more so if the consultant ignores the context the team operated in (e.g. culture, leadership style, business model, organizational complexity, maturity).
You learn to cook by cooking, not by reading a cook book. Only after spending time in the trenches, experiencing real life issues and challenges, a trainer of consultant can truly add value.
The remained of this second part is dedicated to three of the many challenges faced by practitioners.
In part one of this blog, customer focus is mentioned as one of the reasons why Agile appears to be the knight in shining armor. It is also one of its main pitfalls. With the exception of Proof of Concepts and other application with a limited lifespan and complexity, too much focus on early visibility and desires of the business inevitably results in two large and expensive piles. One with technical debt and another with defects.
Unless IT is able to bring balance to the following two forces:
- Feature-focus of business. The payback period of a secure, modular and scalable architecture is much longer than some cool visual gimmick that catches the eye of potential customers. And what about the choice between solving a defect or adding a functional feature? Guess which one is most likely to make it to the top of the backlog when technology illiterate business representatives are driving the backlog? Too one-sided focus on visibility ensures that the attention span of the team is limited to the optics of the current and maybe next sprint demo. As a result, the expected life cycle of at least five years described in the business case approved by the COO and CIO never makes it to the backlog. When technology illiterate business executes are in charge, technology debt and the defect backlog become orphans.
- Power-imbalance. This one acts as an multiplier for the first one. Vanilla Agile pushes IT into the role of Faithful Servant, a positioning described by Van der Zee (I) as “Always helping, wherever possible, no questions asked.” Very different from a Business Partner, which is described as “Top quality, knows what we want, always one leap ahead.” The lower seniority of the developers and scrum master and more technology illiterate the business, the larger the risk (both probability and impact) the project will fail. Giving full authority to the business puts the developers in an underdog position. They are restricted to executing what the Product Owner tells them.
Security (e.g. OWASP 10), architecture, reliability, upgradability, performance and maintainability are neither easy to visualize nor atomize into two-week sprints. They are not part of the happy flow, nor can they be done piecemeal. The same applies to defects. While they can be stored in an ever growing defect backlog, eventually the business has to face the music.
Agile requires the business and IT exit the hamster wheel and plan for the future as equals.
Quoting Moscar’s article Why Agile Isn’t Working: Bringing Common Sense to Agile Principles published on CIO.com: “You cannot replace accountable and responsible project management with some immature utopian myth of self-organization. That’s no science for success.”
I delegate as many responsibilities as possible. It allows me to expand the scope of my control and spend more time on other tasks, which I have plenty of. Micro-managers struggle to keep the big picture in mind and waste a lot of human potential. I don’t like somebody else looking over my shoulder, so why would I subjects others to the same treatment?
At the same time, I was involved in several challenged Agile projects whereby the team failed to sense and act effectively on ‘red flags’. Even worse, instead of asking help from the leadership team, they kept things to themselves (II). I guess one of the bullets in the trainers’ slide deck stated: The development team in Scrum is self-organizing.
Self-organization is a source of flexibility, productivity, creativity and even efficiency. It is also a very complex concept to translate from theory to practice. Furthermore, self-organization doesn’t mean that the team:
- “doesn’t have managers or that they get to decide what problems to solve or what product to build.
- doesn’t have any tooling constraints, or architectural constraints, or budget constraints, or timing constraints.
- decides who is on the team, how big the team is, or how much money everyone on the team should make.”
At least as important in my experience: commitment and the willingness to burn the midnight oil are not enough to be effective as a team. The widely adopted Situation Leadership Model from Hersey and Blanchard explains that individual performance depends on both the willingness and ability perform a task. Most of the challenged projects suffered from decisions made by individuals who were ‘unable but confident’.
Everybody has one or more blind spots: the willingness to take responsibility for a task, but unaware he or she lacks one or more of the required competences or skills. Think of:
- business executives setting unrealistic expectations
- IT promising the business something without overseeing the consequences (e.g. resources, planning budget)
- unable to look beyond theory (e.g. in reality, the business does not have unlimited budget)
- assuming there is no difference between soft skills required to manage a waterfall project or Agile project
- estimating project velocity after developing two simple features and testing the happy-path
- assuming a skyscraper can be build the same way as a one-story house
- ignoring an almost non-existent velocity in the first part of the project, assuming the team will magically make up the difference during the second half
- developing an application module in Java even though the rest of the application portfolio is based on C# (When asked why: “the team wanted to try something different”)
Hence, self-organization is a great goal to strive for, but it is dangerous to underestimate the impact of operationalizing this utopian concept.
Furthermore, most leaders and managers have work within the boundaries dictated by the skill sets, culture, experience and budget available to them. Coaching, training, 360 degree feedback, evaluation meetings and performance reviews can only achieve so much. Somebody used to twenty years waterfall development (being exactly told what to do) will have a very hard time to transition to an Agile approach. Coaching and training can make an introvert but highly experienced and capable developer a bit less introvert. He or she is however unlikely to ever become the life and soul of the party.
Effective self-organization requires leadership, time and common sense from all stakeholders involved.
At the start of the project, the business executive acting as the Product Owner asks the team: build a green car. Two weeks later, at the end of the sprint demo: “Hmmm, one second thought, paint it red”. The following week the business executive visits a potential client (result: paint it white!). Two months later, after visiting a conference with some colleagues: black is new white! Another twelve months pass and out comes a blue car with yellow dots and white stripes on the roof.
Ask critical questions and the typical answer will be along the lines of “Its Agile. Everybody is doing it this way.”
When the scrum team is funded by IT, considerable waste is almost inevitable as there is no incentive for the team to think things through. Not only feature-wise, but also regarding architecture, security and so on. Even when the business funds the project, wasteful practices can go on for a while, especially when senior executives don’t feel comfortable enough to call the bluff of the Theorists/Evangelists/Zealots.
Contrary to waterfall and Spiral Model, there is no explicit exit strategy in Agile. Agile loves changing requirements, even late in the development process. It is one of its strengths, but also a potentially expensive weakness. Left unmanaged (read: lack of executive oversight), the Product Owner and the rest of the team make sure the backlog remains filled. There are always new features, defects, performance issues, optimisations for the Ops team and tweaks.
Hence, proper governance, portfolio management, program management, performance management and even project management did not suddenly become obsolete with the introduction of Agile. Don’t let trainers and consultants fool you. When you have to go back to your CEO, COO, CFO for the fifth time to ask for more budget, you either have a really good story, or it is most likely time to continue your career somewhere else.
Unmanaged, Agile hides low business maturity (e.g. sloppy requirements) and low IT maturity (e.g. code quality, velocity)
In the third part of this blog, I will share several (expensive) lessons from the trenches to better unleash the full potential of Agile.
Notes and references
(I) Van der Zee, H., Measuring the Value of Information Technology, 2003.
(II) I am aware of the possibility that the team members keeps things to themselves due to past behavior the leadership team (e.g. punish mistakes).