Closed Source is for Laggards

Open source wins

Introduction

Microsoft is becoming an open-source company, sharing 206 packages in February 2017. In 2016, 16,419 contributors affiliated with Microsoft worked on open-source GitHub projects. The company has embraced the view of Jim Zemlin, the Linux Foundation’s executive director: shared development is enabling faster development with higher quality and lower costs. This is causing the software value chain to change.”

It is a 180 degree turn for a company known for aggressively protecting its IP and which head of the security response team argued in 2001 that closed source is more secure because “because nobody’s reviewing open source code for security flaws.

Why did Microsoft reframe its strategy and what can other technology-driven companies with a traditional business model learn from Microsoft’s radical new view on open source?


 

Shareconomy

Globalization, sustainability, urbanization, digitalization and sharing shape the way we live, work and interact with each other. To thrive, companies have to sense and act on these and other key trends. Sharing, or ‘using instead of owning’, is driven by customers faced by:

  • an abundance of choice (read: supply outstrips demand),
  • a desire for instant gratification,
  • a decline of stable and full-time employment, and consequently
  • a decline in purchasing power.

Technology is an important enabler of sharing as its reduces the friction between ‘customers’ and ‘suppliers’. Apps in combination with a scalable backend platform allow for a free flow of information providing the necessary convenience and trust (e.g. via reviews).

Using instead of owning gave rise to a whole new industry, including companies like Uber (car), AirBnB (house), TaskRabbit (labor), Kickstarter (funding), Wallapop (used goods), Udacity (education), Repair Cafe (repair) and Facebook (personal content). Some are mission-driven, but most are profit-driven or at least a hybrid. According to research by PWC, the size of the sharing economy is expected to grow to $335 billion in 2025 from $15 billion in 2014. Hence, sharing does not necessarily means ‘free’.

According to Havas Worldwide, 25-34 year olds are driving the shareconomy, with 51% preferring share over owning. At least as interesting is the observation by JWT intelligence that 40% share to learn new skills or to support good causes. Which brings me to the first reason why it is important for technology-rich companies to look into open source.

Attract and retain Key Personnel 

Highly skilled developers and other professionals want to work with peers, using online communities to collaborate and challenge each other. They use sharing to demonstrate their cleverness and use feedback to learn. These developers don’t want to work in a black box, unable to communicate with others due to a classic business model where (I)

“commercial software development is based on the exploitation of the monopoly created by copyright for competitive advantage. It makes sense in that system to avoid any process that would undermine the advantage, such as, for example, the sharing of source code with thousands of potentially competing strangers.”

Why is this relevant you might ask. The demand for highly skilled developers outstrips the supply, shifting the power balance towards the developer. Senior and lead developers can choose from a dozen or more vacancies at any day. In  2016, The App Association estimated there were over 223,000 openings for developers in the United States alone. Quoting a recent report by EY on the market in the UK:

“It’s a particularly salient point given that a recent study by O2 suggests that the UK will need to fill more than 750,000 new digital jobs by 2020 and train almost 2.3 million people to meet the demand for digital skills.”

The automation of increasingly complex knowledge work, the internet of things (IoT), the shift towards the Cloud, advanced robotics and next-generation genomics are some of the trends driving an almost unlimited demand for highly qualified software and data professionals. Quoting Marc Andreessen: “Software is eating the world.” And the end is nowhere in sight.

Many leading companies already adapted their business model and hiring practices, including the aforementioned Microsoft with 16,419 GitHub contributors, Facebook (15,682 contributors), Docker (14,059 contributors), Angular (12,841 contributors) and Google (12,140 contributors).

In short,

  • Demand for highly skilled developers outstrips supply
  • Allowing employees to share positions the company as an attractive employer

Open source as driver of new business models

According to Lerner et al (II), “it will make sense for a commercial company to release proprietary code under an open-source license if the increase in profit in the proprietary complementary segment offsets any profit that would have been made in the primary segment, had it not been converted to open source.  While this statement covers the revenue model of traditional business models, it has at least two shortcomings.

First. by focusing on direct revenue streams, it overlooks the previously mentioned effect of open source on the ability to attract and retain developers, the key driver of profit. No developers, no profit. Closed source or not.

More importantly, the statement does not cover the new generation of platform business models which use open source to:

  • quickly boost the size of the ecosystem or platform (e.g. attract developers of apps, games and other source of content that strengthen the customer lock-in)
  • establish open standards (e.g. via ISO, IEEE, W3C or OASIS or another organization with a recognized consensus process)
  • improve brand awareness (spin-off of the first bullet, creating a positive feedback loop)
  • disrupt the competition (e.g. reframe established revenue models within the industry).

Sharing code lowers the entry barrier for developers to build on top of a platform, boosting its overall attractiveness. In 2014, more than 1000 developers contributed to the 107 open source projects initiated by Facebook. By July 2015, Google had released  20 million lines of code and over 900 projects as part of their open source initiative.

Defining open standards is another way to boost a platform business model, like Rackspace and NASA did with their OpenStack project. By being first or at least early in combination with adopting an open source policy, a company can create a community and hence momentum difficult for competitors to catch up with. Quoting Knorr:

“Open source is leading the way in technology development. It’s become the vehicle of choice for startups to gain traction, as customers (mainly developers within companies) take new technologies for a spin, provide feedback, and eventually put them into production. Meanwhile, other developers see what’s hot and start building an ecosystem around a core project, as has occurred with Docker, Hadoop, OpenStack, and others.”

Considering the widespread disruption caused by startups in almost every industry, open instead of closed source should at least be seriously considered. Especially those companies heavily relying on technology.

In short,

  • Open source and open standards are important enablers of disruptive business models
  • Open source allows new business initiatives to scale fast or fail fast

Open source disrupts existing licensing models

Nor Microsoft Office 365 or Google’s G Suite are open source (LibreOffice is), but they provide a clear cut example of changing revenue models within the software industry.

Google’s productivity apps (‘G Suite’) and Microsoft’s Office 365 offer comparable functionalities. Both suites offer email, storage, a word processor, a spreadsheet, a presentation program, and the ability to create webpages to share documents and other types of content. Where Googles offering targets everyday use, focuses Microsoft on doing everything imaginable. Their revenue models reflect this.

Everybody can use Google apps for free due to ad generated revenue. For those power users or companies that want an ad-free environment, support and more storage, a paid premium model is available. In 2015, Google Apps passed 2 million paying companies, demonstrating the viability of the business model and potential to disrupt the de facto monopoly of Microsoft’s Office.

IBM followed a similar path when it was unable to compete with Microsoft in the server space. It adopted open source Linux in order to undercut the license prices of its competitor and regain lost market share. RedHat’s business model is also build on open source. Red Hat shares all its code with the community, relying on value adding support services for its revenue streams.

More generally, business models based on open source software are based on offering:

  • value-added services (e.g. consultancy, training, implementation, optimizations)
  • software as a service (e.g. including service desk, maintenance, hosting)
  • advertising or another cross-subsidy model
  • dual licensing model (e.g. commercial companies have to pay)
  • freemium (e.g. paid optional proprietary extensions)

WordPress is open source and by far the most popular content management system (CMS) with around 15,886,000 websites in January 2017 (50-60% market share). According to Sketch Themes, WordPress was the most requested skill in the world in 2014, with developers charging an average $50 an hour. By January 2015, Freelancer.com had closed 243,161 WordPress projects at a total value of $60,571,205. That is serious money.

In short,

  • Disrupt or be disrupted. Traditional business models have an expiry date
  • Done right, open source can be a money printing machine

Open source as driver of productivity

Productivity is “a measure of the efficiency of a person, machine, factory, system, etc., in converting inputs into useful outputs.” Productivity can be increased by reducing the input for a given output (e.g. initiative to consolidate datacenters to better leverage scale) or increase the output for a given input (e.g. use the same team to create higher added value services). Open source can improve the companies’ productivity by:

  • providing innovative processes to improve operations (e.g. Facebook reporting a 24 percent decrease in cost and 38 percent increase in energy efficiency after switching to open source hardware designs).
  • creating, technology-rich, high value-added value propositions (e.g. The Open Bank Project is an open source API for banks allowing them an secure ecosystem of 3rd party applications and services).

More generally: “It is common for people working for a technology company to suffer, at least a little, from the belief that all the really innovative people in their particular technology happen to work at that company. This can cause such a company to work too hard to produce every last bit of related technology, which is often not the best competitive approach” (III). With Github reporting 19.4 million active repositories written in 316 unique programming languages, smart companies combine externally sourced open source components with their own capabilities to create a distinctive value proposition.

Many companies already found the pot of gold at the end of the rainbow. With  Github reporting 331k+ active organization, including 44% of Fortune 50 companies in 2016, these companies understand the value of being part of a distributed network whereby the sum of parts create more than all the individual parts can (e.g. one bee versus the hive as a whole).

Only laggards think they can keep up with the relentless  increase in complexity and uncertainty by acting like a clamp.

In short,

 


Notes and references

(I)  Andrew M. St. Laurent, Understanding Open Source and Free Software Licensing: Guide to Navigating Licensing Issues in Existing & New Software, 2006.

(II)  Josh Lerner, Parag Pathak, Jean Tirole, The Dynamics of Open-Source Contributors, 2006. In: The Roots of Innovation, Vol. 96 No. 2.

(III) Ron Goldman, Richard P. Gabriel, Innovation Happens Elsewhere: Open Source as Business Strategy, 2005.

Additional reading

More on business models used to monetize on sharing can be found here.

More on why and how Microsoft transformed itself from a traditional software vendor struggling to keep up with the competition into a leader here.

Agile: Creator and Destroyer of Value – part 3

Agile creative mess

Summary

This third blog on Agile emphasizes the importance of critical thinking instead of blindly following what others say or write (me included). Every new software project has its specific context (e.g. relative importance of speed-to-market, ability of product owner to articulate the requirements). Hence, forcing every software development project to be cast in the same mold will inevitably result in a waste of scarce development resources. Furthermore, the name of the game is agility, not Agile.

You can find the first part of this blog here and the second part here.


 

From dogmatic to pragmatic

Some forget that Agile is a means to an end. It is a cog in the machine required to translate the need of the business into an effective and efficient solution. Other cogs include the choice between COTS and custom software, the decision of make or buy, and the Agile readiness of the organization. Combined with the uncertainty and complexity of the requirements, required speed-to-market and expected lifespan of the solution, these topics shape the context the solution has to operate in. A context which allows Agile to either add or destroy value.

Hence, the decision to apply Agile Scrum has to be based on a rational decision making process. Quoting from the article Post-Agile: A Design Thinking Approach to Software Development by Tom Dabson:

“Regardless of how trained and focused your people are, when the projects you work on vary in complexity and time finding a single project management solution that works for all is impossible.”

In order to balance flexibility (‘every project is unique’) and efficiency (‘every project is the same’), the business and IT should agree on a set of principles to govern, develop and support new solutions (‘comply or explain’). Using waterfall in a highly dynamic environment does not make sense, nor does Vanilla Agile for an one-on-one port of a Cobol application to C#. Like the example depicted in figure 1, these principles should enable the stakeholders to objectify decisions regarding the:

  • governance model (e.g. distribution of decision-rights, trust or control-based management),
  • sourcing model (e.g. ASP, SaaS, developed internally or development by offshore vendor)
  • architecture  and technology (e.g. PoC on PaaS platform, business critical applications in Java or PHP)
  • development methodology (e.g. Agile, waterfall, S-model), and
  • operations model (e.g. best-effort support or SLA-based support).

The approach depicted in figure 1 allows for more parameters than Ralph Stacey’s complexity matrix which focuses on the certainty of the technology and certainty of the scope. For native digital and hybrid business models, the context of the solution is shaped by more than these two parameters. With the digitalizing of business models, the protective layer provided by the business to shield IT from the external market is all but gone. Today, evolving customer demand, the entrance of competitors, globalization and new regulations are as relevant for IT as they are for the business.

Context-driven development
Figure 1: Context-driven development (source: book Digital Manifesto).

The left side of the continuum in figure 1 captures the most risky needs and wants from the business. Think of  very complex and/or highly uncertain requirements and/or technologies. Here, desk research, interviews, brainstorm sessions, clickable demos and simulations can be used to provide a rough indication of the required budget and reduce risk profile to a more manageable size. Simulations are useful to validate certain key assumptions before committing scarce and expensive development resources while a clickable demo can be used to visualize one or more key concepts of the solution.

Moving to the right, prototypes and proof of concepts are effective tools to reduce the risk profile of the project without committing to a large scale development effort. They have a limited feature set, lifespan and number of users, allowing both the business and IT to ‘test the water’. Proof of concepts and prototypes can also be used to obtain an accurate cost estimate of the main development phase by realizing a representative cross section of the back log (e.g. 2 simple features, 2 moderately complex, 1 very complex).

The third category is a hybrid; a development approach combining elements from Agile and Waterfall. Think of:

  • A project consisting of two phases:
    • a Minimal Viable Product with a fixed scope, planning and budget in combination with
    • one or more sprints to develop nice-to-haves (typically 20-30% of budget), leveraging on insights gained during the first phase of the project thereby reducing the number of wasteful sprints
  • Fusing elements from project management methods (e.g. PRINCE2, PMBOK) with Agile Scrum / DevOps principles. Whether Agile Zealots like it or not, the business expects a certain result at a certain point in time after spending a pre-agreed amount of resources. A deviation of 10-20 percent is in most cases acceptable, but that’s about it. The project manager supports the scrum master by taking care of the managerial aspects of the project (e.g. report progress and issues in steering committee), allowing the scrum master to focus on the more operational side of the project (e.g. sprint backlog, quality, learning). In most cases, one project manager is able to take care of multiple Scrum teams.
  • Porting the back-end of a legacy application using waterfall while an Agile Scrum team builds a modern front-end in parallel. When modernizing a legacy application, a large part of the functionalities remain the same, allowing for a waterfall approach to maximize efficiency (I). Other areas (e.g. UX, self-service features for customers, mobile apps) benefit from an Agile Scrum approach.

Last but not least, some projects should be realized using waterfall due to their specific context. The business executives of a nuclear power plant have an extremely low risk appetite, while a straightforward one-on-one port of a well-documented legacy application can benefit from waterfall’s higher efficiency (e.g. less idle test hours).

You may have noticed that the continuum depicted in figure 1 closely resembles the life cycle most solutions go through. Every business process and value proposition goes through a life cycle, starting with the translation of a vision or market opportunity into a value proposition customers are likely to buy. This means the business is not looking for a team of risk-averse IT administrators, but colleagues who are pro-active, business savvy and creative. In case the value proposition indeed appeals to a strong customer need, it moves towards the Growth phase. Here, the experimental nature is replaced by the need to scale and become profitable. Investments in fixed assets and a more permanent team structure become a safe bet.

At a certain point in time, the business process matures and the focus shifts from product optimizations (read: add features) to process optimizations (read: improved efficiency). The business demand will change accordingly, resulting in fewer software releases and more focus on IT costs. Hence, with the digitalization of business models, the business life cycle becomes as important for IT as it is for the business. Aligning the business life cycle with the solution life cycle is at least as (if not more) important for IT than the traditional life cycles it tends to manage (e.g. application life cycle, project life cycle, information life cycle).

In short:

  • The decision to apply Agile practices should be driven by the context the solution has to operate in (let’s call it ‘context-driven development’).
  • Agile and other development methodologies are not mutually exclusive.
  • With the progression of time, the context of the solution changes and both the business and IT have to act accordingly.

Insight in the ‘bigger picture’ is not the only precondition to reap the benefits from Agile, DevOps and Continuous Delivery. At least as important is translating that picture into a coherent operating model.

The objective is agility, not Agile

Agile companies develop and nurture the ability to  change their competitive position quickly and efficiently by integrating isolated organizational competencies using a combination of collaboration, coordination, speed and perseverance. Value is destroyed when a business unit operating in a fast moving and dynamic market engages an external vendor to develop a proof of concept, but is forced by the procurement and legal departments to follow traditional contracting practices (e.g. detailed specification, no risk sharing). Similarly, a governance structure and control framework optimized for stable markets (e.g. strict top-down decision-making, fixed multiyear investment plan, comprehensive control framework) can quickly smother any isolated effort to improve the companies’ agility.

Hence, investing in one or more Agile Scrum teams does not equal an agile company. Only when part of a company-wide, top-to-bottom, initiative, an investment in Agile development practices yields a positive return. Even better: an initiative to improve the agility of the end-to-end value chain as down and upstream business partners either add or destroy agility-related value. It is useless as a company to invest in bimonthly release schedule when key business partners already struggle to release their part of the value proposition once a year.

The good news is that, unlike access to rare and expensive earth metals like neodymium and yttrium, agility is in reach of every company. Agility requires flexibility, change, creativity, integration, collaboration and problem-solving; all traits humans excel in. However, this observation also means that implementing ‘agile’ processes, tools, ‘5 levels of Agile planning‘ or a new organizational structure will fail to improve organizational agility.

Agility depends on the companies’ leadership style, culture, management practices and individual skills, as depicted in figure 2. A team with the right skill sets and mindset will automatically come up with the processes, tools and other structure-related items to organize their work more effectively. For them, it will be a matter of common sense.

Organize for stability and agility
Figure 2: People drive agility (source: book Digital Manifesto).

It is equally important to notice that agility and stability enjoy a symbiotic relationship. When successful products mature, value optimization is achieved through a more traditional, efficiency-focused, operating model. The value proposition and delivery processes become standardized and investments focus on economies-of-scale and process automation. This stable part of the company is also important for another reason. Quoting from an interview with Aaron De Smet, senior partner at McKinsey:

“Agility needs two things. One is a dynamic capability, the ability to move fast—speed, nimbleness, responsiveness. And agility requires stability, a stable foundation—a platform, if you will—of things that don’t change. It’s this stable backbone that becomes a springboard for the company, an anchor point that doesn’t change while a whole bunch of other things are changing constantly.”

The same applies to the IT team. Laying off IT staff solely based on the fact that they are responsible for the ‘mature’ part of the IT portfolio is a good way to destroy value. It may take two or even three years, but eventually reality will catch up with your company. Agile is only a part of the puzzle. Every IT solution eventually matures and even Proof of Concepts have to play along with the existing data and application landscape. Hence, there is value in both perspectives depicted in figure 2. It is OK for part of the team to be more introvert, preferring standard procedures and more centralized decision-making. Despite what some Agile Zealots may preach.

In short,

  • An isolated investment in Agile development practices does not improve the agility of the company, business model or value chain.
  • Agility requires stability for most companies and consequently most IT teams.

The last Agile-related aspect covered in this blog is the misconception that Agile teams create value. They don’t. Developers create features which turn into value somewhere downstream in the value chain.

Agile teams create features, companies create value

Value creation is the core objective of every company. Consequently, any decision that creates new value or protects existing value is a ‘good’ decision and vice versa. A company can create and protect value for consumers, shareholders, employees, suppliers, debt providers, government and even the society as a whole.

There is no universally accepted definition of the term ‘value’. It can, for example, be tangible, such as financial, speed-to-market or customer retention, and intangible, like brand value or intellectual property. What most agree on, however, is that at the end of the day, value is most meaningful when expressed by a financial number.

From a customer perspective, value is created in case of a ‘customer surplus‘: “the difference between the total amount that consumers are willing and able to pay for a good or service (indicated by the demand curve) and the total amount that they actually do pay (i.e. the market price).” According to this article in the Financial Times, a company creates value when:

  • The equity market value increases
  • Shareholder value is added
  • The shareholder return increases.
  • The required return to equity is met or surpassed

I don’t even bother with explaining the bullet points as it is beside the point of this blog (just follow the provided link if you are interested or read a book on Corporate Finance). The point is the inability of a Product Owner to be responsible for either customer surplus or value creation at company level. Features, points or the usefulness of realized features/points (often referred to as ‘business value’) do not equal value as the rest of the organization understands it. Similarly, the Earned Value approach considers value ‘earned’ when the feature or point is complete. It is a very useful project management concept, but there is no one-on-one relation with customer and company-level value metrics. Nevertheless, the widely used Scrum Guide states the following:

Page 2: “Scrum (n): A framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value.”

Page 5: “The Product Owner is responsible for maximizing the value of the product and the work of the Development Team.”

Agile’s close cousin Lean does a better job with Value Stream Mapping. Value Stream Mapping looks at the end-to-end value chain, “analyzing the current state and designing a future state for the series of events that take a product or service from its beginning through to the customer.” The development and operations team(s) are responsible for several of those events, but other events take place in the manufacturing, logistics, marketing and sales departments. Together, they create customer and company-level value.

Another widely used method to identify and map value is the Business Model Canvas. It consists of nine elements allowing the team to visualize existing business models and develop new ones. Specifically tailored to startups are the Lean Business Model Canvas and the AARRR metrics. The latter were developed by Dave Mcclure and represent all of the behaviors of a customer (Acquisition, Activation (conversion), Retention, Revenue and Referral).

Value Stream Mapping, visualizing business models and customer development require a cross-functional team effort as they are holistic in nature. They look through functional silos, tying together all organizational elements that create value. For business models with a high technology density, this includes the Product Owner, business analyst, lead architect and/or CTO in order to

  • understand the essence and context of the business need or want (e.g. the problem behind the problem, market and competitor dynamics, key business constrains, the unwritten rules of the game),
  • contribute (e.g. reframe existing business practices, technology as enabler, manage expectations)
  • convey the actual need and wants of the business to the rest of the IT team in order to solve the problem behind the problem.

Agile is a point solution. It allows the development team to release features faster, but that does not necessarily mean they are solving the problem of the customer. Garbage in still results in garbage out. Unmanaged, you just produce more garbage. Value creation in hybrid and digital markets requires a coordinated effort by a team combining business and IT competencies.

In short,

  • Software developers create features or points, the company as a whole creates value.
  • Software development is not the same as customer development

 


Notes and references

(I) Gibbs, R. D., Project Management with the IBM Rational Unified Process: Lessons from the Trenches, 2006.

Additional reading

Agile assumes the organization has, or is willing to, adopt a certain culture and leadership style. Differences in culture can be observed at team, company and country-level. The challenges faced by those who want to implement Agile in Asia are described here, while Germany is covered here.

This article by Nerur et al. describes the importance of accessing the ‘Agile readiness’ the organizational, covering topics including control style, communication and processes.

Some of the founding authors of Agile try to claim Agile back from the Zealots by going back to its core principles.  Dave Thomas wrote Agile is Dead (Long Live Agility) and Andy Hunt and Jared Richardson launched their own rescue effort by introducing the GROWS Method.

Agile: Creator and Destroyer of Value – part 2

Agile creative mess

Summary

The second part of this blog on Agile covers the transformation of Agile as a valuable set of development practices into a money printing machine for trainers, consultants, tool companies and certification bodies. It reduced Agile to a PowerPoint presentation, a set of bullets points deemed applicable to every situation. Those who actually work in the trenches know reality is far more complex than that.

You can find the first part of this blog here.


 

Lets starts with a positive note. More companies engage external service providers to build custom software. According to a research report by Forrester, the market grew from  $43 billion to $136 billion between 2011 and 2015, a CAGR of 33 percent. Due to the widespread adoption of Software as a Service (SaaS), the amounts may look counter-intuitive.

But consider this: with the digitization of business models, companies inevitably start looking for ways to digitalize their strategic differentiators. SaaS solutions are optimized to automate standardized business processes. The business model of SaaS vendors is based on ‘build once, sell many times’. Hence, a company that wants to strengthen its unique strategic differentiators by fusing certain business and IT capabilities will be on the lookout for a custom solution. The high upfront investment is offset by the ability to tailor the solution to the specific needs of the company and ability to own the intellectual property rights.

If I were to believe the stories of Agile Evangelists, the full $136 billion invested in custom software would have resulted in business value if all those projects had adopted Agile development practices. However, according to an article by Vik Bogdanov based on research by Wrike:

“38% of organizations globally are using Agile and only 64% of all Agile projects meet their goals. As many as 70% of companies admit having at least one failed project in 2015. In addition, a staggering 43% of all Agile projects are challenged (late, over budget, and/or with fewer than the required features and functions) and 18% fail completely (either cancelled prior to completion or delivered and never used).”

Similar figures in the Standish Group 2015 Chaos Report. Of the 10,000 projects analyzed for the report, the average Agile project had a 39 percent change of success.  For large projects, the success rate was 18 percent and 58 percent of the small projects reported success. While waterfall projects reported even lower success rates, Agile fails to materialize as the knight in shining armor.

How come?

Money-Printing Machine

Several founding authors of the Agile Manifesto declared Agile dead. Not literally, but they expressed their frustration with the commoditization of their brainchild. Quoting Andrew Hunt:

” What was supposed to be an adaptive, collaborative, and customer-centric approach to development has instead become an over-hyped, buzzword-ridden exercise that delivers almost no value to the organizations that have embraced it.”

Dave Thomas, another founding author is even more pessimistic:

” The word “agile” has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products.

….

Once the Manifesto became popular, the word agile became a magnet for anyone with points to espouse, hours to bill, or products to sell. It became a marketing term, coopted to improve sales in the same way that words such as eco and natural are.” 

One can hire Agile Developers and Agile Testers, buy Agile Tools and become a Agile Certified Practitioner. Apart from the fact that agile is an adjective, not a noun, the Agile Manifesto is “a set of values based on trust and respect for each other and promoting organizational models based on people, collaboration, and building the types of organizational communities in which we [the 13 founding authors] would want to work.” It is not a set of strict rules or methodology. The founding authors expected us practitioners to be pragmatic and use our common sense.

But ‘mushy’ stuff is difficult to sell, let alone certify against. Trainers, consultants and tool vendors need easy to convey bullet lists in their PowerPoint presentations.

The same trainers, consultants and tool vendor also have to be mindful to catch the next wave. In this particular case, expect your consultant, education company or tool vendor to start talking more about DevOps or Continuous Delivery. According to Matthew Kern, DevOps

“is the “heir apparent”. If you buy software development services at a medium or large scale then someone will be in your office selling you DevOps this year. (When they do, they will say it is better than mere Agile. You have to buy it.)”

Miko puts his money on Continuous Delivery as the successor of agile.

“The paradigm of “Continuous Delivery (CD)” seems to be the logical successor to Agile. Continuous Delivery provides an umbrella term that does not specify methodology–and doesn’t require much of a manifesto. Everything you need to know is in the title–you just deliver shippable software in as continuous of a way as possible.” 

Or maybe Lean Agile? Regardless of the label, the next wave will come, including the accompanying Peak of Inflated Expectations, Trough of Disillusionment and Slope of Enlightenment. When reaching the last one, common sense kicks in again, a precondition to reach the Plateau of Productivity.

But there are other, more relevant, reasons why Standish Group reports an average success rate of 39 percent for Agile projects.

Zealots and Theorists

William Edmondson suggests that “developers should stop focusing solely on standups, burn-down charts, refinement meetings, or sprint velocity, and start delivering true value to clients, i.e. quality products that meet their budgets and expectations, and are able to provide an appropriate ROI.” While I agree with the message the statement wants to convey, I think the core of the problem is not the developer, but the trainer who taught the developer.

I used to be a consultant and trainer myself so I understand the importance to grab a wave and make money until the wave dies out. At that point, you or your colleague has identified the next wave and the cycle starts again. The  first wave I surfed was implementing ITIL V2 processes, followed by BSi15000 and ISO20000. I switched career before the bureaucratic nightmare ITIL V3 arrived (more on that in a future blog).

With every implementation project, I slowly learned to translate theory into practice. Culture, leadership style, business model, organizational complexity and the maturity of both the business and IT all shaped the implementation. So now and then, my pragmatism collided with the views of colleagues who took the term ‘best practice’ literally. These ITIL Evangelists or zealots ignored the organizational context and expected the IT department to strictly follow and execute everything according to the Book (actually ten books in case of ITIL v2, one for every process).

Zealots ignoring context is only part of the problem. Trying to capitalize as quickly as possible on the next wave, consultancy firms and training companies send their employees on a week-long training and certification crash course; and sell them as experts the next Monday. As a result, a trainer proficient in for example  ITIL, PRINCE2 or ISO2700 suddenly has to train a group of developers in Agile Scrum. The trainer has no other choice but to stick closely to the standard slide deck. All practical questions from the attendants are answered with theoretical answers. The focus of the certification exam on regurgitating theory only reinforces the mismatch between an imaginary perfect world and the real world.

Similarly, consultants with years of experience in professionalizing support and operations teams are suddenly expected to transition a development team used to waterfall to Agile Scrum. If the experience of the consultant with software development is restricted to being told that ‘waterfall is bad, Agile is good’, the client company is in for a treat. Even more so if the consultant ignores the context the team operated in (e.g. culture, leadership style, business model, organizational complexity, maturity).

You learn to cook by cooking, not by reading a cook book. Only after spending time in the trenches, experiencing real life issues and challenges, a trainer of consultant can truly add value.

The remained of this second part is dedicated to three of the many challenges faced by practitioners.

Early Visibility

In part one of this blog, customer focus is mentioned as one of the reasons why Agile appears to be the knight in shining armor. It is also one of its main pitfalls. With the exception of Proof of Concepts and other application with a limited lifespan and complexity, too much focus on early visibility and desires of the business inevitably results in two large and expensive piles. One with technical debt and another with defects.

Unless IT is able to bring balance to the following two forces:

  • Feature-focus of business. The payback period of a secure, modular and scalable architecture is much longer than some cool visual gimmick that catches the eye of potential customers. And what about the choice between solving a defect or adding a functional feature? Guess which one is most likely to make it to the top of the backlog when technology illiterate business representatives are driving the backlog? Too one-sided focus on visibility ensures that the attention span of the team is limited to the optics of the current and maybe next sprint demo. As a result, the expected life cycle of at least five years described in the business case approved by the COO and CIO never makes it to the backlog. When technology illiterate business executes are in charge,  technology debt and the defect backlog become orphans.
  • Power-imbalance. This one acts as an multiplier for the first one. Vanilla Agile pushes IT into the role of Faithful Servant, a positioning described by Van der Zee (I) as “Always helping, wherever possible, no questions asked.” Very different from a Business Partner, which is described as “Top quality, knows what we want, always one leap ahead.” The lower seniority of the developers and scrum master and more technology illiterate the business, the larger the risk (both probability and impact) the project will fail. Giving full authority to the business puts the developers in an underdog position. They are restricted to executing what the Product Owner tells them.

Security (e.g. OWASP 10), architecture, reliability, upgradability, performance and maintainability are neither easy to visualize nor atomize into two-week sprints. They are not part of the happy flow, nor can they be done piecemeal. The same applies to defects. While they can be stored in an ever growing defect backlog, eventually the business has to face the music.

Agile requires the business and IT exit the hamster wheel and plan for the future as equals.

Self-Organization

Quoting Moscar’s article Why Agile Isn’t Working: Bringing Common Sense to Agile Principles published on CIO.com: “You cannot replace accountable and responsible project management with some immature utopian myth of self-organization. That’s no science for success.”

I delegate as many responsibilities as possible. It allows me to expand the scope of my control and spend more time on other tasks, which I have plenty of. Micro-managers struggle to keep the big picture in mind and waste a lot of human potential. I don’t like somebody else looking over my shoulder, so why would I subjects others to the same treatment?

At the same time, I was involved in several challenged Agile projects whereby the team failed to sense and act effectively on ‘red flags’. Even worse, instead of asking help from the leadership team, they kept things to themselves (II). I guess one of the bullets in the trainers’ slide deck stated: The development team in Scrum is self-organizing.

Self-organization is a source of flexibility, productivity, creativity and even efficiency. It is also a very complex concept to translate from theory to practice. Furthermore, self-organization doesn’t mean that the team:

  • “doesn’t have managers or that they get to decide what problems to solve or what product to build.
  • doesn’t have any tooling constraints, or architectural constraints, or budget constraints, or timing constraints.
  • decides who is on the team, how big the team is, or how much money everyone on the team should make.”

At least as important in my experience: commitment and the willingness to burn the midnight oil  are not enough to be effective as a team. The widely adopted Situation Leadership Model from Hersey and Blanchard explains that individual performance depends on both the willingness and ability perform a task. Most of the challenged projects suffered from decisions made by individuals who were ‘unable but confident’.

Everybody has one or more blind spots: the willingness to take responsibility for a task, but unaware he or she lacks one or more of the required competences or skills. Think of:

  • business executives setting unrealistic expectations
  • IT promising the business something without overseeing the consequences (e.g. resources, planning budget)
  • unable to look beyond theory (e.g.  in reality, the business does not have unlimited budget)
  • assuming there is no difference between soft skills required to manage a waterfall project or Agile project
  • estimating project velocity after developing two simple features and testing the happy-path
  • assuming a skyscraper can be build the same way as a one-story house
  • ignoring an almost non-existent velocity in the first part of the project, assuming the team will magically make up the difference during the second half
  • developing an application module in Java even though the rest of the application portfolio is based on C# (When asked why: “the team wanted to try something different”)

Hence, self-organization is a great goal to strive for, but it is dangerous to underestimate the impact of operationalizing this utopian concept.

Furthermore, most leaders and managers have work within the boundaries dictated by the skill sets, culture, experience and budget available to them. Coaching, training, 360 degree feedback, evaluation meetings and performance reviews can only achieve so much. Somebody used to twenty years waterfall development (being exactly told what to do) will have a very hard time to transition to an Agile approach. Coaching and training can make an introvert but highly experienced and capable developer a bit less introvert. He or she is however unlikely to ever become the life and soul of the party.

Effective self-organization requires leadership, time and common sense from all stakeholders involved.

Institutionalized sloppiness

At the start of the project, the business executive acting as the Product Owner asks the team: build a green car. Two weeks later, at the end of the sprint demo: “Hmmm, one second thought, paint it red”. The following week the business executive visits a potential client (result: paint it white!). Two months later, after visiting a conference with some colleagues: black is new white! Another twelve months pass and out comes a blue car with yellow dots and white stripes on the roof.

Ask critical questions and the typical answer will be along the lines of “Its Agile. Everybody is doing it this way.”

When the scrum team is funded by IT, considerable waste is almost inevitable as there is no incentive for the team to think things through. Not only feature-wise, but also regarding architecture, security and so on. Even when the business funds the project, wasteful practices can go on for a while, especially when senior executives don’t feel comfortable enough to call the bluff of the Theorists/Evangelists/Zealots.

Contrary to waterfall and Spiral Model, there is no explicit exit strategy in Agile. Agile loves changing requirements, even late in the development process. It is one of its strengths, but also a potentially expensive weakness. Left unmanaged (read: lack of executive oversight), the Product Owner and the rest of the team make sure the backlog remains filled. There are always new features, defects, performance issues, optimisations for the Ops team and tweaks.

Hence, proper governance, portfolio management, program management, performance management and even project management did not suddenly become obsolete with the introduction of Agile. Don’t let trainers and consultants fool you. When you have to go back to your CEO, COO, CFO for the fifth time to ask for more budget, you either have a really good story, or it is most likely time to continue your career somewhere else.

Unmanaged, Agile hides low business maturity (e.g. sloppy requirements) and low IT maturity (e.g. code quality, velocity)

In the third part of this blog, I will share several (expensive) lessons from the trenches to better unleash the full potential of Agile.

 


Notes and references

(I) Van der Zee, H., Measuring the Value of Information Technology, 2003.

(II) I am aware of the possibility that the team members keeps things to themselves due to past behavior the leadership team (e.g. punish mistakes).

Agile: Creator and Destroyer of Value – part 1

Agile creative mess

Summary

Agile development is not a silver bullet, there are numerous situations where the Spiral Model or even waterfall is a better choice. More importantly, methodologies are always trumped by common sense and highly skilled team members. Only these latter two always result in success.


 

There is a lot of value in Agile, but only when combined with a healthy dose of common sense. Therein lies the problem.

All but a few pitch the Agile Manifesto as the greatest thing since sliced bread. Consultants and trainers earn large amounts of money by selling Agile as the solution for all development-related problems. To them, the Agile Manifesto is like the Bible.

In reality, many Agile projects take forever to complete or are abandoned after failing to deliver the desired results. I see companies rehiring the project managers and architects they fired a couple of years ago as part of implementing Agile. These companies went through the Peak of Inflated Expectations and are now crawling out of the Trough of Disillusionment.

The good

Every new day is more unpredictable and dynamic than the day before. The higher the uncertainty of a market, the more the business and IT crave for flexibility and experimentation. In these situations, the business cannot articulate and fixate all requirements upfront, a prerequisite of traditional waterfall programming. At first sight, Agile enters the scene therefore as the knight in shining armor as it:

  • puts the customer first,
  • delivers rapidly (e.g. develop and deploy most valuable requirements first),
  • strives to eliminate waste (e.g. no elaborated requirements analysis for a proof of concept),
  • amplifies learning (e.g. customer feedback after every sprint demo),
  • makes decisions as late as possible (e.g. adjust priorities backlog along the way),
  • empowers the team (e.g. encourage instead of motivate),
  • has built-in integrity in (e.g. soft controls, authority based on expertise not hierarchy), and
  • sees the whole (e.g. work towards a shared business goal).

All good stuff, which makes Agile such an easy sell. Vanilla Agile (I) is actually great to realize proof of concepts and to determine the expected velocity of a large development project. In the latter case, several sprints are used to develop several features of various complexity, thereby providing the project team with an fairly accurate proxy of the velocity of the project as whole.

Realizing the software equivalent of a fifty story building with Vanilla Agile is however not such a good idea. More likely than not, the building will collapse before reaching the tenth floor.

Before moving towards the bad and ugly stuff, first some neutral, but nevertheless relevant, observations about Agile.

Agile is a business concept

Samsung introduced its wearable Gear, and even the Gear 2, before Apple introduced its Apple Watch. The reviews of especially the first generation Gear smartwatch were mixed at best, and consumers had a similar view considering a 30% return rate in Best Buy locations (II). For Samsung, being first was more important than offering the best experience. To mitigate the risks associated with this go-to-market strategy, Samsung uses a relatively high release frequency for its products. Apple follows a different strategy. Products are tested and refined within the company walls until considered mature enough to appeal to the customer segments targeted by Apple. Both companies use iterations, but their implementations differ.

The business and IT teams of a successful upscale brick-and-mortar retailer tasked with digitalizing the business model face a similar challenge: make the new digital platform available to all customers as soon as it reaches Minimum Viable Product status or wait until a more polished release becomes available.  The main benefit of the first option is speed-to-market, reaching wealthy online customers before the competition does. However, launching an unpolished platform to high-end customers may also alienate them.

Hence, while Samsung, Apple and the retailer all benefit from iterative sprints and other Agile-related concepts, there is no single “best” implementation of Agile. Depending on the market (e.g. risk taking/averse customers, low/high competition) and strategy (e.g. Prospector versus Defender), the business and IT have to decide on the company-specific implementation of Agile (III).

Due to the digitalization of business models, Agile is first and foremost a business concept, even though it was created to improve the agility of IT.

Agile did not replace waterfall

The 8 million lines on board of the F35 Lightning II are embedded in thousands of components, such as the radar, engine and cockpit, manufactured by hundreds of suppliers. To ensure they, nevertheless, function in perfect harmony, the architecture, technologies and functional requirements all have to be defined upfront. Furthermore, all the code required to fly has to be fully functional and error free when the plane taxis to the runway for the first time. There is no room for continuous delivery when in there air, nor a long defect backlog. The same applies to the software used in nuclear power plants or x-ray machines. They all require a first-time-right approach.

Another situation that benefits from a more traditional development approach is an one-on-one conversion of a well-documented and stable application written in a legacy language. Here too, the potential benefits of Agile (e.g. demo progress to customer, frequent deployments) are outstripped by the additional costs related to iterative development. More generally, continue to consider waterfall-oriented or the V-model (Validation & Verification model) in situations where:

  • the requirements are or have to be understood completely at the beginning
  • the first deployment has to cover the full scope and/or has to be error free

Besides more efficient staffing and resource allocation, waterfall development also allows for more room for more time-intensive tasks like documenting user cases and drafting manuals for the operations and support team (IV). Sometimes the business needs a fast, agile and fuel-thirsty speed boat while other situations call for a predictable, robust and efficient container ship.

Agile and waterfall both have their sweet spots, they are complementary.

End of part 1 of this blog on the pro’s and con’s of Agile.

 


Notes and references

(I) I use the term ‘Vanilla Agile’ when implementations and/or trainings are based on theory only, ignoring common sense and lessons learned from practical experience.

(II) Grant, R.,  Samsung Galaxy Gear smartwatches have embarrassing 30% return rate, Venturebeat.com, October 2013.

(III) See also these blogs: 1, 2, 3.

(IV) Gibbs, R. D., Project Management with the IBM Rational Unified Process: Lessons from the Trenches, 2006.