So this post has been on my mind for a while, but something happened at work today….
It would seem to me that it’s human nature to explain occurrences in the simplest manner possible, often in the form of linear “cause and effect” relationships. You know, after you kick a ball, you realize the that the kick (cause) led the ball to move (effect) and so we are always looking for that “kick” that led something to happen. For example, why did we find fewer defects this sprint? Well, it’s because we went from a 3 week sprint to a 2 week sprint.
Is it really that simple? I don’t think so. I’m reminded of the phrase “correlation does not imply causation“. In the domain of systems thinking (and every one responsible for the system should have a basic understanding of systems thinking), we realize that there are often many variables at play in the system.
Take the “fewer defects” example I mentioned earlier. While it is true that the sprint duration was shortened, is it possible that other conditions changed as well? Was the team now a bit more familiar with the work? Was their less pressure because they committed to less in that particular sprint? Was this work less complex than the work before? Was technical expertise that had been missing before made available? What about the conditions we are not aware of? I mean, we can’t be aware of everything can we?
The additional challenge we have is that we rarely have the luxury of repeating our experiments. Every time we try something new, the conditions are different. How many times do we get the opportunity to develop the same feature repeatedly just to capture how effective a change is? I assume rarely, if ever. Our laboratories don’t allow us to reset and run the exact same experiments over and over.
So where does this leave us? (Especially those of us who consider ourselves to be expert problem solvers!). Does this imply that we cannot attribute cause to the things that happen around us? I think we need to be a bit more respectful of the complexity of the problems we often face. We need to understand that the underlying structures and conditions continually impact one another. To use systems thinking parlance, structures reinforce and balance one another. There are often many factors at play and we should be careful latching onto a single one as THE (only) cause. This complicates things for us in some regards, but put us in a position where we can actually dissolve problems.
It is rarely as simple as we often make it out to be.
PS. Some links to my posts on systems thinking.
A current (ok now its old) conversation on LinkedIn was the source for the first post of 2015. (Yes I’ve been slacking a bit).
While on vacation, I spent some thinking about the fact that many organizations have “Product” groups that are separate (read completely different organizational structure) from the “Technology” group. In fact, the only organizations where I have not seen this are the technology startups that I have been involved with.
I happened to bring up this point in the LinkedIn conversation only to be told the following:
Software Engineering – Build what product management tells you to build. Worry about predictability, quality, productivity and extensibility
Product Management – What should be build? What is the business case for it?
Really? Is that really what Software Engineering is about? Building what product management tells you to build? Could it be true? We shouldn’t care about what we’re actually delivering? I believe that such an opinion leaves a lot on the table and yet I’ve found it to be a common opinion in more than one organization. Ever heard this: “don’t ask why, just do it”! Unfortunately, statements like this reveal a complete lack in understanding of software engineering.
I won’t get into a discourse on software engineering as there are many institutions of repute (and Google) that can you help you out with that but I do want to point out an important aspect of software engineering that trained software engineers are very familiar with – requirements engineering. Requirements engineering is all about determining what problem needs to be (dis)solved and what outcome is desired. I realize that “requirements” is often a red word in Agile circles because of its literal application in many command and control type organizations. I mean, who hasn’t heard the “go read the requirements” reply before? But when you really think about it, you’ll see that Agile methods promote performing requirements engineering throughout the entire product development process.
So its my belief that software engineers need to understand “what should be built” and “the business case for it”. In fact, I’d suggest that it’s irresponsible not understanding why. Sure, there may be another group (for example product) focused on idea generation and problem identification but instead of walls between product and development, let’s have a mesh with ideas and information flowing both ways. It is product development afterall, isn’t it? Software engineers ultimately need to understand the problem to be able to deliver the best solution (they are capable of delivering).
As I pointed out here:
To get the “best” solution, it seems to work best IME when those delivering the solution understand “why” and the outcome desired. I have been told many times in the past to do X because it was needed but if I had understood “why” and the outcome, would have done X+ or maybe Y. A key component of software engineering is requirements engineering. Requirements engineering asks us to understand the problem.
So friends, don’t let friends tell you that software engineering is just about building what we are told to build. It’s more than that!
Plans fail, and they tend to fail quite often. In all walks of life, even the simplest plans tend not to unfold as originally designed.
Organizations spend a lot of money and time creating groups solely focused on ensuring that plans do not fail. Senior leaders hire individuals and give them the sole responsibility of managing plans to success. It is quite the lucrative profession to be given such responsibility.
Recently, I asked our church pianist to perform special music at church. Early in the service, a group of children came up to perform special music. As they were singing, I walked over to the pianist to discuss a couple of things but before I could say anything he whispered in my ear “the kids are singing my song!” I couldn’t believe it! What were the odds that the children and pianist would choose to sing the same song on the same day? Pretty small. But it had just happened.
So our best plans fail to go exactly as we thought they would when we drew them up. It is rare that we are 100% certain that things will work out exactly the way we expect them to yet we often behave as if that’s the case. There is always the chance that something unforeseen will swoop in, alter the landscape and impact our plans.
How then do we proceed? Do we stand a chance of getting anything done? Do we have to spend all our effort trying to ensure that our plan unfolds as we originally designed it?
Back to my story… I was beginning to feel very concerned for our church pianist as I didn’t know what he would do given that the children were singing the song he had prepared. At that moment he leaned forward and whispered “don’t worry I prepared two songs, I’ll sing the other one.”, He had prepared two songs! He had options! His options allowed him to adjust when his original plan did not work out as he thought it would have.
When you develop plans, do you identify options for when things go wrong? Do you explicitly consider the fact that even the best plans fail? Some may consider options to be “contigencies”, “alternatives” or “backups”. As I type this post, I’m at Owo-Ahiafor, Nigeria – my hometown. When I planned the trip to Nigeria, I identified options for various pieces of the plan where things could go wrong. Having options is part of developing a risk mitigation technique. It’s common sense but then again common sense is often not that common. Options should be identified based on the likelihood of the plan not working out as originally designed or based and/or on the impact (financial, social, physical) of the plan not working out. In the case of my church pianist, the odds that the two songs he prepared would be sung on the same day, were pretty slim because it was a regular church service. However, if it had been a concert, he may have needed three or four song options to reduce the odds even further.
At the organizational level, I’ve observed that many an organization will spend a lot of time creating plans and yet don’t spend the time to consider the fact that even the best plans fail, ergo, they have no options or scramble to find options when things don’t go according to the plan. As a result of this, certain people in the organization are incentivized to make sure the plan works at all and any cost. Often, they are not the individuals actually executing the plan leading to unnecessary friction within the organization. I’ve lost track of how many times I’ve seen people recognized because they “rescued the plan”. In previous posts, I’ve hinted that outcomes are what we truly care about. The plan is often a means to an end.
Do you have two songs in case someone else sings the song you originally intended to sing. Do you have options for when things don’t work out as planned? As we head into 2015, remember, even the best plans fail.
It’s often suggested to me (via Twitter, LinkedIn, work – thanks Karen), that I’m anti-project managers in Agile (or Lean). This is incorrect (although I am opposed to the way many project managers (and managers in general) actually work but that’s a post for a later date). I do however, have strong opinions on our usage of “projects” in software development. It is the project approach that often leads to the engagement of project manager. These feelings are right up with my thoughts on the usage of the word “resources“.
In the loosest sense of the word, a project can be anything. It can be driving to work, taking a shower or entering time at the end of the day. But we all know that’s a bit silly, we just call those activities. The PMI institute has a definition for project which I like but then uses a software development example that muddies things a bit.
So what is a project? I answer it this way: Is the scope (outputs) needed to achieve the outcome and end investing known – to a large degree – upfront? If the answer is yes, then it feel project-y to me. I’ll admit that this is probably incomplete, but it get’s me started in my assessment of what I am working on. For example, if I choose to remodel my kitchen and largely establish what done looks like from the onset, then I believe I have a project on my hands. How about porting a database from Oracle to SQL Server or moving from Java to J++ to C# (yes I’ve done those things and they are not pleasant)? What about upgrading servers from one operating system version to the next or developing an integration pipeline with a partner so that they send/receive files to/from a system? How about developing and delivering a software solution based on defined scope as is practiced by custom software development shops – and is probably the example PMI is referring too.
These all seem project-y to me as the outputs needed for outcomes to be met and investment to end are known upfront. The decision to stop investing is largely driven by the fact that certain outputs have been delivered. Hopefully outcomes are met as well. This doesn’t mean that discovery doesn’t happen during the process (a good reason to leverage Agile and Lean techniques) but the path is largely charted.
On the other hand, what about building out a supply chain solution or a personal health monitoring platform or a tele-medicine platform (my latest project er I mean startup)? What about starting a blog or having a family (ok now I’m stretching it)? Are these projects? Maybe if one takes a very liberal definition of the word yet I think not. These endeavors are journeys where the end is in often vague and obscure. In a weird way, we actually hope there is no end (at least not soon). We may have some ideas of the outputs needed but we also accept that we are (hopefully) going to spend a lot of time learning about more outputs needed to achieve our outcomes. This is exactly why I spent over a decade at one organization developing a single product. Tell me what’s temporary about that!! We stop investing when we are no longer learning (or making money or having fun).
Should you be approaching your work a bit differently? Are projects your only option?
PS: The project approach to software development can lead to a bunch of bad (organizational) habits but that’s for a later post as well.
Or is it?
Ok, it isn’t but I’m tired of reading posts or comments via various outlets that try to remind us that business agility (or whatever) is THE goal and Agile is not. While there is truth in that, I believe that such statements lead to the proverbial baby being thrown out with the bath water and we end up with faux Agile – afterall it isn’t THE goal is it?
I play the piano. I’m in the middle of learning a jazz arrangement of “Georgia” and I have THE goal of peforming the piece by the end of the year. Let me repeat, THE goal is to be able to perform the piece by the end of the year. In order to achieve THE goal, I actually came up with a goal of practicing the piece daily. (This goal has basically led me to get up at 4am for the past couple of days to practice). If I don’t meet the goal of practicing daily, THE goal of playing “Georgia” by year end is in jeopardy. Practicing daily is a goal I have to meet if I want to achieve THE goal. It’s that simple. Now it’s really easy to making practicing daily THE goal and lose sight of the original goal. If I do that, I probably won’t be performing “Georgia” at the end of the year as well. So I recognize that “practicing daily” is a goal I set to help me achieve THE goal of performing the piece at the end of year and ensure that my daily practices help me get closer to THE goal.
If you believe Agile (as in Agile Manifesto Agile) will help you achieve THE goal (whatever your goals are), then employ Agile (as a goal) such that it helps you achieve THE goal. If you don’t think it will help you achieve THE goal, don’t use it. It’s that simple.
Or is it?
A large number of organizations attempt to to have their software developments go “Agile” for a variety of reasons. The sheer number of discussions in Agile discussion groups is a reflection of this craze. Many of these attempts fail and fail miserably.
A careful read of the Agile Manifesto makes it clear (to me) that the signatories anticipated a certain type of environment for Agile software development to thrive in. Consider the following principles:
- Principle 4: Business people and developers must work together daily throughout the project.
- Principle 5: Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- Principle 8: Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Principle 11: The best architectures, requirements, and designs emerge from self-organizing teams.
Each principle implies a lot about the type of ecosystem in which Agile software development has a chance of being successful. Many organizations that are trying to do or be Agile are only aware of a set of codified practices and are oblivous to the principles that I just mentioned. Unfortunately, the Agile manifesto itself doesn’t explicitly address the ecosystem needed for Agile to have a chance of being successful. Organizations that are interested in Agile software development need to actually look to other sources for information on how to cultivate such an environment. Sources such as Argyris, Deming, Drucker, Weinberg, Ackoff, Senge, Hackman, Pink, Mintzberg (to list a few) are a great place to start for anyone who is interested. If you are a manager and most of these names are not familar to you, I would strongly suggest that you look them up. In fact, if you get nothing else out of this post, I hope you just start a reading list.
The fact, however, is that many (if not most) large organizations have systemic structures (politics, hierarchy, funding models, appraisal systems, control systems etc etc) in place that are at odds with the structures needed for Agile software development to actually succeed. Several thought leaders in the Agile space have largely given up on “the large organization” and have suggested that it’s waste of human potential trying to create the conditions for Agile to succeed in such environments. Others have fused other methods such as Lean Thinking and Theory of Constraints to create a more holistic approach. I’ve especially found the work of Bob Marshall (@flowchainsensei) and Rightshifting to be particular motivating for me in this regard. Let’s take a look at few examples of organizational structures typically found in the large organization and the challenges they present to Agile.
The larger an organization, the more of an influencer “politics” is. People protect their territory (which often includes other people) at the expense of the overall good of the organization. Transparency is lost as a result of folks continually manuvering. Alliances form beind the scenes and nothing gets done. In a highly political charged environment, Agile will struggle.
If your organizations sees technology as an auxillary function (a cost center) that provides utility services then technology will be funded in a such a way. Very often that funding leads to the execution of projects. I’ve written in the past about projects and products so I won’t rehash that here. But, in a nutshell, if your organization delivers projects, it may be challenging (or impossible) for your organization to be Agile. (I realize that the manifesto uses the word, project. Don’t let that mislead you.) On the other hand, if your organization sees technology as a strategic lever with the understanding that product development is primarily an exercise in discovery, then Agile may work for you as you’ll find that you work be doing projects.
This is related to the previous structure but it still warrants being called out explicitly. Cost accounting or throughput accounting? Your accounting structure can impact your ability to truly work in a Agile way. What accounting structures do you have in place?
Traditional organizational structures are largely Taylorist in nature. Managers (and this includes all titles involved in the management) are brought in to ensure that people (very well educated and compensated people mind you) are doing the right things. I’ve also written about this recently and so I won’t repeat what I said. I’m saddened by how many managers are unaware of the role of the manager in an organization. Your organization will struggle with Agile if the organizations view on hierarchy and management remains Tayloristic. If you don’t plan on making a true commitment to enable self-directed teams (see Hackman and Kimball) and network-based structures, then Agile is probably not for you.
Decision Making Structures
Heavily centralized or heavily decentralized ? What is prevalent in your organization? Do people have to run everything up the chain of command for approval before they can actually do their jobs? Is information centralized in a “special few” (think Director of Agile Coaching) roles in the organization and nothing happens if they are not involved? Is leadership at all levels promoted or does your organization just pay lip service to such an idea. Are ideas valuable based on the title of the individual who came up with the idea or are all ideas considered equally? If your organization doesn’t make a concerted effort to push information out to as many people as is possible to enable quick decision making, Agile may not be for you.
It has been said that “culture eats strategy for breakfast”. A study of the Toyota Production System reveals how important the culture of an organization is. I believe the same to be true for any organization interested in the Agile thing. Having teams try to do it on their own is (a) a local optimization and (b) not Agile. An organization interested in Agile (and continuous improvement in general) needs to make a commitment to changing it’s existing structures if it realizes that its structures impede its ability to “do Agile”.
Your can have the perfect cake batter, but if you don’t bake your cake at the right temperature, you won’t have a cake. Is your oven (organization) at the right temperature? If no, what are you doing about it? Can you do anything about it? Better yet, should you do anything about it?
I can barely hear what you are saying.
Yes, your actions are the truest indicator of what you believe; a revelation of the theories that lead to your behavior.
Based on the work of Chris Argyris and Donald Schon, we understand that we all have our espoused theories, for example, let’s say I believe that teams should be allowed to make tool decisions, but then I turnaround and make the decision for the team. My behavior actually reveals the theories that govern how I act. It exposes our mental models which we are often unaware of – yes very often we have no clue. Examples of the difference between espoused theory and theory-in-use abound. Ever heard “there is no i in team” followed by action that says “who was responsible for this “? Or how about, “let’s empower the team to fix their problems” followed by action that says “just give me control and I will fix the problems of the team”?
Unfortunately, many in leadership positions are unaware of the fact that how they say they will/would act doesn’t match how they actually act. Not discussing this inconsistency leads to a huge tax on trust within the organization. “Leaders” are perceived as being deceitful and conniving. The erosion of trust impacts the ability of the organization to deliver and meet its goals. It’s always a pity to see an organization working against itself.
Identifying differences comes from taking the time to reflect on one’s behavior and soliciting feedback from others and comparing the observations and feedback with our espoused theories. This doesn’t happen on accident, it needs to be intentional. Differences should lead to creative tension (Senge). The absence of creative tension implies lip service to an espoused theory; we are saying all the right things just to get by and not because that’s how we want to act or behave.
Remember, our actions speak so loud, people can barely hear what we’re saying (with our mouth).