Practice of Practice Gamelan

Exercise: Wheel of Expertise

Thu, 23 May 2024 00:05:05 +0000

Spin the Wheel of Expertise!

Requirements

People: 3-25
Timebox at 30-90m
Facilitators: 1
Themes that contain expertise: 6-12
“Carnival wheel” with empty pie slices (physical or virtual)

Facilitator Instructions

Session Setup

This game was developed on a small physical carnival wheel with a whiteboard surface.

Divide the face of the Wheel into equal sections. In each, place a one- or two-word theme that connects with some aspect of the complex-system-at-large. These can be anything from a specific technology (‘bucket’, ‘cache’), a layer of your stack (‘Service’, ‘API’), a process (‘failover’, ‘restoral’), and so on.

Fill the Wheel with these terms, somewhere between six and a dozen should be enough to fit. If the game is being played remotely, whoever has the Wheel is the proxy-spinner for each person on the video call.

To Start

Introduce the game with the sections already filled out.
Volunteer to go first.
Spin the Wheel of Expertise (WoE) to get a new term for exploration.
Show by example how you would dig deeper into this term. If you are a novice, show what you’d do to learn more. If you are the expert, guide us through what you know.
- Share your screen.
- Talk through what you’re doing.
- Do what you’d really do to figure this out under pressure, i.e.: “In an incident, how would you quickly learn more about this?”
- Ask for help if you get stuck.
- Each player has ~10m to play.

To End

There is not a real finite close to this activity, typically the timebox ends and the current discussion is wrapped up.

Do some sort of write-up about the session and post it in whatever shared chat channel this community of practice uses. Depending on what your Slack @mention culture is, give people credit for participating. Some examples of this appear below.

Tips on guiding this activity:

Encourage conversation, this is not a test. A good prompt is: “Show us what you know about this from your perspective, and how you’d learn more.”
This is not a technique for putting people on-the-spot. Those not comfortable with doing this activity should not be required to do it, observers are welcome.
Decide if you want to replay themes (which can be interesting on its own) or always spin for a new one (which will reduce choices as the game progresses).
Always encourage players to ask for help if they get stuck. We are here to share.
Respect the time to let enough people, if not everyone, have a chance to spin.
Give the deep-dive a timebox itself, no longer than 10-15m. This might be challenging, an expert may have a lot to say about their area and might need to be reminded of the time.
Conversely, the conversation could become helplessly engaging as new things are discovered. This is exactly the time “to get in the weeds” about it, being flexible about the length of each turn is perfectly acceptible.

Example Activity

No. 1: Who, What, Where

The first version of this game asked to pick a question to answer about the theme, in the form of: “who?”, “what?”, or “where?”. Every participant chose “what” and it became evident that this granularity of questioning wasn’t necessary: picking “what” implied “how” which covered most cases of “who” and “where”. From this point forward, the game was simply known as Wheel of Expertise.

The themes you see are code names for different parts of the application stack: Atlas / Locko / Blueprint / Incident / Blamo / Resto / IAM

WoE the First

Summary of Gameplay

The gameplay is like this: Willing participants spin the wheel. Where it lands on one of our mission critical services, they get to choose: Who, What, or Where. Then they give an answer by sharing their screen and showing their process for finding the answer. They then call on the next participating wheel spinner. They get no help unless they “escalate” and ask for it!

For this session, “What” became an overriding theme. Which was great as we dove into these services learning what they did, what they contained, what they talked to, and what outcomes (or bugs!) they produce. Today we covered (and often made connections between): Blamo (@_user1), Resto (@_user2), Locko (@_user3), Blueprint (@_user4), and “IAM / RBAC” (@_user5 with an assist by @_user6). The wheel did not allow us time with Atlas or Incidental, but there’s always future sessions! @_user7 provided a lot of great input and perspective with deeper questions, and I helped move the discussion along and timeboxed.

One of my favorite parts of this exercise is not only just hearing the answers, but also the sharing of how people find them. Learning from someone’s expertise is often not the answer itself, but peeking inside their mental model and process in coming up with answers… or more questions. This included where people turn first for answers and where they go next when stumped, but also even how we use Zoom to collaborate.

– Matt Davis, Autumn 2021

No. 2: The Sharp End of Storage

Diversity is an important aspect of Practice of Practice Gamelan. In addition to SREs and developers, we had regulars from Product and Marketing and Customer Support. Expanding on the original concept, themes were crafted to allow both technical and non-technical perspectives depending on the expertise and background of the attendees. We wanted to help non-technical people feel they could contribute to the session.

The resulting themes are more generic: Runbook / Error / Search / Cert / Image / Login / Deploy / Queue / RBAC / Bot / Comm / Node / Jira / SDM

WoE the Second

Summary of Gameplay

Preparing for today’s Practice of Practice Gamelan I was thinking about two things: sharp-end work and ambiguity. I set out to create a Wheel of Expertise that matched, but wasn’t sure if the concept would work out or not. By all accounts, I think it did. Our session was a comfortable chamber quartet, with @_user1 randomly spinning the wheel to CERT (not staged! we swear!), @_user2 landing on IMAGE, @_user3 driving into DEPLOY, and @_user4 with the noble ERROR.

This is the conversation about IMAGE that took us through a deep dive of our storage buckets in GCP. Wow. Talk about some deep-diving, sharp-end marvelousness. We stuck with a semblance of the “what?” about these topics, but really it felt a lot more natural than that. We simply took the word and applied it to our own perspective or experience in our work.

To give you a taste: CERT may seem pretty obvious to the guy who refactored our cert-manager system, certainly we got an even more detailed view of what happened in the demo. But ERROR? We got a tour of a sequence of terraform + GCP errors that @_user4 just dealt with in Terraform Cloud. DEPLOY? @_user3 treated us to a CircleCI problem he had the pleasure of wrestling with SRE. IMAGE? @_user4 admitted, he would have gone the route of our Google Container Registry and docker… but this was after @_user2 had taken us through a discovery of the many pieces that come together to store a customer image on the incident timeline (a lot of which I didn’t know and we all discovered together!!!). This was an amazing journey today through a lot of things we discovered along the way. Every single one of us was displaying our own ways of digging the system. We commented several times how nice it was to be able to explore these things in this way outside of cognitive stress and the pressure of an incident.

– Matt Davis, Spring 2022

The Storage Bucket Incident

As you may have guessed, an incident involving the system of the second example occurred not more than a month after we had the session. The SRE On-Call during the incident was present at PoPG that day. All of that detail digging and context uncovering about a legacy service nobody knows that becomes so stressful and haphazard during an incident… it already got done. We did it, slowly, together, without any production pressure.

An important aspect of building functions like Practice of Practice Gamelan is that the organization not only supports the effort, it makes room for the effort to have wings. The things we do in Operations that do not contribute to Feature development are traditionally difficult to get prioritized. Applied Resilience Engineering activities like this and Incident Learning Reviews are considered “expensive activities” because they are not leveraging the “raw talent” of the people involved.

Put another way: if you’re playing a game about incidents for an hour, that means you’re not able to code for that hour. A quantitative Return on Investment for Resilience Engineering is difficult to see, to someone counting the numbers it can feel a lot like buying insurance. Seen in that light, I would judge that we saved no less than the time it took for the session to happen, maybe double, on our “time to clarity” for whatever the hell was going on with this system. Two or three hours given back is a solid quantitative measure for ROI, especially during incidents where seconds matter.

No. 3: Spawn of the Wheel

The format of WoE became a regular go-to for our Gamelan and created several spinoffs. One of them developed into a full-fledged incident training exercise called The Great Incident Baking Show. The goal of the exercise is to use real tooling and real alerting to practice real declaration of incidents that use all the ‘production’ services that would be touched in a real event: the alerting platform, the incident management platform, the ticketing platform, the chat platform, and the processess themselves.

I recall the attendees joking that we were “Chaos Engineering our incident process”, and that’s pretty spot-on for what went down. We not only practiced the process, we dug into bugs in our platform. In fact, Practice of Practice Gamelan is a wonderful place to merely use the tools we leverage for coordination, communication, and collaboration. There were times that we experimented with what we could accomplish with Zoom and times that we were learning its features together.

That’s a more reliable, distributed style of training than deferring any learning for actual incidents. While there are opportunities to learn during accidents, the ability for people to learn drops while we are under stress or trauma. What humans pay attention to in these circumstances is volatile, what they remember even moreso. We can indeterminately learn by being dropped into a stressful situation, but if we can learn these things without production pressure we can more easily avoid becoming traumatized about it (skipping the learning entirely). Plus, we can emphasize the things we want to focus on learning, because such things may not ever come up if we wait for just the right incident.

Themes for this were bigger, our ‘argot’ (or jargon) for the following: ServiceNow integration / Incident Tags / Event Driven Updates / Trial Clusters / Microsoft Teams integration / Slack integration / Communication Workflows

Summary of Gameplay

“In Practice of Practice Gamelan this week we got out the Wheel of Expertise and used it to drive a hands-on exercise: Declare an Incident in Our Platform. This was a time for attendees to experience declaring incidents. To my surprise, the majority of the Gamelan had not! So we did skew the activity to those who haven’t had the opportunity.

A very special shout-out to @_user1 and @user_2 from the Go-To-Market team who may not have known what they were getting themselves into! Y’all did awesome. I confess I put @user_1 on the spot first, but not arbitrarily. As our new Product Marketing lead, I saw an opportunity. So he led by performing some configuration on a new incident Type we are using for this session.

We covered several configuration topics, learning some of the gotchas that hit us when we try to accomplish “Work-as-Done” and butt up against incongruent “Work-as-Imagined” or “Work-as-Prescribed”. This was illuminating, raising questions about limitations that we adapt around to get our jobs done. For example, you cannot define “no retro required” in a Type, it must be selected post-declaration, and when selected completely removes the ability to do a retrospective (even if you want to do a non-required one), not to mention it assumes this means you finished the incident and it will auto-archive the Slack channel. These are things we have to track cognitively instead of allowing the tool to keep the structure for us.

For the rest of the time, @_user3 and @_user4 joined @_user1 and @user_2 in declaring incidents. Those of us who have done this a lot were there to guide and instruct when the person in the hot seat got stuck. The procedure went something like:

Spin the Wheel, get a Function. Not everything on the wheel was a named service.

Declare an incident from our new #gamelan Slack channel, where incident bot announcements are now configured to be sent.

Lookup the Function in the Sorted Service Ownership sheet (which we also spent some time discussing, for instance we learned that one of our Services has ownership split between teams).

Identify what owner team should receive a page.

Switch to the new IR channel created by the incident bot and run through the process of paging the team using the bot, using a special dedicated Test service so all pages went directly to the facilitator (me) as a proxy. [Sidebar: using the PagerDuty integration through the bot actually failed silently on the first runthrough, but then worked fine for the next three. We still don’t know what happened here, but this was a failure in our production service.]

That’s it. We assumed that whatever the circumstance, we had been called upon to declare the incident and page the team. We didn’t go into assigning roles or anything else, this simple exercise prooved to be quite enough on its own.

I have to say I loved @user_2’s turn, because we interrupted them at “life stuff” (grilling onions for lunch I believe?), a very real interruptive scenario for someone answering a call-to-action for an incident. She annoucned that first she would need to take time to take the onions off the stove and then could attend to the incident.

I love these moments. But what I love more is seeing everyone there making discoveries and learning new things. Every time! One of my favorite aspects of doing these kinds of dives in PoPG is that we take the time to chat about what underlying processes are involved. We question why things are done how they’re done, why work is being prescribed or imagined a certain way when we actually perform the work in a different way.

So it’s way beyond merely “how to use the product” and into critical discussions about our featureset, UX design, and integrations. For this reason, I wouldn’t call this “on-the-job training” as much as I would call it practicing to work together. Which is what the Practice of Practice is all about.

– Matt Davis, Autumn 2022

WoE the Many

Reasoning

The Wheel of Expertise and its variants uncover opportunities for experts to share knowledge. It does this in a variety of ways:

Knowledge Elicitation spreads more understanding of the system to more people.
Sharing is done so that we witness real work, not just talk about hypotheticals.
Building reciprocity strengthens “we’re all in it together”, which can reduce anxiety.
Sharing mental models leads to synthesizing new patterns for recognition.
Building empathy between different company functions bridges context about priorities.
Humans learn exceptionally well when engaged in the act of play and discovery.
Assumptions are revealed while areas of little attention get examined.

Whether your exercise deploys a physical wheel or not, it is meant to be a catalyst for conversation. It is a useful way to break the ice and get people talking about things that they may not normally think about on a regular basis, or (even better) have opinions to share about how it works.

Discussion

When I was first hired as an SRE (2013) and first played the game Wheel of Misfortune I was really excited about it. The general idea involves a wheel filled with incident ticket numbers, and the player must explain how they would go about dealing with the problem.

One of the big lessons I learned studying Resilience Engineering is that we’re interested in what makes the system succeed just as much as we want to focus on failure. When I first heard this message it wasn’t that difficult to wrap my head around it. I think I even said aloud in the class: “that’s like when Miles Davis says there are no wrong notes in jazz.”

We who run and make systems reliable are those jazz players, incidents are our unexpected jam sessions. But we are well-rehearsed together, so improvisation - or Adaptive Capacity - comes quickly to us. We are operating there at the “sharp-end” so we can apply direct therapy to support resilient operations.

The “sharp-end” is what the name implies: where instructions and controls issued by the “blunt-end” (i.e.: management or planners) are translated into actionable as-it-happens work. We all have “sharp-ends” when we work, regardless of our titles. They are the set of functions we know we can accomplish (and probably some we merely hope we can).

We don’t actually know very much about what other “sharp-ends” look like, we’re not doing that work, we can only imagine it. Getting to know each other’s actual Regular Work is helpful because it builds reciprocity and empathy, we get to know each other, and we learn where patches of Adaptive Capacity - our ability to improvise as a team - are nourished.

Often the only regular opportunities that bring these edges of work together are incidents. We’re often very focussed on stopping the bleeding. There’s also rarely room in a retrospective to get into the weeds, raise concerns, or ask probing questions about how we work. Learning Reviews get into this area, but they still require the right events that cover what it is we hope to learn.

Sharing in a dedicated, interactive setting like this forms strong bonds. We tend to recall information better when it is learned in a group setting, especially when it includes an element of fun. Some minds learn better when they can physically do the thing, coupling it with a visual experience or a particular place. And then in some cases like the storage bucket incident, an immediate return on investment is felt because of taking the proactive time to talk about our system.

Incidents and problems are not the only way to peer inside the complexity of sociotechnical systems. We can draw upon the same cognitive demands in other forums, outside of production pressure, without the atmosphere of an incident to burden us. The Wheel of Expertise uses interactivity to stimulate our curiosity in a group setting, which helps us share patterns between us, recompiling and integrating with our own dictionary for pattern recognition. This is a game that works especially well with participants from multiple areas of the company, because it strengthens intra-team bonds and emphasizes the work we do in coordination, collaboration, and communication.

Epilogue

One final word on the Wheel and its conception.

This came at a time when we were working 100% remotely all across the globe while a pandemic raged. We needed innovative ways to remain connected as humans across video calls and chat channels. This motivated the kinds of interactive collaboration games I began inventing for us to play weekly.

At the same time, my wife Kary and I picked up the habit of watching Good Mythical Morning while cordoned in-place. At the end of each show, Rhett and Link spin the Wheel of Mythicality and play some kind of mini-game in the follow-up show, Good Mythical More. That’s when I had the idea to see if I could find a wheel of my own to build a game I could play on Zoom calls with our SREs and people On-Call.

It is my version of a Wheel that can play all the notes, regardless of whether they’re wrong or not.

– Matt Davis, May 2024

Exercise: Decision Requirements Table

Fri, 12 Apr 2024 18:04:46 +0000

Build a DRT

Requirements

People: 3-12
Timebox at 30-90m
Facilitators: 1

Facilitator Instructions

Session Setup

Choose a Work Activity. This should be something in need of exploration or discovery. The group gathered should all have some way of touching this Activity, even if it’s in their own domain.
Create a Table. Each participant gets a single row, one item of difficulty each. The columns are:
1. Critical Decision / Assessment
2. Reason Why Difficult / Critical
3. Comments / Notes / Solutions
Gather your session together, virtual or hybrid or physical, it doesn’t matter. Each participant should have the ability to be heard by the entire group. Everyone should get a turn in a round-robin fashion. Let people know they should come prepared to participate.
Introduce the session as an examination of our Regular Work, or “Work-As-Done”. We are coming together to enhance our mental models and learn more about navigating complexities in the system. We will talk Real Work experiences, not hypotheticals or models.

To Start

It isn’t necessary to introduce the concept of a DRT beyond describing what we’ll be doing. The title is self-evident, you want to avoid overloading the session with too much theory. Our goal is to look at this Work Activity, closely examine the difficulties, and try to reason what makes that Activity so difficult.
Announce that you are asking yourself the question: “What is the one most difficult thing about doing this work?” This could be a critical decision, an in-the-moment risk assessment, or an ambiguous situation. Make it real, don’t make it up. It needs to be something that has happened to you.
You are giving an example response so be brief and precise, one sentence or phrase is enough. You are the first row: Put your Decision in column A, your Reason in column B.
Once you’ve presented your topic, if others aren’t already agreeing with it, ask if anyone else has this problem here. Get discussion going about your topic, limit it to just a couple of minutes. Notes can go in column C.
How you pick the next person is up to you. Asking for volunteers can be difficult, especially with people new to this. Instead, you may want to treat it as a game of surprise and require each person to call the next person to go.

Sometimes it will be difficult to keep people away from models and rulebooks and procedures. Language like “we do it this way” or “the procedures say to” are signs we’re moving too far in the abstract. Get things back on track by asking specifics about the occurance of the difficulty. I sometimes ask when in their workday it happened, for example.

To End

Timebox this session and keep a timer to make sure everyone has a fair chance to speak. The goal is the journey, there is no real “end” unless you have identified an outcome through your Work Activity (some Examples below do this, like redesigning On-Call).

Example Activity

Each example reflects real sessions I have facilitated. Some of them had outcomes and followup items because a design was being sought, while some were entirely self-contained as learning and collaboration practice. Both types influenced and inspired teams outside of the session, even running DRT exercises of their own.

Your team has a function that is lacking maturity, you’d like to understand the gaps.
You are designing a command line tool for a group of users, you’d like to understand where they are having difficulties in order to design a solution.
Your organization is redesigning how you do on-call coverage and incident response, you’d like to have a deep and wide survey of what it will take to get there (e.g.: I facilitated four DRT sessions for an On-Call reboot initiative).
A task force has been assigned to improve Developer Experience, you’d like to build bridges with those Developers and understand the maturity of their toolset and processes.
An extremely brittle production deployment process has been orphaned multiple times, you’d like to get a diverse user group to collect as many perspectives as possible to triage.

How would each of these be presented in a session? Do this by translating your Work Activity into a simple question: “What’s the most difficult thing I have to do to accomplish this work?” The more specific it can be, the better. The On-Call quadrouple DRT had multiple variations and different audiences, for instance one of them was dedicated to Post-Incident activities.

There will be times the facilitator may need to get creative about framing a Work Activity in a highly diverse group. We had a customer-facing person join for a session about difficult On-Call response, which was more than welcome. We were able to craft a question that helped them pick out a critical decision point in their evaluation of supporting a customer.

Reasoning

Why do this? Is it worth the time away from writing code?

Here are some of the reasons I believe it is.

People can let off some steam and air difficulties in how they do their work, together.
They feel they’ve made a contribution to a large project that introduces change.
Knowing where expertise lies and how people do their jobs helps us identify, as a team, where our adaptive capacity is: we recall who has the chops to help in a time of need.
People gain a sense of empathy for their coworkers.
Cross-divisional participants gain insight into other expertise, what it takes for that Real Work to get done, which brings teams closer as humans.
Expertise is shared through the stories of how people deal with challenges in the very real way they worked on them.

Supporting these reasons is the idea of keeping it connected to a real event that really happened to us. This dials in the memory of the event, giving it substance. The person can strongly identify exactly what the difficulty was, sometimes accompanied by emotional memories.

This is why a safe space is recommended. Encouraging disagreement and calling out problems while feeling freedom from judgement is the type of atmosphere you want. This is not always possible, but we strive to make it be this way. Everyone is attending because they want to help the community improve with the best intentions possible.

Discussion

Decision Requirements Tables as we did them in Practice of Practice Gamelan can be seen as a form of Rapidized Cognitive Task Analysis.

I didn’t know about this term when I started using this CTA technique. I was reading Working Minds and having all kinds of creative doors opening for me, I had started this Practice of Practice Gamelan session and wanted to invent some group exercises around our work. CTA techniques fit really well, especially TRDs.

So I just took it and ran with it. The idea that we are here to talk about real things is a fundamental aspect of Practice of Practice Gamelan that is mirrored in the way a TRD exercise works. As such, it ended up as the backbone of lots of our gaming and discussion, and in different formats than round-robin spreadsheet building. At one point it was such a loved activity at the company that the Marketing team asked me to do a series of Webinar/Podcasts in which I had guests in the SRE space on to answer the question “What’s Difficult About _____?”.

Accellerated Expertise is a recent compendium that summarizes research in expertise. How people display it, how they come about it, and how we can help spread it to our cohorts beside us. The volume is tied to the Naturalistic Decision Making discipline and makes the assertion that we can accomplish accellerated learning through CTA techniques. When I read this after having led so many of our PoPG sessions, I knew I was on to something good.

In fact, another thing this book shows is that the NDM community needs to know about what we’re doing in software. That’s why I built the popg.xyz site, to start cataloging the CTA work I’ve been doing and talking about at SREcon and many other places. I think there’s some wonderful progress we’re making in Tech that the world wants to know about that is greater than code, it’s the way humans need to come together to make code happen.

– Matt Davis, April 2024

The Practice of Practice

Tue, 07 Nov 2023 23:11:15 +0000

Formal Statement of an Idea

There is this little place in Baltimore called The Red Room. I played a few shows there when I was active in the “Philly/Baltimore/DC” improv scene. I got to know a lot of people through the venue and bookstore and got to witness amazing music (including Jaap Blonk).

One of the things I love about this place (beside the longevity) is how it became a cultural center for like-minded people - artists and audience alike. It instilled in me one of my biggest dreams: to have a place like this where I can run weekly improv sessions and create a community of like-minded people.

It started to make some sense to me that I could find a forum for doing something similar in SRE. Not only was my instinct to create this sort of session something that jived with how musicians do teamwork, it also blended together the things we use to support our own sources of resilience.

My dream comes from a love of coming together through improvisation, by trying something new and experiencing discovery together. My research in and around Resilience Engineering led to formalizing it as a practice. It has become a dimension to SRE work that I will always try to maintain. The reason I’m documenting it on this site is to share the benfits with others in their journey to be expert teams.

Note the word Practice

I have done a lot more research on this subject since writing this over two years ago. The language is aging well, but it might need some annotation (a site lexicon wouldn’t be a bad idea either).

For instance, Theory of Practice here is like sophia: the learned skill of performing a role. Declarative knowledge that can be learned through observation and study, connected strongly to intellectual activity.

By contrast, Practice of Practice finds its germination in non-intellectual activity. Mindfulness. Instincts. Intuitions. Insights. It lies closer to phronesis: practical knowledge that is more like navigating complexity than following procedures. Learned through experience.

Deliberate Practice contains both of these. It takes deliberate practice to become skillful through theories of the technique, it takes deliberate practice to work in concert with a group of humans. Put another way, a group of experts is not automatically an expert team. So both styles of Practice are necessary for good collaboration, communication, and coordination.

Practice of Practice Charter

To build group intuition through a discipline of iterative socio-technical coordination and collaboration.

These words have a heritage in music.

In any type of activity, especially where expertise is seen, there are two forms of practice:

The skill we obtain, that of our craft. The Practice Room. The Laboratory. The Gym. QA. Theory of Practice.
The experience we absorb when we practice our craft. Real life. Improvisation. Production. Incidents. Practice of Practice.

Those final two terms - Theory of Practice and Practice of Practice - are lifted from Derek Bailey’s writing in Improvisation: Its Nature and Practice in Music. He refers eponymously to musicologist Ella Zonis as she describes in Classical Persian Music how the performances do not match theoretical descriptions of how to perform.

Bailey says this is what the book is mainly about: “wherein the dictates of traditional procedures [theory of practice] are integrated with … immediate mood and emotional needs [practice of practice]”.

In other words, it is the emergent blend of craft with intuition. Rote learning with field action.

It is where our entire team deals with surprises at the edges of system boundaries. The most famous example of this is the Chaos Engineering Game Day. Another type I have participated in and led is the Wheel of Misfortune. There are others, like a 20 minute timeboxed data collection exercise of an incident flow in a chat channel.

I am starting up a block of time on Thursdays for these types of sessions and calling it Practice of Practice Gamelan after the Indonesian collection of diverse musical instruments. It also looks like it says Game.

Because these sessions should be fun. They should feel like a small band or a jazz ensemble getting together to play and learning how to improvise as a group. They are informal, not stressful, and the ultimate goal is that we learn how to collaborate together and gain common grounding.

So these words also keep the spirit of music with them.

Matt Davis, May 2021

About

Tue, 24 Oct 2023 19:10:48 +0000

craquemattic is craque is matt is

more to come, click the icons at the bottom of the page for other social media.

Practice of Practice Gamelan

Tue, 24 Oct 2023 06:10:30 +0000

Canon

This is the original full article I wrote in 2021 on Practice of Practice Gamelan while running the weekly program as part of Site Reliability at Blameless.

Officially, it appears (edited for brevity) on The New Stack as Site Reliability Engineering and the Art of Improvisation (or read at archive.org if you want to skip the ads).

There are manifestos before this. I developed the first talks at Shop 23b hackerspace in Fullerton, CA in 2019. In fact a trilogy of talks emerged, drawing their inspiration from how we build intuition in team settings in order to perform during ambiguous and complex situations:

Music in Resilience, aka The Practice of Practice, aka Teamwork in Complexity (SCaLE & many others)
Groove with Ambiguity (DevWeek, LISA21)
Human Observability of Incident Response (SRECon23 Americas)

In the coming days, more content will appear here to document and share my ongoing practitioner research. Building an awareness of improvisation. Helping teams that seek more reliability.

Not only do I think this is of instrumental value to building highly reliable organizations, I think it makes our work in technology more fun.

And that’s worth sharing.

Oct 24, 2023 – Matt Davis, aka craque, aka craquemattic

Part of the trepidation of being on-call is encountering unfamiliar emergency scenarios where we are surprised by suddenly not knowing how to do our jobs. We feel lost and alone, complicated by the world around us, powerless to resolve or even mitigate the problem.

On-call need not be a solo affair full of fear and anxiety. There are ways we can employ practice and open collaboration outside of incidents to prepare us better. We can learn how to adapt when our systems experience breakdowns at the edges, how to improvise when the answers are elusive, and how to evolve our understanding of the jobs we do by sharing our work together.

It is raining. Day one at a developer conference and you think you might have food poisoning from hotel clam chowder. You are the only operations engineer in the company, on-call for server uptime. The pager goes off. You make haste through the courtyard to your room, soaking wet and shivering from the torrent outside.

Shortly you’ve jumped into a chat with all eyes on you as you begin to work through complicated machinations towards system recovery. The counter measures seem like they’re going long, is something failing to rebound? Stuck in a state of dependency loss?

Tensions increase, services should be restarted by now, and the lead developer practically shouts through the internet at you to ask why you’re not just using the web UI control buttons that were made for this purpose. Overwhelmed and flushed by the emotional coordination surprise, you locate the correct sequence of buttons, services recover, and you can finally assume a prone position next to a bucket.

How can we overcome the strain of situations like this? Is there a way for teams to work together so that these moments of common ground breakdown don’t hit us in the middle of high-stress, high-velocity, production incident pressure?

The Practice of Practice

Well, we could practice. But by that I do not mean learning how to technically do our work. I’m not talking about the kind of practice you use to obtain a skill. Rather, this approach is about the kind of experience we absorb when we practice that craft.

Imagine you are back at the hotel, the sickness and rain have both waned and you feel up to exploring. The concierge recommends a cool jazz bar down the block that you can only see if you look for it, and indeed the hole-in-the-wall club appears to be hollowed directly out of the brick. Ducking off the cold street you hear on the far side of the cavern a trio playing piano, trumpet, and standup bass. They all have thick folders in front of them, the words “Real Book” emblazoned in block lettering on the front, but it doesn’t look like they’re following sheet music.

These jazz experts are watching each other, listening and amalgamating with each other’s sound, swaying with a connected funk and syncopated soul that lifts itself like a great big smile out of the floor. It seemed like they would never stop trading twos and riffing on solos and improvising for hours on just one or two pages from that book.

If we compare an incident in software to the improvised jazz show, we can relate to the high-stakes, supremely idiomatic nature of the event. This group met regularly and learned how to play together. They brought their practiced craft of scales, chords, rhythms, and melodies with them to incorporate the practice of jazz improvisation.

Instant Choreography

The kind of practice we can use to help us derive understanding from complex systems and be prepared to improvise intuitively during incidents is what The Practice of Practice entails. I have defined it as an action: To build group intuition through a discipline of iterative socio-technical coordination and collaboration.

These words have a heritage in music, but in any type of activity where expertise is seen, there are two forms of practice:

The skill we obtain, that of our craft, as Theory of Practice. The Practice Room. Fundamentals. Testing.
The experience we absorb when we practice our craft as Practice of Practice. Improvisation. Production. Incidents.

These two ideas - Theory of Practice and Practice of Practice - are lifted from Derek Bailey’s Improvisation: Its Nature and Practice in Music. Bailey was a prolific guitarist and improvisation expert, his seminal book of field research is a series of interviews with musicians from different disciplines and idiomatic styles.

In the introduction he refers to musicologist Ella Zonis describing how the performances in Classical Persian Music do not match theoretical descriptions of how to perform. Bailey says this is what the book is mainly about: “wherein the dictates of traditional procedures [theory of practice] are integrated with … immediate mood and emotional needs [practice of practice]”.

In other words, it is the emergent blend of craft with intuition. Rote learning combined with field action, it embodies how we can interact with the world around us as events unfold, good and bad. The sheet music is only there as a guide, the real beauty of the dance comes with the actions themselves, taking hold of us as we experience them.

Practice of Practice Gamelan

I run a weekly continuous learning session at Blameless that takes its title from the traditional Indonesian orchestra: the Gamelan (pronounced “gah-meh-lahn”). Mostly percussion, lots of tuned gongs and mallet instruments, a few strung and wind ones, usually a male or female singer, all joining together through rhythm and extemporaneous song writing.

You see, a key element of gamelan is that the music is written by the group as it is practiced, with the belief that music should grow and change. As they meet time and time again, members are continuously evolving new versions of the songs every time they play and carry them forward. Practices begin to look like performances, and vice versa. A lot like improvising jazz, where the gig is merely another time to get together and play.

What we’ve created at Blameless is an opportunity for learning and a time to come together in a collaborative way to share mental models and tell stories about different areas of the system in a safe and unpressurized way so we can carry learning forward. This way, incidents are also merely another time we can apply our powers of intuition, because we’ve put techniques for addressing them into practice.

My motto has sometimes been that it doesn’t much matter what we do together as long as we’re doing it together. Regardless of attendance, the discussions always dive into shared perspectives and allow participants a safe space to explore things without fear of judgement or anxiety of an incident. It is impossible for any single person to know the full complexity of networked software, so it becomes critical to know where to find expertise and how to learn from doing instead of trying to follow prescriptions or hastily reviewed runbooks.

One of my favorite things about running these opportunities for learning is seeing the participants employ aspects of their regular work while we answer questions or explore one UI or another. This allows others to peek into their coworkers’ mental models. What might seem like mundane, ordinary tasks to one may illuminate an understanding for another, or even lead others to embellish their own patterns and work style.

Socio-Technical Praxis

Our themes and agendas are somewhat loose but usually pre-planned so we’re not just staring at each other over our instruments… er, keyboards. Nevertheless, sometimes we are required to adapt. There was one session held on the same day as a large vendor outage that disabled our ability to use a portion of our own UI to support that day’s game. So, we pivoted, and it became a session with two of our experts on the subject of the vendor outage (in this case it was root certificate authorities and the SSL/TLS protocol).

Although there is an emphasis on the operational parts of our complex system, the participants are far from just Infrastructure Engineers and SREs. We have sessions including people from Technical Writing, Software Development, Customer Service, Strategy, Marketing, and even Management. We make the calendar invite optional, company-wide, and we do not call it a meeting: it’s a session, where we can share stories and have fun in a live setting.

Some of the other ways we transform and evolve our understanding include:

Walkthroughs of observability toolsets, a.k.a. ‘Morning Vistas’: What do you observe when you open the laptop to start the day and look across the operational landscape? This provides fresh perspectives on how our colleagues approach their regular work.
Decision Requirements Table building, for instance the most difficult decisions faced during on-call or live maintenance to our Kubernetes clusters. These help us think about how we can make improvements to support responders making decisions under duress.
Team Knowledge Elicitation, like deeper views into NGINX Ingress logging or attempting a Dependency Matrix for our Critical Path. Very useful for squeezing some of that juicy knowledge out of our experts’ brains.
Asking the question “Why do we have on-call?” to share mental models of how different people at the company view and engage with it. We learn about each other’s expectations, how we might alleviate the fears of being on-call for the first time.
Spin the Wheel of Expertise! a.k.a. “Who? What? Where?”. Here we explore our technology stack and services through gameplay, asking each individual to spin the wheel and require them to show us first hand how they would come up with the answer (or how they would escalate if they simply didn’t know).

In all these activities we seek to open doors that people might be afraid to go through, learning by experiencing how our peers answer questions about a service or technology. We pick up on the patterns and praxis of others and this enriches our own set of intuition responses, creating new pathways and new connections in our own mental models. This enriches our view of the system and provides the foundation to be adaptive when responding to incidents.

Cognitive Play

Often subjects come up during our sessions that have relationships with the large breadth of research and experience in the field of Cognitive Systems Engineering. For many people this is the first time they are introduced to writings like Sydney Dekker’s “The Field Guide to Understanding ‘Human Error’” or the “Four Concepts for Resilience” as outlined by Dr. David Woods.

We supplement our erudition with Resilience Engineering precepts like the characteristics of reliability and resilience in Dr. Richard Cook’s “Line of Representation”; Rasmussen’s “Dynamic Safety Model” of system Economics, Performance, and Workload; or the “Varieties of Human Work” described by Steven Shorrock as Imaged, Prescribed, Disclosed, and Done.

In fact, Work-as-Done and how experts do their jobs is what Practice of Practice seeks to illuminate. It is here to guide our teams through managing not just efficiency vs. thoroughness tradeoffs, but also how to become experts ourselves. When we support continuous learning in our organizations, we become poised to adapt when we are called upon to improvise during high-stakes events pushing at the unknowable boundaries of our systems.

Moreso, when we encourage play as a method for learning, we’re drawing on the same fundamental principles of education that drive children to explore their burgeoning worlds. We are enriching the unknowable by playing with what we know, so that the doorway to improvising and adaptation becomes a little less scary. We can make decisions in confidence knowing we are part of a supportive group, not lone wolves in the night.

Build to Adapt

In the grand socio-technical scheme of things, the Practice of Practice enables us to build upon the resilience that blossoms like the harmony of well-practiced jazz musicians. The magic and excitement found in discovery is food for our brains, our synapses hunger for enriching pattern recognition, combining new experiences with old ones and other mental models to form new ones.

French philosopher Giles Deleuze might call this emergent approach to learning a sustenance for the “Body without Organs”: the version of yourself that is continuous and forever on the verge of now, the intersection of complexity and consciousness. The superhero-like power of instantly pulling solutions out of seemingly nowhere has its origins in bringing our practiced scales, melodies, theories, rhythms, and other patterns together with inspiring combinations.

Instead of suffering the stressful common ground breakdowns during incidents that translate to a poor customer experience, we seek new ways to choreograph our socio-technical systems more confidently. We see as an organization that there is power in this kind of collaboration, participants have praised these sessions as some of the best on-the-job learning they’ve ever done.

So it is true that having a firmer handle on how to cope with rather than eschew ambiguity comes directly from knowing how to do our jobs better at the sharp end. But we’re not in it alone. We do this by drawing on our rich network of humans in collaborative joint activity, recognizing how our regular work interrelates and feeds into the very complexity we seek to understand.

Not a whole lot different from how musicians influence and support each other through their playing. Imagine how differently extremely uncomfortable events can be lightened by an unassuming session on what choices you have when your very reliable servers go down. Yes, you’re sick and dry heaving into a too-small hotel room trash can, but the team has got your back. This is a situation you have all practiced, so it’s just another time you’re getting together to make music.

References

Bailey, Derek (1980). Improvisation: Its Nature and Practice in Music. Da Capo Press. https://www.goodreads.com/book/show/341932.Improvisation
Cleese, John (2020). Creativity: A Short and Cheerful Guide. Hutchinson. https://www.goodreads.com/book/show/50719532-creativity
Cook, Richard I.; Woods, DD. (2017). Above the line, below the line: A preview of the SNAFUcatchers Stella Report. Keynote from O’Reilly Velocity Conference, NYC. https://www.oreilly.com/radar/above-the-line-below-the-line/
Cook, Richard I. (2000). How Complex Systems Fail. https://how.complexsystems.fail
Crandall, Beth; Klein, Gary; Hoffman, Robert R. (2006). Working Minds: A Practitioner’s Guide to Cognitive Task Analysis. MIT Press. https://www.goodreads.com/book/show/5120871-working-minds
Dekker, Sydney (2014). The Field Guide to Understanding ‘Human Error’. CRC Press. https://www.goodreads.com/book/show/36517806-the-field-guide-to-understanding-human-error
Deleuze, Gilles; Guattari, Félix (1980). A Thousand Plateaus: Capitalism and Schizophrenia. University of Minnesota Press. https://www.goodreads.com/book/show/118316.A_Thousand_Plateaus
Edwards, Robert; Baker, Andrea (2020). Bob’s Guide to Operational Learning: How to Think Like a Human and Organizational Performance (HOP) Coach. Pre-Accident Investigation Media. https://www.goodreads.com/book/show/56659820-bob-s-guide-to-operational-learning
Hamzeh, Farook R.; Alhussein, Hasnaa; Faek, Farah (2018). Investigating the Practice of Improvisation in Construction. Journal of Management in Engineering, Vol 34 Issue 6. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000652
Klein, Gary; Feltovich, Paul; Bradshaw, Jeffrey; Woods, DD. (2005). Common Ground and Coordination in Joint Activity. Organizational Simulation (pp.139 - 184). http://dx.doi.org/10.1002/0471739448.ch6
Mendonça, David; Wallace, William A. (2004). Cognition in Jazz Improvisation: An Exploratory Study. Proceedings of the Annual Meeting of the Cognitive Science Society, 26. https://escholarship.org/uc/item/46v6s1qk
Phillips, Eva C.; Scrinzi, Amy (2013). Basics of developmentally appropriate practice: An introduction for teachers of kindergartners. National Association for the Education of Young Children. https://www.goodreads.com/book/show/18053781-basics-of-developmentally-appropriate-practice
Rasmussen, Jens (1997). Risk management in a dynamic society: a modelling problem. Safety Science, Vol 27, Issues 2–3 (pp. 183 - 213). https://doi.org/10.1016/S0925-7535(97)00052-0
Roud, Ensieh (2021). Collective improvisation in emergency response. Safety Science, Volume 135, March 2021, 105104. https://doi.org/10.1016/j.ssci.2020.105104
Shorrock, Steven (2016). The Varieties of Human Work. https://humanisticsystems.com/2016/12/05/the-varieties-of-human-work/
Woods, DD. (2015). Four concepts for resilience and the implications for the future of resilience engineering. Reliability Engineering and System Safety (2015). http://dx.doi.org/10.1016/j.ress.2015.03.018

Practice of Practice Gamelan

Exercise: Wheel of Expertise

Spin the Wheel of Expertise!

Requirements

Facilitator Instructions

Session Setup

To Start

To End

Example Activity

No. 1: Who, What, Where

WoE the First

Summary of Gameplay

No. 2: The Sharp End of Storage

WoE the Second

Summary of Gameplay

The Storage Bucket Incident

No. 3: Spawn of the Wheel

Summary of Gameplay

WoE the Many

Reasoning

Discussion

Epilogue

Exercise: Decision Requirements Table

Build a DRT

Requirements

Facilitator Instructions

Session Setup

To Start

To End

Example Activity

Reasoning

Discussion

The Practice of Practice

Formal Statement of an Idea

Note the word Practice

Practice of Practice Charter

About

Practice of Practice Gamelan

Canon

A Call to Share Action

Pager Storm Breakdown

The Practice of Practice

Instant Choreography

Practice of Practice Gamelan

Socio-Technical Praxis

Cognitive Play

Build to Adapt

References