Feeling the Pain, A Framework For Solving Obscure Problems
I don't tend to have too many mantras when it comes to engineering leadership, but one key saying that I keep coming back to over and over as the years progress is, “feel the pain.”
No, this isn’t some riff on “no pain, no gain,” it’s more about problem identification and problem solving.
In this case, the pain is not physical, it's more the emotional pain or the difficulty to get something done. Whether it’s with software or within an organization, you have to feel the pain and sometimes you have to experience it in order to really get a sense of how to solve the problem. I’ve helped build out Triage processes at three separate organizations and have used this methodology at each. It’s a few simple steps, though the timeline for each is different:
Identify the real problem
Feel the pain yourself
Let others feel the pain
Rid the pain together
The goal for all of this is solving the problem, solving the right problem, and making future solutions faster. You’ll never be able to gain all the necessary context that your stakeholders have in their brains, but you can short-circuit the solutioning process.
The problem is not always the problem
I continue to find myself working on projects and teams that are internal focusing, giving myself and my teams a direct connection to internal stakeholders. One of the reasons I really enjoy working with these teams is because you get a very quick feedback loop, whether you want it or not.
If you deploy code and it immediately breaks a workflow, you hear about it right away. Who needs monitoring when you’ve got Slack notifications from 30 frustrated users?
Of course you should want to be able to detect business-halting issues with monitoring, testing, or observability, but you don't understand the pain as quickly as getting a Slack message that says, “I can’t do my job because this workflow is borked.” As I've worked with internal teams, it has really helped me understand problems as they come up. But the problem they identify isn’t always the true problem.
When a workflow breaks, the end user might not be close enough (or in some cases is too close) to understand what is actually happening. “Oh, hey, I can't do this thing anymore.” If we take the surface-level statement, we might go on a wild goose chase trying to solve the problem.
It’s possible the update removed an extra click in the workflow, making it inherently more efficient. When someone is in the same application every single day, and they're trying to do the same task every single day, one small change can disrupt the entire process that they're used to, even if the intent was improved efficiency.
Change management can help with the above, but it's important to remember when this is brought up, you don't just take it blindly and say, “Oh, yep, sorry, we'll, we'll put that back.” Instead, you must dig a bit further to understand the true issue.
When you have a direct line to your end-user, the fastest path to a solution is to feel the pain yourself.
Feeling the Pain Yourself
I do this in pretty much every organization that I go into.
Learn about the problem
Put myself in the middle of the problem
Get pelted by the problem until the root causes become clear
Start planning how to resolve the problem
I start with feeling the pain myself. “We get a ton of triage tickets or a ton of requests.” All right, cool. Bring them all to me. Wat. Yes, bring them all to me.
By shoving myself into the center of the problem, I become a funnel. And for the low price of free you can become a bottleneck too!
The goal here is not to be a bottleneck, but to see all of the problems, understand the problems, and begin pattern-matching. Ask yourself:
Are there commonalities in the problems?
Can we categorize the problems?
Do we already have systems to solve the problems?
Is this a tech problem, process problem, or training problem?
How important are these problems?
To quote Sun Tzu (facepalm): “If you know yourself but not the enemy, for every victory gained you will also suffer a defeat.” It’s all too easy to take a ticket one at a time, never looking at the whole and getting stuck in a rut (aka suffering a defeat).
Collect, analyze, and formulate a path forward. That’s the core strategy of feeling the pain yourself. Once you can articulate the problems, you can begin to articulate their solution. Then as you begin to think of solutions, consider letting others feel the pain.
Letting others feel the pain
It's very easy to stay as the only person feeling the pain, but
I worked with an individual who was taking on a lot of that responsibility related to triage resolution and when they left, the dam broke and we suddenly had hundreds of triage tickets to deal with every week. The individual also had write permissions to the database, which made resolution easy, but was also a giant security risk.
While this individual was doing quite a bit, they were also preventing the engineering team from understanding the true issues. In other words they weren’t feeling the pain, so they didn’t know there were broader issues.
By pulling in engineers to think about the problems we’ve collected and letting them think about the patterns of problems, one or two features can eliminate 30-50% of the problem. Mileage may vary across industries, but because the engineers know the product already, they can start solving the right problem.
It's too easy to think as a manager, “oh, I'm protecting my team and letting them focus instead of dealing with the little things.” Not untrue, but you're also never solving the problem holistically. Work the fixes into your sprint plans to ensure you are making consistent progress to reducing the broader issues until the list dwindles.
Ask your team the question you’ve already asked yourself:
Do we already have systems to solve the problems?
Is this a tech problem, process problem, or training problem?
How important are these problems?
Software will never be without issues, code away the 80% problems and find simple workarounds (likely manual) to deal with the less frequent 20% of problems (edge cases).
Ride Alongs
One last idea is you don't necessarily need to be the one that feels the pain. You can toss the developer (or yourself) straight into the problem with a primary user to see how they experience the pain firsthand. I tend to push this idea as developer ride alongs with key stakeholders.
You always think you know how an end user will use your software, but if you’ve never seen them use it, you’ve likely made some large assumptions. By having that firsthand experience of riding along with a stakeholder, the feedback cycle becomes tighter. Both you and your engineers get a really good sense for where the process is breaking down and where you might be able to make improvements.
Ride alongs have the wonderful side effect of building empathy with your core customer and gaining a deeper understanding of how the code your write is used day-to-day. The best part, you just need a stakeholder willing to answer questions while they go about their day-to-day.
Stakeholders can think of this as “take your engineer to work” day. Just make sure you’ve got a stakeholder with patience for questions like, “wait, why did you use that workaround,” or “oof, why is it doing that.” Both questions are a good sign that the engineer is learning some of value.
Final Thoughts
When you feel the pain first, you get a better sense of what areas are truly problematic. You’ll be closer to answering questions like:
What is the real struggle?
What is the real problem that you're trying to solve and how often does it happen?
Why did we build it this way?
If it's happening once a month, it's probably not a big deal, but if you look at the impact of the once-a-month problem and it's a million dollar issue, your view is probably very different.
By viewing the issues initially by yourself, you can get a better understanding of the context and what goes into each problem. From there, expand it out to folks that can fix it and come up with better ideas than what you'll have to remove the problem completely.
Aim to code away a majority of your problems and continue to filter the rest. There are always improvements to make. Software is never 100% done. Anybody that says that is a liar! By understanding the pain, by seeing it yourself, by having your engineers experience it, your product gets better and better and better. You just have to give them the time and visibility to do something about it.