Confidence to Zag

Flat.

How the hell is this flat?

When you're not Google or Meta or some other behemoth you don't have millions of daily users. Reaching real statistical significance can take weeks on any A/B proposed change. When speed matters, changes can't be just quant. It's the direction the numbers are leaning, plus what you already know about the problem and the people using it. Yet you're still secretly hoping for that lift before the test reaches significance. In my own experience, I start to notice when a series of tests all just end up being flat. Does this sound at all familiar?

A single flat result, that could just be noise. Run enough tests and some of them come back flat by chance alone. But test a handful of genuinely different options and watch all of them come back flat together, little movement anywhere across the set, that's most likely not chance anymore. But it doesn't tell you why and that can be exasperating. This leaves you with a choice: write it off as noise and continue moving in the same direction, or pause and take it seriously. Most people don't even notice they're making a choice. When immovable numbers start holding back assumed momentum, they pick momentum and hope their expertise will get them through. I'm not here to tell you not to trust your gut, but I am going to tell you to give the data a second look. Sure, it's not telling you what you expected. Maybe it's telling you you're not solving the most important problem. When the project is heading one direction and you start to see a pattern of results going somewhere else, you need the confidence to go against the grain and figure out why you're seeing what you didn't expect to see. Call it recalibration, not scope creep.

I call this choice Zag. You're moving one way, you cut sharply, you straighten back out, you keep going. It's shaped like a Z. Zag. Get it? Yeah, yeah you get it, I'm mansplaining my own metaphor. Awkward. Onward.

I've been talking numbers because numbers are easy to point at. But the Zag doesn't care whether the surprise came from a p-value or a conversation, a focus group, or an observation. Some of my clearest ones came from people showing me they used my product in a way I didn't expect to see, no dashboard anywhere in sight. Writing it off would have been the easy move, but it's confirmation bias wearing a disguise. Taking it seriously was harder.

Here's the part that took me a while to say out loud: if something you built isn't working, it's rarely a build problem, it's a listening problem. You didn't build what users needed, or don't trust what you built enough to use it. Same rule I use for users applies to your own data. If your direction isn't getting results, chances are you're not listening to the clues sitting right in front of you.

Finding that signal through the noise isn't always easy. Only a third of features that get to production actually move the needle when they ship. That's the true benchmark of doing this well, not a red flag. You can't be expected to get it right every time, but you do need to be able to learn from your mistakes, and take risks.

The process is the same no matter what stage you're confused in. Be open, don't immediately dismiss anything out of hand. Gather the assumptions you're most sure of, then ask questions, tying every one of them back to the user. Not business requirements, not what success is assumed to look like. The user. What is the user trying to do here, right now, on their journey? Do we understand the exact problem the user is trying to solve? Is the problem sitting in front of us the real one, or a symptom of something bigger underneath it? Is the user doing something here that was never intended by what we originally built? Separate what you know to be true from what you hope to be accurate.

Once you have a grasp on the problem, plenty can still be wrong under the hood: the premise, the implementation, the experiment itself, the data. This sounds like a lot. It isn't. You're refamiliarizing yourself with the problem you came here to solve, not staring at repeated test results that don't agree with your assumptions. Give yourself a couple hours, max. This isn't meant to be a days-long exercise. You'll end up with a handful of plausible, often contradicting explanations. Look for the minimum amount of data you need to test each new theory. Fight against analysis paralysis; you won't have enough time to rule them all out properly. Make the best call you can with the information you have on hand, and you move. Experiment fast, make a decision, and continue on to the next. Your goal is to rule possibilities out.

You'll get to your epiphany, your ah-ha, your smack-yourself-on-the-forehead moment, when you finally see the thing that was eluding you the whole time. Maybe the problem was wrong. Maybe the assumptions were incorrect. Maybe the user's steps were out of order, or their goals never aligned with the UI in front of them. Whatever it was, you'll see it: you were optimizing the wrong thing, and something else just became your biggest concern. Eureka, you're in the middle of a Zag.

You're not done, though. Sorry. Nope. Here comes the hard part: you have to persuade those around you the new direction is the right one, and that's where leadership comes in, separating people who ship good work from people who ship whatever they started with. You're not presenting data at this point: you're the bearer of bad news, the monkey wrench, the one saying it's important to pivot. People disagree on plenty, but one thing holds pretty steady across demographics, gender, and age: they hate change. While everyone else is zigging, you're the one telling them to za— you get the symbolism.

Now we're into psychology, and I can't help you with the change management piece. Smarter people than I have written tomes on the subject, but I can tell you this: if you're emotionally intelligent and open enough to see the importance of the pivot, you already have the observation skills to understand the motivations of the people you need to win over next.

In my experience, the occupants of that room fall into three categories. Sometimes it's a partner you're building something with side by side, someone who needs to feel the shift with you, not just hear about it secondhand. Sometimes it's a manager, someone a level removed who needs a different kind of case made: less "trust me," more "these are the revenue targets I need to hit to make it through Q2." And sometimes, maybe the hardest version, that room is just you. You're the one most attached to the idea you had first. Confirmation bias doesn't feel like bias from the inside. It feels like being right.

Working with colleagues is the same as working with users: listen to their needs, acknowledge their objections, and make compromises that stay true to what you found. You're confident in your findings; stay the course. What usually works for me at this point is returning to the user's goals and tying the new findings to what success should look like. When you're the one putting up the biggest objection, take a minute to reflect on why. This isn't about ego. It's about doing what's best for the product.

Over my career, having the confidence to Zag has let me reprioritize multi-year roadmaps, renew million-dollar contracts, and open new sales pipelines. Were these easy to do? No. Were they the right thing for the product? No regrets, I'd do it again.

I believe in this enough that I ask every designer who interviews with me about their own moment they pivoted after data surprised them. I don't call it a Zag going in, obviously, they've never read this. I just ask what surprised them, and what they learned. If they can't name anything specific, that's a flag. It's not a flex to claim they've never been wrong. I read that as they've only ever gone looking for proof they were right.

So what about you, stuck in a flat plateau? Open to a little Zagging? Yuck. That reads like a euphemism. Zag doesn't work as a verb apparently. Ahem.

The first question, however, remains.