The Incognito Test: Why Most AI Work Reveals Nothing About the Person
PortSwigger is about as close to a genuine AI-first company as you'll find outside a lab. When their CEO Daf declared AI a "tidal wave" in 2023, the company didn't issue a memo and move on. They built a dedicated internal squad to go deep on the technology, evangelized it at every all-hands, and pushed access to the entire company with no usage caps. By the time most organizations were still debating where AI fit in their workflow, PortSwigger's kitchen team was using it to rework supplier relationships and reformulate recipes for better afternoon energy. Not the engineers. The kitchen team.
So when Alex Craig, their Chief of Staff, talks about what comes after adoption, it's worth paying attention. Because his company has already seen that movie.
|
❝
If you could start a completely incognito context and produce the output you've produced, what's the point?
Alex Craig, Chief of Staff, PortSwigger
|
The next question, it turns out, is harder than getting everyone on the tools. It's about what happens when output volume explodes but judgment doesn't scale with it. Alex described a pattern I've heard echoed across the CoS community: employees generating AI content they haven't fully read, then forwarding it upward for leadership to sort through and decide on. The execution gets faster. The accountability for it doesn't move.
"The rule of thumb I always use is: if I could have used Claude or ChatGPT to get this same output within five minutes, why have I bothered delegating it to you?"
The question lands on management and leadership, not just the individual contributor who drafted it. And Chiefs of Staff and their Principals feel this more acutely than most. Part of what makes the role valuable is serving as a gatekeeper on quality: making sure that whatever lands on a leader's desk is structured, specific, and actually decision-ready. AI has made it faster to go from zero to a first draft. It hasn't made it easier to get from a first draft to a real recommendation.
"If you could start a completely incognito context and produce the output you've produced, what's the point? Where are you bringing your insight? Where are you bringing your preferences?"
That's the incognito test, and it's a useful frame for the next phase of AI adoption at any company. Getting employees to use the tools was phase one. Getting them to inject their own judgment, context, and stakes into the output, and take it multiple steps further toward an actual business result, is the harder and more important problem. PortSwigger is already living in that phase. Most of us are just catching up to the question.
👉 Listen to the episode
Alex also wrote up the full technical breakdown of the system he built, and fittingly, the system drafted the post. He read it, gave feedback, and decided what shipped.
👉 I'm a Chief of Staff. So I Built Myself One.
| 
