Pull the Plug Test: We Turned Off the AI in 15 'AI-Powered' Products to See What Actually Changed
Photo: Sky99, CC BY-SA 3.0, via Wikimedia Commons
Every product page these days has the word 'AI' splashed across it like a sticker. But here's a question almost nobody asks: if you ripped the AI out, what would be left?
We decided to find out. Over three weeks, our team ran what we're calling the Pull the Plug Test on 15 popular tools that market themselves as AI-first. We couldn't literally shut off anyone's servers (we asked — the answer was no, followed by a very confused legal email), but we got creative: downgrading to the non-AI tier, disabling AI features in settings, using each product's offline or fallback mode, and in a few cases, blocking the API endpoints the AI relies on at the network level.
Then we asked one simple question after each test: did anything meaningful break?
How We Scored It
Each tool got a score from 0 to 10 based on how much its core value collapsed without AI:
- 9–10: Product is unusable or pointless without AI. The label is earned.
- 6–8: AI does the heavy lifting, but a degraded fallback exists.
- 3–5: AI is a convenience layer. The product works fine without it.
- 0–2: The 'AI' is marketing paint on an ordinary feature.
The Tools That Earned the Label
Descript (9/10). Strip out the AI transcription, voice cloning, and filler-word removal, and you're left with a video editor that's actively worse than free alternatives. The AI isn't a feature — it's the whole reason the product exists.
Grammarly (8/10). Turn off the suggestions engine and you have a spell-checker from 2009. The tone rewrites and clarity suggestions are where the money is, and they're genuinely model-driven.
Otter.ai (9/10). Without transcription AI, Otter is a voice recorder. Full stop. There is no fallback mode because there's no product underneath.
Midjourney (10/10). Obviously. We included it as a control, and honestly, it was nice to test something that doesn't pretend to be anything other than what it is.
The Middle of the Pack
Notion AI (6/10). Notion is a genuinely great workspace with or without its AI layer. When we disabled the AI features, our databases, docs, and wikis carried on just fine. The AI is a nice accelerant — search summarization and drafting — but the product underneath is solid. That's not an insult. It's just not an 'AI company.'
Canva (5/10). Magic Write, background remover, the design suggestions — all helpful, all optional. Canva was winning at design templates long before the AI boom, and disabling the magic features changed our output time by minutes, not hours.
Zoom AI Companion (4/10). Meeting summaries are handy, but we've all survived meetings without them since roughly the invention of the meeting. The recording, screen sharing, and video core are untouched by AI.
Calendly (3/10). Its new AI scheduling suggestions barely registered in our tests. A calendar link is a calendar link.
The AI-Washing Hall of Shame
A popular email assistant (2/10). Its 'AI-powered prioritization' turned out to be a rules filter with a chatbot stapled on. When we blocked the model endpoint, nothing changed except a small 'AI' badge disappeared from settings. That badge was doing more work than the model.
A note-taking app we won't name (1/10). The 'AI search' was keyword matching with slightly better synonyms. We confirmed this by feeding it a typo-riddled query that any modern LLM would ace. It whiffed completely.
A 'smart' CRM (2/10). Its AI lead scoring produced the same rankings as sorting by 'last contacted date.' We checked. Twice.
Why This Matters for Your Wallet
Here's the practical takeaway: AI-washing isn't just annoying, it's expensive. Several of the tools in our bottom tier charge a 30–60% premium for an 'AI tier' that adds little beyond the badge. If a tool scored 3 or lower in our test, you're almost always better off on the base plan — or with a different product entirely.
Our rule of thumb after this experiment: if the vendor can't tell you what happens when the model fails, the model probably isn't doing much. The honest products — Descript, Otter, Grammarly — know exactly what breaks, because everything breaks. The pretenders shrug.
One Caveat
Some vendors use AI invisibly and well — fraud detection, spam filtering, autocomplete infrastructure. We're not saying those don't count. We're saying they shouldn't be the headline. If AI is the plumbing rather than the product, market it that way.
The Full Verdict, Ranked
- Midjourney — 10
- Descript — 9
- Otter.ai — 9
- Grammarly — 8
- Notion AI — 6
- Canva — 5
- Zoom AI Companion — 4
- Calendly — 3 9–15. The rest — 1 to 2, with three products we've referred to our legal team's spelling of 'misleading.'
The next time a landing page screams 'AI-powered,' ask the only question that matters: pull the plug, and what's left? If the answer is 'nothing,' congrats — you found a real AI product. If the answer is 'the same thing,' you found a sticker.
