DEV Community

Matthew J. Moore
Matthew J. Moore

Posted on

The AI's answer was correct. Why did it still take 10 comments to fix his problem?

I've been answering Windows activation questions on Microsoft's Q&A forum lately, mostly because I deal with this stuff all day anyway. Every question there now gets an AI answer automatically. And honestly? They're good. Well written, accurate, links to the right docs.

Today a guy posted that his Windows key stopped working after he swapped in a new hard drive. The AI answer was textbook: install the same edition, sign in with your Microsoft account, run the activation troubleshooter. All correct.

It didn't fix anything.

What actually happened took about ten comments to dig out. The "key" he'd saved before swapping drives was a placeholder, not his real one, so a key checker called it invalid and he thought his license was gone. The troubleshooter showed zero PCs on his account, even though he had three others signed in, because none were ever linked to the license. Then we got one error code, then a different one, and only then did the real picture show up.

None of that was in his original question. He didn't know which details mattered, so he couldn't have written them down. The AI answered the question he asked. The humans in the thread answered the question he should have asked, by asking him things back.

That's the part I keep thinking about. AI is great at "here's the answer." It's much worse at "wait, where did that key come from?" Asking a follow-up and noticing something sounds off is most of what real troubleshooting is. Every good support person I know spends more time asking questions than giving answers.

And the thread itself is now the useful thing. The next person who swaps a drive and hits the same wall will find the whole conversation, the wrong turns included, which a single polished AI answer can't give them.

So I'm curious what you've seen:

  • Have you had an AI give you a perfectly correct answer that didn't help, because the real problem was something you didn't think to mention?
  • Do you think AI assistants will get good at asking the follow-up questions, or is that always going to be the human part?
  • If forums fill up with instant AI answers, do fewer humans bother replying, and does that kill the back-and-forth?

Genuinely want to hear from people who do support work, the AI answers and the human ones.

Top comments (9)

Collapse
 
ayushdhanani369 profile image
Ayush Dhanani •

I build Aladdin, a job agent. Your placeholder key is the case I design for. The AI answered the question he asked, and the real problem sat in the details he did not know mattered. I built Aladdin the other way round. If a form cannot be read or an answer is not clearly covered by the user's own facts, it asks the user instead of guessing. The ask is the feature, and the answer is saved with the user's own facts.

Collapse
 
krbkeys profile image
Matthew J. Moore •

"The ask is the feature" is a great way to put it. That's basically what the 10 comments were, me asking the questions the AI skipped. Curious how you decide when Aladdin should ask vs just go with its best guess? Asking too much gets annoying fast, so I'd love to know where you draw the line.

Collapse
 
ayushdhanani369 profile image
Ayush Dhanani •

Good question. The line is simple. Aladdin never guesses on a form answer. A fresh worker reads the actual page and form for each application. If the form can be read and the answer is clearly covered by the user's own facts, it fills it. If the form cannot be read, or the answer is not clearly covered, it asks the user instead of guessing. The ask is short. The answer is saved with the user's facts, so the same question does not come up again. That is what keeps the asking rare. A wrong guess on an application costs more than a short question.

Thread Thread
 
krbkeys profile image
Matthew J. Moore •

Saving the answer so it never asks twice is the smart part, that's what makes asking feel helpful instead of annoying. Same idea works for support honestly: once you know someone's setup, you shouldn't have to ask them the same thing again. "A wrong guess costs more than a short question" might end up on a sticky note on my monitor.

Collapse
 
zhang_jimmy_54a0414571400 profile image
zhang jimmy •

Your placeholder-key detail is the crux: the AI answered the question as asked, but the user didn't know the saved key was a placeholder, or that zero PCs in the troubleshooter mattered when three other machines were signed in. That gap is exactly why a correct answer can still fail. I think the fix is less about smarter answers and more about a short diagnostic checklist before answering — ShowKey output, linked-device count, and the exact error code. Do you think Microsoft's Q&A should prompt for those three facts up front, so the AI's first reply is a question rather than a textbook answer?

Collapse
 
nexusverify profile image
Nexus Verify •

The detail that tells the story is the troubleshooter showing zero PCs even though he had three others signed in — none of them were ever linked to the license, so the textbook steps couldn't work. The first check I'd do in that spot is whether the saved key is his real key or a placeholder, and whether any device at all shows as linked to his Microsoft account. If nothing is linked, no reinstall or troubleshooter run will fix it until that link is established.

Collapse
 
krbkeys profile image
Matthew J. Moore •

Exactly, that zero-PCs screen was the moment it clicked for me too. And funny enough the ending proved your point: the fix wasn't the troubleshooter at all. He ran ShowKey and found an OEM key sitting in his motherboard firmware the whole time. Entered it, activated, answer accepted. Nothing was ever linked, so the "official" path was a dead end from the start.

Collapse
 
nexusverify profile image
Nexus Verify •

Thanks for closing the loop — that ShowKey / OEM key detail is exactly the missing fact. The "official" path being a dead end from the start is the part worth remembering.

Thread Thread
 
krbkeys profile image
Matthew J. Moore •

Yeah, lesson learned for me too. Now ShowKey is one of the first things I ask people to run, before any troubleshooter. Do you have a go-to first check like that for activation issues?