I had an old Android TV project that I wanted to rebuild on a newer Android TV template while retaining all existing functionality. I decided to hand this task over to an LLM.

I handed the entire source code over to Gemini 3.1 Pro—the current flagship model—and simultaneously created a new directory containing the latest Android TV template. My prompt was roughly: The original codebase is outdated; I need you to recreate it on the new version while maintaining all features. Please review the code directly, evaluate it, propose a solution, and ask any questions you might have along the way.

After reviewing the code itself, it first offered a string of interpretations regarding the project—the specific details of which I can no longer remember—and finally replied with a single conclusion:

No problem.

Next, it provided a spec. That spec looked like an outline:

1. Replicate feature X
2. Replicate feature Y
3. ...

Every single item was simply “Replicate feature X,” with no further details.

Faced with this outline, my reaction was that I had no feedback to offer. This wasn’t because the outline was well-written or because I agreed with it, but because it was so featureless that I couldn’t find a single point to challenge—“Replicate feature A,” “Replicate feature B”—what exactly was I supposed to object to? So, I just let it proceed with the implementation.

When it finished, I opened the result and saw nothing but an Android TV template, like a bare-bones starter project. The UI looked completely different from the original, not to mention the lack of any detailed functionality. I was dumbfounded on the spot, and so frustrated that I didn’t even bother to investigate what it had actually managed to replicate, or if it had replicated anything at all.

The Point Is Not the Failed Porting

A failed porting attempt isn’t really worth writing about. What I want to figure out is why my own review process was completely disarmed the moment it said “No problem.”

This specific kind of failure—where a model confidently delivers an empty output—is not something I am encountering for the first time. The earliest instance was with a GPT model from the 4.x era (that’s an older version now; it is already at 5.x, though I haven’t gone back to test it). Back then, I relied on it to plan two projects. The planning phases looked beautiful, but halfway through the execution, the projects hit a wall. When I went back and asked it, “Why did you think this was feasible in the first place?”, it brushed me off with a fabricated excuse. That experience made me realize something for the first time: no matter how plausible a model’s plan looks, that plausibility can be completely decoupled from whether it can actually execute the plan.

So when Gemini 3.1 Pro failed this time, I wasn’t surprised that it couldn’t pull it off. What surprised me was that, despite having seen this exact type of failure before, my review process was still bypassed. The problem stemmed from two places.

”No Problem” Is Not Proof of Understanding

The first was the phrase “No problem.”

It read the code, offered a string of interpretations, and then said there was no problem—this entire process made it look like it understood the codebase. But “No problem” is not proof that it understood; it is merely a conclusion it offered. And the appearance of this conclusion—confident, definitive, seemingly backed by evaluation—happened to perfectly disarm my vigilance.

This was exactly the lesson that the 4.x GPT had taught me: a model’s confidence and its actual ability to deliver are two completely different things. The difference is that back then, I only learned this in hindsight, after two projects had collapsed. This time, I should have been on high alert the very moment it said “No problem.”

An Irrefutable Outline Is Not a Correct Outline

The second point of failure was that outline.

It only went as far as “Replicate feature A/B/C,” without any concrete details. I said I had “no feedback to offer”—and that statement itself was the problem. A spec that is only at the level of an outline is so featureless that it gives me nothing to grasp onto, so of course I couldn’t offer any feedback. It passed my review not because it was correct, but because it had no shape. Here, vagueness became a shield that blocked the review process.

A confident evaluation, combined with an irrefutable outline—these two factors together completely dismantled my review mechanism. I didn’t realize the output was empty until the moment I opened it.

I Removed It from This Type of Task

I didn’t go back and ask it to fix the code. The decision I made was this: remove Gemini 3.1 Pro from this type of task.

This doesn’t mean it’s useless. In my view, it is suited for tasks with explicit requirements—work where the needs are clear, the boundaries are well-defined, and it doesn’t have to evaluate “how to do this whole thing” on its own. It is perfectly capable of handling that kind of work. However, open-ended tasks like “read the code yourself, evaluate it, propose a solution, and build it” are its minefield. It is precisely this kind of task that stretches the gap between its confidence and its actual capabilities to the absolute maximum. In my workflow, this means relegating it solely to executing tasks with explicit requirements.

Conclusion

After this incident, I have an added layer of wariness toward a model saying “No problem, I can do this.” It doesn’t act as a signal that “I understand”; rather, it feels more like an unverified promise. The real trouble is that when this promise is paired with an outline that reveals no obvious flaws, it tricks me into thinking it has passed review—when in reality, the review was simply bypassed by that featureless surface.

The heuristic I have established for myself is this: when a model gives a highly confident evaluation but pairs it only with an outline-level spec, that gap itself is a red flag. The weight of its confidence should match the concreteness of its proposed solution. Being confident but hollow is actually far more dangerous than plainly stating “I am not sure”—the latter keeps me watching closely, while the former tricks me into letting go.