On one official GTA VI screenshot, Gemma 3 returned "blue" for a field whose contract permitted only yes, no, or unclear.
The word fit the scene and was close to the expected yes. It was still invalid input. A validator cannot infer whether an undocumented value is true, a colour label, or a malformed response.

What I tested
I used Rockstar's publicGTA VI screenshot gallery, not YouTube, walkthroughs, leaks, or player captures. I selected 12 screenshots across character, city, water, nightlife, and rural scenes. The audit record holds the source URLs, gallery labels, expected answers, and both model runs.
Before running the model, I wrote three binary labels for every screenshot: 24 facts that are present and 12 facts that are absent. The runner used local gemma3:4b through Ollama with temperature set to zero. Every image received the same question format.
Answer each question from visible pixels only.
Return JSON exactly as:
{ "answers": { "id": "yes|no|unclear" } }
Use yes only when the frame visibly supports the statement,
no only when it visibly contradicts it, and unclear otherwise.I did not alter the prompt after seeing an answer. The first run is the reported score. The immediate duplicate run did not change any answers; it does not create a better score.
31 of 36 strict answers were correct
The first run scored 31/36 under a strict rule: a response is correct only when it exactly matches the pre-labelled yes or no. The invalid value blue was wrong, even though the question asked whether the scene had purple or blue lighting. A calling program needs exact values before it can branch on them.
strict_accuracy = count(run_1_answer === label) / 36
schema_validity = count(run_1_answer in {yes, no, unclear}) / 36
repeat_agreement = count(run_1_answer === run_2_answer) / 36
# An "unclear" answer scores as wrong here:
# every question was labelled yes or no before inference.Absence was weaker in this sample
On questions whose answer was yes, the model was strictly correct 23/24 times. On questions whose answer was no, it was correct 8/12 times. Three times it choseunclear for an obvious absence. Once it answered yes to a helmet that was not in the underwater scene.
| Expected label | Model response | Why it failed |
|---|---|---|
Blue or purple lighting: yes | blue | Schema violation. Semantically plausible, operationally unusable. |
Motorcycle rider wears a helmet: no | unclear | Abstained despite the rider's uncovered head. |
Woman with handgun is underwater: no | unclear | Abstained despite rain on a window and a vehicle interior. |
Underwater diver wears a helmet: no | yes | Confused diving equipment with a helmet. |
Dog in the cash frame: no | unclear | Again, uncertainty replaced a clear negative answer. |
A one-item retry with an explicit enum-only instruction returned yes. The first-run score remains 31/36. A validator still needs to see the bad value and choose whether to retry, escalate, or fail closed.
How I would use this in a product
A content-moderation tool, game-search feature, or media-analysis agent should validate the output schema before it writes a fact into a record. It should also distinguish “object present” from “object absent.” The two tasks had visibly different behavior in this small run.
Object presence, readable text, clothing attributes, and scene interpretation fail in different ways. An aggregate vision score hides that distinction.
What would make this a benchmark
- Two independent human reviewers and adjudication rules for every label.
- More screenshots from a declared sampling plan, including hard negatives.
- Several models, repeated runs, and model-version records.
- Per-class precision, recall, and an abstention policy fixed before the run.
This run used one quantized local model and twelve screenshots selected and labelled by one reviewer. There was no independent adjudication. The score does not compare models, predict game-play performance, or measure a production vision system. It records one fixed local run and the validation controls a production integration would need.