← All writing

Vision evaluation · GTA VI

I Gave a Vision Model GTA VI. It Was Very Sure. I Checked the Answers.

Gemma 3 answered 36 fixed visual questions about 12 official Rockstar screenshots. It got 31 right. An immediate duplicate run returned the same answers, including one invalid enum.

On one official GTA VI screenshot, Gemma 3 returned "blue" for a field whose contract permitted only yes, no, or unclear.

The word fit the scene and was close to the expected yes. It was still invalid input. A validator cannot infer whether an undocumented value is true, a colour label, or a malformed response.

Original neon coastal illustration with four conflicting computer-vision overlays over a rainy road.
Original illustration. The experiment uses official Rockstar source media; the article does not reproduce Rockstar frames.

What I tested

I used Rockstar's publicGTA VI screenshot gallery, not YouTube, walkthroughs, leaks, or player captures. I selected 12 screenshots across character, city, water, nightlife, and rural scenes. The audit record holds the source URLs, gallery labels, expected answers, and both model runs.

A twelve-card map of the official Rockstar screenshots used for the audit. Each card has two present facts and one absent fact.
The selection spans several scene types. It is documented, not random.

Before running the model, I wrote three binary labels for every screenshot: 24 facts that are present and 12 facts that are absent. The runner used local gemma3:4b through Ollama with temperature set to zero. Every image received the same question format.

Answer each question from visible pixels only.
Return JSON exactly as:
{ "answers": { "id": "yes|no|unclear" } }

Use yes only when the frame visibly supports the statement,
no only when it visibly contradicts it, and unclear otherwise.
Question-contract diagram: every source screenshot has two labelled present facts and one labelled absent fact, and the model may respond only yes, no, or unclear.
The model did not define the task after the fact. The question sheet and permitted response values were fixed before inference.

I did not alter the prompt after seeing an answer. The first run is the reported score. The immediate duplicate run did not change any answers; it does not create a better score.

Protocol diagram: official Rockstar frames, claim sheet, frozen prompt, and item-level scoring.
The audit keeps the frames, prompt, and checks separate. Every caption still needs evidence.

31 of 36 strict answers were correct

The first run scored 31/36 under a strict rule: a response is correct only when it exactly matches the pre-labelled yes or no. The invalid value blue was wrong, even though the question asked whether the scene had purple or blue lighting. A calling program needs exact values before it can branch on them.

strict_accuracy = count(run_1_answer === label) / 36
schema_validity = count(run_1_answer in {yes, no, unclear}) / 36
repeat_agreement = count(run_1_answer === run_2_answer) / 36

# An "unclear" answer scores as wrong here:
# every question was labelled yes or no before inference.
Gemma 3 got 31 of 36 pre-labelled visual questions correct across 12 official GTA VI screenshots. It recognized present objects in 23 of 24 cases and correctly rejected absent objects in 8 of 12 cases.
The repeat reproduced all 36 outputs, including the invalid enum. Repeatability does not make the initial mistake safe to consume.
A schema validator rejects the model value blue for a yes or no question, then a restricted retry returns yes while the original 31 out of 36 score remains unchanged.
A retry can recover an operational failure. It leaves the first-run score unchanged.

Absence was weaker in this sample

On questions whose answer was yes, the model was strictly correct 23/24 times. On questions whose answer was no, it was correct 8/12 times. Three times it choseunclear for an obvious absence. Once it answered yes to a helmet that was not in the underwater scene.

Of twelve absent-object questions, Gemma 3 answered eight no correctly, gave three unclear answers, and gave one false yes answer.
Negative labels need their own acceptance criteria and error analysis.
Expected labelModel responseWhy it failed
Blue or purple lighting: yesblueSchema violation. Semantically plausible, operationally unusable.
Motorcycle rider wears a helmet: nounclearAbstained despite the rider's uncovered head.
Woman with handgun is underwater: nounclearAbstained despite rain on a window and a vehicle interior.
Underwater diver wears a helmet: noyesConfused diving equipment with a helmet.
Dog in the cash frame: nounclearAgain, uncertainty replaced a clear negative answer.

A one-item retry with an explicit enum-only instruction returned yes. The first-run score remains 31/36. A validator still needs to see the bad value and choose whether to retry, escalate, or fail closed.

How I would use this in a product

A content-moderation tool, game-search feature, or media-analysis agent should validate the output schema before it writes a fact into a record. It should also distinguish “object present” from “object absent.” The two tasks had visibly different behavior in this small run.

Object presence, readable text, clothing attributes, and scene interpretation fail in different ways. An aggregate vision score hides that distinction.

What would make this a benchmark

  • Two independent human reviewers and adjudication rules for every label.
  • More screenshots from a declared sampling plan, including hard negatives.
  • Several models, repeated runs, and model-version records.
  • Per-class precision, recall, and an abstention policy fixed before the run.

This run used one quantized local model and twelve screenshots selected and labelled by one reviewer. There was no independent adjudication. The score does not compare models, predict game-play performance, or measure a production vision system. It records one fixed local run and the validation controls a production integration would need.

Scope diagram separating what the twelve-screenshot Gemma 3 run supports from claims it cannot establish.
This is a reproducible claim audit. It is not a general vision-model ranking.

PK / RESEARCH NOTES

Research notes, when they're worth sending.

AI/ML systems, experiments, and the results that did not survive scrutiny.