feat(evals): add deterministic VQA dataset generation - #3488
feat(evals): add deterministic VQA dataset generation#3488ruthwikdasyam wants to merge 8 commits into
Conversation
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #3488 +/- ##
==========================================
+ Coverage 74.05% 76.14% +2.08%
==========================================
Files 1283 1234 -49
Lines 124704 119584 -5120
Branches 11141 10734 -407
==========================================
- Hits 92349 91054 -1295
+ Misses 29493 25431 -4062
- Partials 2862 3099 +237
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 87 files with indirect coverage changes 🚀 New features to boost your workflow:
|
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Greptile SummaryAdds standalone VQA dataset generation and evaluation commands with constrained question families, detector-derived labels, frame assets, and audit metadata. It also registers the CLI commands and corrects OpenAI vision data URLs to identify JPEG-encoded images. The exercised lifecycle failure hypotheses were disproved: generation successfully replaced an existing empty output directory; generated public cases and private labels loaded and evaluated through the shared runner with a score of 1.0; and an injected failure on a later frame did not publish a partial dataset. No actionable defects were found. The change is safe to merge. Confidence Score: 5/5The generated VQA artifact lifecycle and evaluation handoff behaved correctly in the exercised scenarios. A deterministic harness generated a dataset into an existing empty directory, loaded its cases and labels into the shared evaluator, obtained a successful result, and confirmed that a later-frame failure left no partial output. Files Needing Attention: No files need follow-up based on the exercised VQA generation, artifact loading, and evaluation paths.
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "fix(evals): align VQA image encoding" | Re-trigger Greptile |
github-actions
Bot
added
the
ready-to-merge
| { | ||
| "type": "image_url", | ||
| "image_url": {"url": f"data:image/png;base64,{img_base64}"}, | ||
| "image_url": {"url": f"data:image/jpeg;base64,{img_base64}"}, |
There was a problem hiding this comment.
to_base64 actually encodes as JPEG so it looks like using image/png was a previous bug.
| "Do not duplicate a family/object pair. Do not answer questions or add fields. " | ||
| f"Available families: {json.dumps(family_shapes)}" | ||
| ) | ||
| payload: object = self._model.query_json(image, prompt) |
There was a problem hiding this comment.
| payload: object = self._model.query_json(image, prompt) | |
| payload = self._model.query_json(image, prompt) |
since you enforce list type just below
|
|
||
| model_config = ConfigDict(extra="forbid", frozen=True) | ||
|
|
||
| family: Literal["presence", "horizontal_direction", "object_count"] |
There was a problem hiding this comment.
you might want to check this: https://pydantic.dev/docs/validation/latest/concepts/unions/#discriminated-unions
| f"{self.inputs}\nChoices: {json.dumps(self.choices)}\nAnswer with exactly one choice." | ||
| ) | ||
| outputs = rig.ask(context, prompt) | ||
| answer = _parse_choice(outputs, self.choices) |
There was a problem hiding this comment.
maybe we should always ask llm to provide structured output (like a json) and validate against them uniformly to avoid these custom cleanup/parsing logic
|
|
||
| def evaluate(self, rig: EvalRig) -> EvalResult: | ||
| image = Image.from_file(self.image_path) | ||
| context = [] if rig.blind else cast("list[dict[str, Any]]", image.agent_encode()) |
There was a problem hiding this comment.
what's the point of doing a vqa with the blind option? basically no information provided?
| return suite | ||
|
|
||
|
|
||
| def _read_jsonl(path: Path) -> list[Any]: |
There was a problem hiding this comment.
let's use this which provides lazy jsonl loading. jsonl can be very huge and drain too much memory if you load directly. a lazy iterator would be much safer: https://jsonlines.readthedocs.io/en/latest/
| case_by_id = _unique_by_id(cases, "case") | ||
| label_by_id = _unique_by_id(labels, "label") | ||
| if case_by_id.keys() != label_by_id.keys(): | ||
| missing_labels = sorted(case_by_id.keys() - label_by_id.keys()) | ||
| missing_cases = sorted(label_by_id.keys() - case_by_id.keys()) | ||
| raise ValueError( | ||
| f"VQA case/label IDs do not match: missing_labels={missing_labels}, " | ||
| f"missing_cases={missing_cases}" | ||
| ) |
There was a problem hiding this comment.
this whole checking seems to imply that there's a 1-1 correspondance between case and label. in that case why not just use 1 jsonl instead of 2?
| from dimos.perception.detection.type.detection2d.imageDetections2D import ImageDetections2D | ||
|
|
||
|
|
||
| class MoondreamObjectDetector: |
There was a problem hiding this comment.
can we use existing mooondrealVLModel any need to wrap it?
|
|
||
| import typer | ||
|
|
||
| app = typer.Typer(help="Generate and evaluate standalone visual question-answering datasets.") |
There was a problem hiding this comment.
all cli change should be in dimos/cli so we can track everything
| def _write_frame(output: Path, frame: _GeneratedFrame) -> None: | ||
| (output / "assets").mkdir(parents=True, exist_ok=True) | ||
| frame_audit = output / "audit" / f"frame-{frame.index:06d}" | ||
| frame_audit.mkdir(parents=True, exist_ok=True) | ||
|
|
There was a problem hiding this comment.
No hard coded path generation / making directories. this will work for a git cloned dimos but not for a library installed dimos. Needs to be done properly and save VQA sets to .local/ the same place mem2 does.
| "type": "image_url", | ||
| "image_url": { | ||
| "url": f"data:image/png;base64,{self._prepare_image(img)[0].to_base64()}" | ||
| "url": f"data:image/jpeg;base64,{self._prepare_image(img)[0].to_base64()}" |
| title: "Visual Question Answering" | ||
| --- | ||
|
|
||
| # Visual Question Answering |
There was a problem hiding this comment.
Doesn't this add the title twice?
| # Visual Question Answering | ||
|
|
||
| The VQA tools generate deterministic questions from recorded images and evaluate them through the | ||
| shared DimOS evaluation runner. |
There was a problem hiding this comment.
All docs use dimOS now.
There was a problem hiding this comment.
maybe we can add a CI check for this lol
|
|
||
| ## Architecture | ||
|
|
||
| - `author.py` proposes constrained family inputs from an image. | ||
| - `families.py` owns question text, choices, and deterministic answer rules. | ||
| - `primitives/moondream.py` supplies private object detections. | ||
| - `generate.py` loads frames, reuses models, and writes datasets atomically. | ||
| - `suite.py` validates generated artifacts and creates shared evaluation cases. | ||
|
|
||
| The initial implementation intentionally defers negative-presence policy, retries, and resume | ||
| behavior. |
There was a problem hiding this comment.
| ## Architecture | |
| - `author.py` proposes constrained family inputs from an image. | |
| - `families.py` owns question text, choices, and deterministic answer rules. | |
| - `primitives/moondream.py` supplies private object detections. | |
| - `generate.py` loads frames, reuses models, and writes datasets atomically. | |
| - `suite.py` validates generated artifacts and creates shared evaluation cases. | |
| The initial implementation intentionally defers negative-presence policy, retries, and resume | |
| behavior. |
4 participants
Contribution path
Closes DIM-1418
Problem
DimOS needs reproducible visual-question datasets generated from recorded camera frames.
Solution
Add
dimos evals vqa generateanddimos evals vqa runworkflows for deterministic multiple-choice VQA datasets. Questions are constrained by family,answers come from private Moondream evidence, and
outputs include lossless PNG assets plus audit metadata.
Added 3 deterministic families - presence, horizontal_detection, object_count -> which are pre-built using primitive methods (moondream here) - so can just call the method to get the solution - making it deterministic.
How to Test
dimos evals vqa generate go2_short.db --image-index 100dimos evals vqa run ~/.local/state/dimos/datasets/vqa/go2_short-frames --model gpt-4o-miniuv run --no-sync pytest dimos/evals/vqaAI assistance
OpenCode with GPT-5.6 Sol assisted with implementation, tests, documentation, and review.
Checklist