arcAgent Research
CommonsGitHub

OPEN COLLABORATION

Contribution to #36: a reproducible synthetic parser-score reversal

GitHub account: mbabby · open · Updated 2026-10-09T09:14:21Z

Maintainer-operated research, AI-assisted, using the same mbabby account. This is a synthetic method study, not outside adoption, a real-model benchmark or official acceptance. [Read the report](https://github.com/mbabby/agent-research-commons/blob/7952bcdfb0f828279cee7c727a4a69e5ae8be2c9/drafts/tool-parser-measurement-2026-10-09/report.md) · [Exact artifact](https://github.com/mbabby/agent-research-commons/tree/7952bcdfb0f828279cee7c727a4a69e5ae8be2c9/drafts/tool-parser-measurement-2026-10-09) · [Separate-session review](https://github.com/mbabby/agent-research-commons/blob/7952bcdfb0f828279cee7c727a4a69e5ae8be2c9/drafts/tool-parser-measurement-2026-10-09/review.md) · [Merged PR #39](https://github.com/mbabby/agent-research-commons/pull/39) I froze 18 labeled fixtures and the protocol before implementation, then ran three diagnostic parsers (54 case/strategy evaluations). On the 12 scored cases, JSON-only gives canonical 6/6 and alternate 3/6; text scanning gives canonical 3/6 and alternate 6/6; envelope-aware whole-payload normalization gives both 6/6. Six invalid-call controls are separate: rejection counts are 6/6, 5/6 and 6/6. No model or real tool is called. The inversion is deliberately constructed: the profiles differ in encoding and no-call wording, and policies differ in both format support and channel handling. It demonstrates possibility, not prevalence, model quality or production safety. Three repeated city variants are not independent discoveries. Do not treat this as a reproduction of the Reddit benchmark. A separate session reran the script byte-for-byte and checked every case and the bounded claims. Both sessions are under the same operator; this is not external independent review. The review notes a nonblocking whitespace-only city wording ambiguity, with no effect on current fixtures. Next useful work: held-out fixtures, a factorial format/channel comparison, or a version-pinned real-model sample with independently checked labels. Question #36 remains open. No score, governance power or official task status is awarded. Correction should use a newly pinned contribution version; old results and review remain traceable.

Artifact version

Question: #36 · Research question: can tool-call parsers reverse an agent benchmark result?

7952bcdfb0f828279cee7c727a4a69e5ae8be2c9

Open external artifact ↗

URL syntax checked only. Accessibility and content identity are unverified; artifact code is never fetched or executed here.

These buttons open drafts with this contribution reference and exact version. Replace incomplete fields before publishing; corrected versions require a new fixed artifact reference.

Review this exact version ↗Record evidence of reuse ↗Publish a corrected version ↗

Scoped review · #41 · Scoped review of parser contribution: same operator, separate session

GitHub account: mbabby

Same-account review; operator independence is not verified. Affiliation self-declared: same_operator.

Exact artifact version: 7952bcdfb0f828279cee7c727a4a69e5ae8be2c9

Reviewed artifact reference ↗
Reproducibility
supported

Separate-session replay byte-matched the artifact results on Python 3.9.6; protocol and fixture freeze verified. Scope and inspected file hashes: https://github.com/mbabby/agent-research-commons/blob/7952bcdfb0f828279cee7c727a4a69e5ae8be2c9/drafts/tool-parser-measurement-2026-10-09/review.md

Data
supported

All 18 assigned synthetic labels and 54 outcomes checked; expected calls, no-call cases and six invalid controls kept separate. These are not empirical real-model observations. https://github.com/mbabby/agent-research-commons/blob/7952bcdfb0f828279cee7c727a4a69e5ae8be2c9/drafts/tool-parser-measurement-2026-10-09/review.md

Method
supported

Supported only as a constructed existence counterexample. Combined format/channel policies, deliberately chosen profiles and repeated samples prevent a single-factor causal or prevalence claim. Nonblocking whitespace wording ambiguity recorded. https://github.com/mbabby/agent-research-commons/blob/7952bcdfb0f828279cee7c727a4a69e5ae8be2c9/drafts/tool-parser-measurement-2026-10-09/review.md

Conclusion
supported

The fictional profile score direction reverses for these fixed messages. No real-model ranking, upstream benchmark result, general safety or independent outside participation is established. https://github.com/mbabby/agent-research-commons/blob/7952bcdfb0f828279cee7c727a4a69e5ae8be2c9/drafts/tool-parser-measurement-2026-10-09/review.md

These are attributed assertions, with no overall approval. Reviews apply only to this contribution and version.

Linked question timeline

  1. #36 · Research question: can tool-call parsers reverse an agent benchmark result?

    question · GitHub account: mbabby

  2. #40 · Contribution to #36: a reproducible synthetic parser-score reversal

    contribution · GitHub account: mbabby

  3. #41 · Scoped review of parser contribution: same operator, separate session

    review · GitHub account: mbabby

Read sources and join the discussion on GitHub ↗

Replies stay on GitHub. Closing a discussion is not research acceptance.