Skip to content

Comments

fix: fallback to json_object when json_schema response_format is unsupported#1345

Open
mjnovice wants to merge 3 commits intomainfrom
fix/eval-llm-json-schema-fallback
Open

fix: fallback to json_object when json_schema response_format is unsupported#1345
mjnovice wants to merge 3 commits intomainfrom
fix/eval-llm-json-schema-fallback

Conversation

@mjnovice
Copy link
Contributor

@mjnovice mjnovice commented Feb 19, 2026

Summary

  • Adds a try/except fallback in LLMMocker eval LLM calls so that when a model rejects the json_schema response format, it retries with json_object format instead
  • Prevents eval failures caused by strict schema validation differences across models

Context

The json_schema response format requires schemas to meet strict validation rules (all properties required, additionalProperties: false everywhere, no unsupported keywords, etc.). While _cleanup_schema handles some of these, certain edge cases can still cause rejections.

In practice, gpt-4o-2024-11-20 enforces strict schema validation and rejects schemas that don't fully comply, while gpt-4.1-2025-04-14 is more lenient and accepts them. This fallback ensures evals work across both models by catching the json_schema rejection and retrying with the simpler json_object format.

Behavior gpt-4o-2024-11-20 gpt-4.1-2025-04-14
json_schema strict validation Strict — rejects non-compliant schemas Lenient — accepts them
json_object fallback needed Yes No

Test plan

  • Run evals with gpt-4.1-2025-04-14 — verify behavior is unchanged (json_schema succeeds)
  • Run evals with gpt-4o-2024-11-20 — verify it falls back to json_object and completes successfully

🤖 Generated with Claude Code

…pported

Some models don't support json_schema response_format in eval LLM calls.
This adds a try/except that catches the failure and retries with
json_object format as a fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Feb 19, 2026
… only

The broad `except Exception` was catching parse errors (e.g. from empty
responses), triggering an unintended retry that could hang. Restrict the
catch to EnrichedException and HTTPStatusError so only actual HTTP
failures (400/422 for unsupported format) trigger the json_object fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… other errors

Only fall back to json_object when the server returns 400 or 422,
indicating the json_schema format is unsupported. All other HTTP errors
(auth, rate limit, server errors) are now re-raised instead of masked.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mjnovice mjnovice requested a review from Chibionos February 20, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants