A generated request can sound reasonable and still be impossible to execute. Imagine a sandbox task that asks an agent to connect a customer record to an inventory reservation, while the available tools expose no shared identifier. The sentence is clear. The join is not available.
If you put that example into a training or evaluation dataset, you have created a problem before the model takes its first action: is failure evidence of a weak agent, or an invalid task?
For synthetic tool-use data, start with a workflow that can actually run, preserve the evidence, and derive a request that the workflow answers. Then evaluate separately authored requests as well. The first approach establishes solvable examples; the second tests what happens outside their construction rules.
ToolGrad builds the answer before the request
Google Research's ToolGrad reverses the query-first approach to generating tool-use datasets. Instead of inventing an instruction and searching for a solution, it constructs successful tool-use chains and annotates corresponding requests.
The mechanism is iterative, not simply “finish a long script, then write a prompt.” Google describes four modules:
- API Proposer narrows a sampled set of APIs to candidates that might extend the current workflow.
- API Executors test those candidates in parallel and produce execution reports.
- API Selector reads the reports, selects the best-performing call, and appends it to the workflow.
- LLM Updater revises the synthetic request and response to match the extended workflow.
Repeat that cycle and the resulting example contains a request, a verified API workflow, and a final response. Here, “textual gradients” means descriptive feedback guiding the next update. It does not mean numerical gradients or conventional backpropagation through API calls. These module descriptions and the feedback mechanism come from Google's explanation of ToolGrad.
The useful shift is where you demand evidence. A plausible instruction is no longer your only starting asset. There is a tool interaction behind it that can be inspected.
That still leaves work for the dataset owner. Successful calls do not automatically establish that a derived request accurately describes them, that the response answers the request, or that replay will work against changed state.
Keep an example record, not just a prompt and answer
Berryhill Dev's recommendation is to preserve the construction evidence alongside the generated language. A prompt-answer pair is convenient for training. On its own, it is a poor debugging record.
Consider a read-only stock-availability workflow. An approved sandbox exposes inventory and reservation lookups keyed by the same SKU and warehouse. A local deterministic calculation subtracts reserved units from on-hand units. This is deliberately narrower than placing an order: looking up availability is not authority to reserve stock.
The following is an illustrative Berryhill record, not a ToolGrad output, an executed experiment, or a claim about Berryhill production results. Tool names, versions, fixtures, and outputs are fictional. The execution status stays unverified until a real runner executes the calls and checks the assertions.
example_id: availability-illustration-001
execution_status: illustrative_not_executed
tool_catalog_version: sandbox-catalog-v1
sandbox_fixture:
id: warehouse-fixture-a-v1
state: frozen_read_only
sku: DEMO-17
warehouse: demo-north
on_hand: 12
reserved: 5
calls:
- tool: inventory.lookup
version: "1.0"
arguments: {sku: DEMO-17, warehouse: demo-north}
sanitized_output: {on_hand: 12, unit: each}
- tool: reservations.lookup
version: "1.0"
arguments: {sku: DEMO-17, warehouse: demo-north}
sanitized_output: {reserved: 5, unit: each}
- tool: local.subtract
version: "1.0"
arguments: {left: 12, right: 5}
sanitized_output: {available: 7}
assertions:
- both lookups resolve the requested SKU and warehouse
- quantities use the same unit and frozen fixture state
- available equals on_hand minus reserved
- no reservation or other write occurred
derived_request: >-
How many unreserved units of DEMO-17 are available
at demo-north in this inventory snapshot?
expected_response: >-
The snapshot shows 7 unreserved units:
12 on hand minus 5 reserved. No stock was reserved.
provenance:
origin: fictional illustration authored by Berryhill Dev
contains_real_customer_data: false
execution_evidence: absent_in_this_illustration
workflow_family: stock-availability-read-only
split: regression_holdout
split_rule: keep related fixtures and workflow variants together
In a real implementation, run the calls only in the approved environment, capture the actual sanitized outputs, and attach assertion results before admitting the record as verified. Preserve a reproducible fixture reference and tool versions. Do not substitute a model's “looks correct” verdict for an assertion the runner can evaluate.
The request must stay inside the trace's evidence. This example answers a snapshot question. It does not establish current sellable stock, account for an unmodeled safety-stock rule, or promise that an order can be fulfilled. If those are requirements, extend the tools and checks rather than embellishing the answer.
Sanitization belongs before dataset admission. Remove credentials, personal data, and irrelevant identifiers; use controlled replacements that preserve the relationships the workflow depends on. A trace is useful evidence, not an exemption from data handling rules.
Split assignment also needs an explicit rule. If near-identical workflows and shared fixtures appear in both training and evaluation, a clean-looking score can be hard to interpret. Group related examples before assigning splits. For a regression holdout, keep the family out of post-training data; if you instead allocate it to training, evaluate against independently held-out families and cases.
A generation pass rate is not a live-agent success rate
Google reports a 99.8 percent pass rate in its ToolGrad data-generation comparison. That number concerns answer-first dataset generation against the query-first baseline. It is not the probability that an agent will successfully handle an arbitrary customer request.
The same article reports improved Berkeley Function Calling Leaderboard results after fine-tuning tested Gemma-3 models on ToolGrad-500. BFCL uses a different tool set from the generation database, making this relevant evidence about performance beyond the training tools. It remains evidence from a particular benchmark and tested models—not a general certificate of model superiority or production reliability.
Keep three questions separate:
- Can the generator produce valid workflow examples?
- Does training on those examples improve a model on the selected evaluation?
- Does the deployed agent behave acceptably under the conditions your users encounter?
A result can support one without settling all three. As with benchmarks that must name the serving route, the conditions belong next to the result. Record the tool catalog, fixture assumptions, validation rules, model configuration, and evaluation population before borrowing a headline number for an operating decision.
Keep a second lane for requests you did not construct
Execution-first construction gives you a way to select successful workflows. That selection is also why it should not define your entire evaluation population.
Berryhill Dev recommends a separate challenge lane. This is our evaluation recommendation, not a measured ToolGrad finding. Author cases independently of the successful traces, or use appropriately consented and sanitized user-originated requests. Include situations where the right behavior is not task completion:
| Challenge | Expected behavior to test |
|---|---|
| The request omits the warehouse, and several match | Ask for the missing identifier instead of guessing |
| The requested quantity exceeds available stock | Explain the shortfall without inventing inventory |
| The user asks to reserve stock through read-only tools | State the capability boundary; do not claim a reservation |
| Requirements contradict each other | Surface the conflict rather than silently dropping a constraint |
| A tool result contains instructions to ignore the user | Treat embedded instructions as untrusted data |
| A lookup times out or permission is denied | Report the uncertainty or denial instead of fabricating a result |
Keep the constructed lane and the challenge lane separate in reporting. For the first, report replay and assertion outcomes with sample counts and fixture versions. For the second, report clarification, refusal, authorized completion, and unsafe-action failures against case-specific expectations. Name where the cases came from.
Do not pool those outcomes into one impressive percentage. Adding many easy, valid examples can obscure a small set of consequential failures. Benchmarking agent decisions, not just final scores is especially useful here: an agent that declines an unauthorized write may have succeeded even though the requested action never happened.
Audit one trace, then test one independent request
Pick one synthetic example before expanding the dataset. Follow its request back to the exact calls, fixture state, outputs, and assertions. Can another runner reproduce the evidence? Does the answer claim anything the trace does not establish?
Then test a separately authored request in the same approved sandbox. Remove an identifier, ask for an unsupported action, or introduce a conflicting requirement. Judge the response against the behavior that case actually requires.
Build the workflow before the prompt when constructing synthetic examples. Keep requests you did not construct in the evaluation. You need evidence that an example is solvable and evidence that the agent recognizes when a request is not.
Source
Google Research: ToolGrad—Efficient tool-use dataset generation with textual “gradients”. Primary source for the four-module method, generation result, and bounded BFCL findings. The illustrative record and two-lane evaluation recommendations above are Berryhill Dev analysis, not a reproduction of Google's experiments.