Tool Use in LLMs: From Function Calling to Autonomous Agents

Function calling is constrained decoding against a JSON Schema. That is the whole mechanism. The model is sampling tokens as usual, but the decoder masks the logits at each step so that only tokens which keep the output a valid prefix of a schema-conforming JSON document have nonzero probability. The model cannot emit a malformed call for the same reason a compiler cannot emit a syntax error: the invalid branches were removed before sampling, not validated afterward.

Everything labeled "agentic" is a loop around that primitive plus a condition for stopping. There is no second mechanism. Once you internalize that, the reliability problems stop being mysterious.

The loop, written out

messages = [system, user_turn]

while True:
    resp = model(messages, tools=TOOL_SCHEMAS)

    if resp.stop_reason != "tool_use":
        return resp.text                      # the stopping condition

    for call in resp.tool_calls:
        result = registry[call.name](**call.arguments)
        messages.append(assistant_tool_use(call))
        messages.append(tool_result(call.id, result))

And a single iteration on the wire, using the Anthropic shape:

--> "tools": [{
      "name": "recall",
      "description": "Retrieve prior context for this project...",
      "input_schema": {
        "type": "object",
        "properties": {
          "query": { "type": "string" },
          "limit": { "type": "integer", "maximum": 10 }
        },
        "required": ["query"]
      }
    }]

<-- { "stop_reason": "tool_use",
      "content": [
        { "type": "tool_use", "id": "toolu_01A",
          "name": "recall",
          "input": { "query": "postgres pool sizing", "limit": 3 } }
      ] }

--> { "role": "user", "content": [
        { "type": "tool_result", "tool_use_id": "toolu_01A",
          "content": "Pool size 20 per instance after the pgbouncer switch (2026-06-02)." }
      ] }

The tool_use_id correlation is not decoration. It is what lets the model handle parallel calls and out-of-order results. Drop it or reuse it and you get behavior that looks like the model hallucinating results, when in fact you handed it a transcript where the results are attached to the wrong calls. In MCP the same exchange is a tools/call JSON-RPC request, with tool-level failures returned inside result rather than as transport errors, which is documented in the MCP specification.

Three ways the loop kills you

Compounding error. If each step succeeds with probability p, an n-step task succeeds with p^n. At p = 0.95, a five-step task is 77% and a fifteen-step task is 46%. This is arithmetic, not pessimism, and it is why agent demos work and agent products do not. The only real levers are raising p (better schemas, better tool descriptions, narrower tools) or lowering n (fewer, more capable tools). Adding a retry raises effective p only for transient failures, and does nothing for the case where the model confidently made the wrong call.

No rollback for side effects. Step 7 of 10 sends an email, and step 8 reveals the plan was wrong. There is no transaction. The model has no undo primitive unless you built one, and most tool registries do not. The practical mitigation is to partition tools into read-only and effectful, run all reads before any write in a planning phase, and require confirmation before the first effectful call. This is unglamorous and it works.

Context growth per iteration.Every iteration appends the tool call and the full tool result. A tool returning 3k tokens, called eight times, is 24k tokens of transcript before you count the model's own reasoning. Cost per iteration grows because the entire transcript is reprocessed, so total token spend on an n-step task is quadratic in n, not linear. Truncating tool results to what the next decision needs is the highest-leverage optimization in a tool-using system, and almost nobody does it because the untruncated version works fine at n = 3.

Schema design is where you buy reliability

Wrong-tool selection is a description problem far more than a model problem. Things that measurably reduce it:

• Fewer tools. Selection accuracy degrades with the number of candidates. Past roughly fifteen to twenty tools in one request, models start conflating adjacent ones. Merge tools that differ only by a parameter into one tool with that parameter.
• Disjoint descriptions. If two descriptions could plausibly both apply, the model will pick between them essentially at random. search_notes and search_documents is a trap. Write the boundary explicitly into both descriptions.
• An explicit negative clause."Do not use this for questions about the current conversation; it only searches sessions that have ended" suppresses a specific failure mode more reliably than any amount of positive description. Models follow prohibitions in tool descriptions better than most people expect.
• Constrain in the schema, not the prose.An enum with four values eliminates invalid values at the decoder. A description saying "must be one of four values" does not.

Idempotency is what makes retries safe

Retries are the main tool for pushing p up, and they are only safe if a repeated call is indistinguishable from a single call. That requires a caller-supplied key:

def send_invoice(customer_id, amount, idempotency_key):
    if seen(idempotency_key):
        return cached_result(idempotency_key)     # replay, no new side effect
    result = do_send(customer_id, amount)
    record(idempotency_key, result)
    return result

Derive the key from the semantic content of the intended action, not from a random UUID the model generates, or a retry with a fresh UUID just performs the operation twice. Hash the arguments plus a task identifier. Then a timeout, an ambiguous network failure, or the model re-issuing a call it thinks did not land are all survivable, and you can retry aggressively without auditing every tool for double-execution.

An agent is a while loop with a model in the condition. Reliability engineering for agents is therefore ordinary reliability engineering: bound the retries, make the operations idempotent, and keep n small.

One thing the loop cannot supply is state that outlives it. When the loop exits, the transcript is discarded and the next invocation starts from an empty message list, which is why an agent that correctly figured something out on Tuesday re-derives it on Wednesday. Unimatrix exposes remember and recall as ordinary tools inside that loop so a conclusion reached once persists across sessions, models, and devices. The schemas are in the MCP reference.