English · 简体中文
Generate API Reference¶
Added in v0.6.x. Source-of-truth: the OCaml function
Runtime.invoke_generateinlib/core/runtime.ml,Generate.runinlib/core/generate.ml, and thegenerate_resulttype inlib/core/types.ml.
This page is the API contract for PAR's pure long-output generation path. It locks the shape of invoke_generate, the generate_result return type, the auto-continuation behavior, and the events callers can observe. If you are writing a long-output agent (PRDs, HTML mockups, plans, documentation) and want to stop hand-rolling direct LLM calls, read the Usage Examples section. If you are wiring the FFI or porting an existing agent that currently bypasses Runtime.invoke, read the Auto-continuation behavior and Limitations sections first.
Overview¶
Long-output generation is a different workload from ReAct reasoning. A PRD writer produces 3,000 to 6,000 tokens of Markdown in one go. An HTML mockup agent emits a single large artifact. None of that involves tool calls, and none of it benefits from an iteration budget. Treating Max_tokens truncation as a loop-consuming failure (the ReAct default) is wrong for this class of work. The truncation is a transport detail. The model finished what it could; the runtime's job is to deliver the complete output, not to penalize the model for hitting the limit.
Downstream integration feedback confirmed the gap: long-output agents in an integrator were bypassing Runtime.invoke and hand-calling llm_chat_raw, keeping PAR only for session and event management. Plan §1 of the long-output generation mode plan documents the survey of four mainstream coding agents (Claude Code, Codex CLI, OpenCode, a comparable coding agent): none of them count Max_tokens as an iteration-consuming event. PAR was the outlier. Runtime.invoke_generate closes that gap by exposing a first-class entrypoint for pure generation. It skips the ReAct loop entirely, auto-continues on Max_tokens truncation, and reuses the same session store, event bus, LLM-service abstraction, and skill overlay machinery that invoke uses.
The generate path is intentionally narrow. It does not run middleware at the ReAct boundaries, does not consult max_iterations, and does not consult max_execution_time per iteration (a single optional total_timeout replaces it). What it shares with invoke is everything that should be shared: provider abstraction, session persistence, event publishing, skill composition, and the streaming callback shape.
When to use invoke_generate vs invoke¶
- Use
invoke_generatewhen the task is producing a long text artifact and no tool calls are needed: PRDs, HTML mockups, plans, documentation, reports, code listings where the model writes the whole file in one shot. It auto-continues acrossMax_tokenstruncation so the caller does not have to implement chunk-concatenation glue. - Use
invokewhen the task needs tool calls, multi-step reasoning, or ReAct loop semantics: agents that search, compute, run bash, query a database, or hand off to other agents.invokeis also the right choice when middleware at LLM and tool boundaries matters, sinceinvoke_generateskips that pipeline.
A rough rule: if the agent's tools list is empty and the output is long, reach for invoke_generate. If the agent has tools, reach for invoke. invoke_generate enforces the first half of that rule at registration time: an agent whose tools list is non-empty is rejected with Invalid_input.
Type definitions¶
generate_result¶
Returned by Runtime.invoke_generate. Distinct from invoke_result because the generate path does not run the ReAct loop, so the shape exposes continuation and token accounting instead of an iteration count.
type generate_result = {
text : string; (* Full concatenated output across the
initial response and all continuation
chunks. Empty only on total failure. *)
finish_reason : finish_reason; (* Stop | Tool_calls | Max_tokens |
Content_filter. Stop is the happy
path. Max_tokens means the
diminishing-returns guard halted
continuation. Content_filter means
the provider blocked the response. *)
continuations : int; (* Number of Continue sub-loop chunks
fired. Zero means the model emitted
Stop on the first response. *)
total_tokens : int option; (* Accumulated usage across continuations
when the provider reports usage.
None for providers that do not emit
token counts (e.g. OpenAI streaming). *)
session_id : string; (* Session the generation wrote to.
Persist with the conversation if you
want to resume later. *)
elapsed : float; (* Wall-clock seconds from entry to
return, including all continuations. *)
}
finish_reason reuses the existing ADT from lib/core/types.ml:
API signature¶
val Runtime.invoke_generate :
runtime ->
agent_id:string ->
message:string ->
?max_output_tokens:int ->
?total_timeout:float ->
?save:bool ->
?update_current:bool ->
?on_tool_event:(Types.event -> unit) ->
?on_chunk:(Types.llm_response_chunk -> unit) ->
?system_prompt_appendix:string ->
unit ->
(Types.generate_result, Types.error_category * Types.conversation) result
Parameters:
agent_idresolves a registered agent. The agent MUST havetools = []. Tool-bearing agents are rejected withInvalid_input.messageis the prompt or user message.max_output_tokensis an optional per-call cap on the initial response. Continuations accumulate beyond this until a Stop condition fires. When omitted, the agent'smodel.max_tokens(or the provider default) applies.total_timeoutis an optional wall-clock cap on the entire generation, continuations included. When omitted, the generation runs unbounded (bounded only by the diminishing-returns guard and natural Stop).save(defaulttrue) persists the generated conversation under the run's session.save=falseleaves the conversation store untouched.update_current(defaulttrue) makes the run's session the runtime's current session. Passfalsefor a background run:generate_result.session_idstill reports the session it used, but the runtime's current session is not replaced, so two background runs mint separate sessions.- Events published while a generate runs carry its session id; when the run ends the event bus returns to the caller's session. The bus stores one session id per runtime, so concurrent runs (for example two background generates) do not get per-run event attribution: once one finishes, events from the other are no longer tagged with its session. Use
on_tool_event/on_chunkwhen per-run attribution matters. system_prompt_appendixappends text to the system prompt for this run only.on_tool_eventis an observation callback. It fires forLlm_request_sent,Llm_response_received,Llm_response_truncated, andGenerate_continuation. No tool events fire because the generate path does not dispatch tools.on_chunkis an optional streaming callback. It fires for eachllm_response_chunkthe provider emits, mirroring the?on_chunkshape onRuntime.invoke. Use it for live UIs that want to render text as it lands.
The Error variant carries (error_category, conversation) so callers can persist the partial conversation even on failure, the same shape Runtime.invoke returns.
Auto-continuation behavior¶
The Continue sub-loop is what makes invoke_generate suitable for long output. The runtime initiates a normal LLM call. If the provider returns finish_reason = Max_tokens, the runtime fires an Llm_response_truncated event for observability, then injects a continuation prompt asking the model to resume from where it stopped, and issues a follow-up LLM call. The new chunk's text is concatenated onto the accumulator. The loop continues until one of these conditions fires:
- The provider returns
finish_reason = Stop(the model finished naturally). - The provider returns
finish_reason = Content_filter(the provider blocked the response). - The diminishing-returns guard trips: a continuation chunk adds fewer than 500 characters, signaling the model is stuck restating itself rather than making progress.
?total_timeoutelapses. If at least one chunk has landed, the runtime returns the accumulated text as a partial result. If nothing has landed, it returnsError (Timeout, conversation).
Each successful continuation emits a Generate_continuation event with the chunk index (0-based: the first continuation after the initial response is index 0) and the character count added. Callers can wire a progress UI off this event without inspecting the text.
This is the same Continue semantics the ReAct path uses (see agent_config.on_max_tokens = Continue in Agent API), factored out into a dedicated loop. The difference is that on the ReAct path, Continue is per-agent opt-in and capped at max_continuation_chunks (default 3). On the generate path, continuation is the default and the cap is removed; the diminishing-returns guard is the budget. Plan §1 documents the four-agent survey that motivated this: none of Claude Code, Codex CLI, OpenCode, or a comparable coding agent counts Max_tokens as a loop-budget event, and PAR aligns with that invariant for the pure-generation case.
Events emitted¶
The ?on_tool_event callback can observe these variants:
Llm_request_sent of { task_id; model }before every LLM round trip, including continuations.Llm_response_received of { task_id; usage }after every LLM round trip.Llm_response_truncated of { task_id; model; finish_reason }when the provider returnsMax_tokens.finish_reasonis alwaysMax_tokenshere.Generate_continuation of { task_id; chunk_index; chars_added }after each successful continuation chunk.chunk_indexis 0-based for the continuation chunks; the initial response is not a continuation.
No Tool_invoked, Tool_completed, Bash_invoked, or handoff events fire on this path. The generate loop does not dispatch tools.
Usage examples¶
Example 1: Basic generation (OCaml)¶
Register a tool-less agent, then call invoke_generate to produce a long PRD. The runtime handles continuation transparently; the caller sees the full concatenated text.
open Par
let () = Eio_main.run (fun _env ->
Eio.Switch.run (fun switch ->
match Runtime.create ~config:<runtime_config> switch with
| Error e -> prerr_endline (Types.string_of_error_category e)
| Ok rt ->
(* Tool-less agent: the only kind invoke_generate accepts. *)
let agent = {
Types.id = "prd-agent";
system_prompt = Types.stable_prompt "You write detailed product requirement documents.";
system_prompt_template = None;
model = { provider = `Openai; model_name = "gpt-4";
api_base = None; temperature = 0.4;
max_tokens = Some 4096; top_p = None; stop_sequences = None };
tools = [];
max_iterations = 1; (* unused on the generate path *)
middleware = [];
retry_policy = None;
context_strategy = None;
resource_quota = None;
max_execution_time = None; (* unused; total_timeout replaces it *)
early_stopping_method = Types.Force;
on_max_tokens = None; (* None = Auto: tool-less resolves to Continue *)
max_continuation_chunks = None; (* None = Auto: unbounded for tool-less *)
tool_timeout = None;
} in
(match Runtime.register_agent rt agent with
| Error e -> prerr_endline (Types.string_of_error_category e)
| Ok () ->
match Runtime.invoke_generate rt
~agent_id:"prd-agent"
~message:"Write a PRD for offline-first sync in a notes app."
()
with
| Error (e, _conv) ->
prerr_endline (Types.string_of_error_category e)
| Ok result ->
Printf.printf "%s\n" result.Types.text;
Printf.printf "finish_reason: %d continuations, %f s\n"
result.Types.continuations result.Types.elapsed));
ignore (Runtime.close rt))
)
The same call works without ?max_output_tokens and ?total_timeout; the defaults are the agent's model cap and unbounded, respectively.
Example 2: Python usage¶
Runtime.invoke_generate is exposed on the Python Runtime class. The return value is a dict with the generate_result fields plus the Ok / Error discriminator shape the FFI uses for all result types.
import json
from par_runtime import Runtime
config = json.dumps({
"persistence": {"tag": "sqlite", "contents": ":memory:"},
"llm_providers": [["openai", {"tag": "openai",
"contents": {"api_key": "sk-..."}}]],
"default_quota": {"max_tokens": 4096, "max_iterations": 10,
"timeout_seconds": 120.0},
})
with Runtime(config) as rt:
rt.register_agent(json.dumps({
"id": "prd-agent",
"system_prompt": "You write detailed PRDs.",
"model": {"provider": "openai", "model_name": "gpt-4",
"temperature": 0.4, "max_tokens": 4096},
"tools": [],
"max_iterations": 1,
"early_stopping_method": "Force",
}))
result = rt.invoke_generate("prd-agent", "Write a PRD for feature X.")
print(result["text"])
print(f"finish_reason={result['finish_reason']}, "
f"continuations={result['continuations']}, "
f"elapsed={result['elapsed']:.2f}s")
The agent config has tools = [] because invoke_generate rejects tool-bearing agents. The Python binding returns the same fields as the OCaml generate_result record.
Example 3: With streaming callback¶
Pass ?on_chunk to render text as it lands. The callback receives the same llm_response_chunk ADT that Runtime.invoke and invoke_stream emit. Concatenate Text_delta payloads to render incrementally.
import json
from par_runtime import Runtime, TextDelta
def on_chunk_json(chunk_json: str) -> None:
chunk = json.loads(chunk_json)
if chunk.get("tag") == "Text_delta":
print(chunk["contents"]["text"], end="", flush=True)
with Runtime(config) as rt:
rt.register_agent(json.dumps({
"id": "mockup-agent",
"system_prompt": "You produce self-contained HTML mockups.",
"model": {"provider": "anthropic",
"model_name": "claude-sonnet-4-20250514",
"temperature": 0.3, "max_tokens": 8192},
"tools": [],
"max_iterations": 1,
"early_stopping_method": "Force",
}))
result = rt.invoke_generate(
"mockup-agent",
"Mock up a settings page with light and dark modes.",
on_chunk=on_chunk_json,
total_timeout=90.0,
)
print() # newline after the streamed text
print(f"[done: {result['continuations']} continuations, "
f"{result['elapsed']:.2f}s]")
The total_timeout caps the whole generation, continuations included. If the model is still going at the deadline, the runtime returns whatever has accumulated.
Limitations¶
- Agent MUST have
tools = []. Tool-bearing agents are rejected withInvalid_inputat theinvoke_generatecall site. This is enforced rather than silently ignored, because the generate path has no tool dispatch. If your agent needs tools, useRuntime.invokewithon_max_tokens = Continueinstead. - No fallback chain. The generate path uses the agent's primary provider only. The cross-provider
fallback_policyconfigured on the runtime does not apply. Long generations that need provider diversity should run multipleinvoke_generatecalls and pick the best output upstream. - Wall-clock timeout returns partial on accumulated text. When
?total_timeoutfires after at least one chunk has landed, the runtime returnsOkwithfinish_reasonreflecting the last provider response and the accumulated text. When the timeout fires before any chunk lands (the initial LLM call hung), it returnsError (Timeout, conversation). - Diminishing-returns guard is fixed at 500 characters. A continuation chunk that adds fewer than 500 characters halts the loop. This catches models that get stuck restating themselves. It is not configurable in v0.6.x.
- No middleware at LLM boundaries. The
agent_config.middlewarepipeline does not fire on the generate path. Logging, retry, and rate-limit middleware that you rely on forinvokewill not run here. Wire equivalent behavior at the call site if you need it. max_iterationsandmax_execution_timeare ignored. They live onagent_configfor ReAct compatibility. The generate path replaces them with?total_timeout. The continuation count has no fixed cap; the diminishing-returns guard is the only ceiling.
See also¶
- Agent API -
agent_config,Runtime.invoke, the ReAct entrypoint and theon_max_tokenspolicy that mirrors generate's continuation logic - Streaming API -
invoke_stream, chunked delivery, and thellm_response_chunkADT that?on_chunkexposes here - Overview - SDK architecture and module map