Docs / Guides
AI agents and models
The LLM and Agent blocks: connect a model (EsaSim, a provider with your key, or your own server) and let an agent review images, logs and tables in a loop.
The LLM block
The LLM block says which language model a block uses. Wire its output into the LLM input of an Agent (or of your own Python block, which then reaches it as ctx.llm). Choose where the model runs:
| Where it runs | What you give | Billing |
|---|---|---|
| EsaSim models | Pick a model from the list | Per token, from the workspace's credits, at the prices shown under the list |
| Anthropic, OpenAI, Google Gemini, Mistral, xAI, DeepSeek | The model name and your API key | By the provider, on your account |
| Custom server | The address of an OpenAI-compatible server (for example Ollama or vLLM: http://localhost:11434/v1) and the model name; a key only if the server needs one | None from EsaSim |
A key you paste is stored encrypted in the workspace and shown afterwards only by its last four characters. It is sent only to the worker that runs the block, as an environment variable; it is never written into the flow, its versions, or the task files.
EsaSim models are called through EsaSim: the block gets a token valid only while its task runs, EsaSim calls the model and charges the tokens used. If the credits run out, the call stops with a clear message and the agent records it.
The Agent block
The Agent works through your data one item at a time. An item is a file (an image, a log, any text) or a row of a table. For each item it:
- gets the task prompt, filled in with the item's name (
{name}), kind ({kind}), position ({index}of{total}) and, for a row, the row itself ({row}); - works in steps: each step it answers with one action, such as
read_textorview_image, and sees the result; - ends with
decide: a label from the allowed labels, a confidence from 0 to 1 and one sentence of reasoning.
Its settings:
- System prompt: who the agent is and how it should judge. Task for each item: the question for every item. Both are plain text you can rewrite.
- Allowed labels: for example
keep, rejectorcollision, near miss, nominal. - Skills: what it may do.
read_textreads a file or row (in pages),view_imageshows it the picture (models that see images),read_tablereads the first rows of a table,list_fileslists every file given to it,notekeeps a note in the results. - Steps per item, At most (items) and a Token budget for the whole run keep the cost bounded. When the budget is reached the agent stops and says so in its report.
It writes three outputs: Decisions (a table with the item, label, confidence, reasoning, steps, tokens and notes), a Report (counts per label and every decision) and Totals. While it works it sends live values on the channels agent (each step) and decisions (each item), which a Telemetry block can show in a table or chart.
Changing the harness
The loop itself is the Agent block's Python code: open it with Edit code. Skills are plain functions in the SKILLS list; add your own (for example one that runs a detector on an image, or reads a ROS bag) and name it in the Skills setting. The model is reached with ctx.llm.chat(messages, system=...), where a message's content is text or a list of parts, including images (esasim_llm.image_part(path)).
Example: curating flight logs and camera frames
The template Agent reviewing files wires Files on a worker (a folder of images and logs), an LLM and an Agent, with a table and a report as views. Set the folder, choose the model, and run it on the worker that has the folder: the agent labels each file and explains why, and the decisions table can feed the next blocks of the flow.