Esacode

Docs / Guides

AI agents and models

The LLM and Agent blocks: connect a model (EsaSim, a provider with your key, or your own server) and let an agent review images, logs and tables in a loop.

The LLM block

The LLM block says which language model a block uses. Wire its output into the LLM input of an Agent (or of your own Python block, which then reaches it as ctx.llm). Choose where the model runs:

Where it runsWhat you giveBilling
EsaSim modelsPick a model from the listPer token, from the workspace's credits, at the prices shown under the list
Anthropic, OpenAI, Google Gemini, Mistral, xAI, DeepSeekThe model name and your API keyBy the provider, on your account
Custom serverThe address of an OpenAI-compatible server (for example Ollama or vLLM: http://localhost:11434/v1) and the model name; a key only if the server needs oneNone from EsaSim

A key you paste is stored encrypted in the workspace and shown afterwards only by its last four characters. It is sent only to the worker that runs the block, as an environment variable; it is never written into the flow, its versions, or the task files.

EsaSim models are called through EsaSim: the block gets a token valid only while its task runs, EsaSim calls the model and charges the tokens used. If the credits run out, the call stops with a clear message and the agent records it.

The Agent block

The Agent works through your data one item at a time. An item is a file (an image, a log, any text) or a row of a table. For each item it:

  1. gets the task prompt, filled in with the item's name ({name}), kind ({kind}), position ({index} of {total}) and, for a row, the row itself ({row});
  2. works in steps: each step it answers with one action, such as read_text or view_image, and sees the result;
  3. ends with decide: a label from the allowed labels, a confidence from 0 to 1 and one sentence of reasoning.

Its settings:

  • System prompt: who the agent is and how it should judge. Task for each item: the question for every item. Both are plain text you can rewrite.
  • Allowed labels: for example keep, reject or collision, near miss, nominal.
  • Skills: what it may do. read_text reads a file or row (in pages), view_image shows it the picture (models that see images), read_table reads the first rows of a table, list_files lists every file given to it, note keeps a note in the results.
  • Steps per item, At most (items) and a Token budget for the whole run keep the cost bounded. When the budget is reached the agent stops and says so in its report.

It writes three outputs: Decisions (a table with the item, label, confidence, reasoning, steps, tokens and notes), a Report (counts per label and every decision) and Totals. While it works it sends live values on the channels agent (each step) and decisions (each item), which a Telemetry block can show in a table or chart.

Changing the harness

The loop itself is the Agent block's Python code: open it with Edit code. Skills are plain functions in the SKILLS list; add your own (for example one that runs a detector on an image, or reads a ROS bag) and name it in the Skills setting. The model is reached with ctx.llm.chat(messages, system=...), where a message's content is text or a list of parts, including images (esasim_llm.image_part(path)).

Example: curating flight logs and camera frames

The template Agent reviewing files wires Files on a worker (a folder of images and logs), an LLM and an Agent, with a table and a report as views. Set the folder, choose the model, and run it on the worker that has the folder: the agent labels each file and explains why, and the decisions table can feed the next blocks of the flow.