Docs / Build
Build your own blocks
Make a block of your own in Python (runs on your workers) or JavaScript (a view in the browser): its ports and their formats, its settings, its code, versions, sharing with the EsaSim library and copies on GitHub.
New to writing blocks? Tutorial: blocks that work together builds a worker block and views step by step, with live values, a button that talks back to a running block, and blocks on two workers. A project in code writes a block and its blueprint as files in your own editor, with git.
What a block is
Every block in the block menu is a definition plus code. Nothing about a block is built into the editor: the same rules apply to EsaSim's own blocks, to the blocks of the EsaSim library and to yours.
| Part | What it holds |
|---|---|
Definition (block.json) | Name, category in the menu, icon, one-line summary, help, inputs and outputs (ports), settings, and for worker blocks the hardware it can run on. |
| Code | A worker block is Python (backend.py) with def run(ctx); it runs on your workers, as a task of each run. A view is JavaScript (frontend.js) with function render(ctx); it draws in the browser, on the flow's View page and next to results. |
A block of your workspace is in the block menu of every project of the workspace, under its category, and keeps a numbered version each time it is saved. Runs use its latest version.
Make one
- Open Projects > Blocks and choose New worker block or New view. In a flow, Add block > Make a block opens the same builder, and Save as my block in a block's settings turns a block you changed in a flow into a block of the workspace.
- Start from Hello world, or from a copy of a built-in block (on the Blocks page: Start a block from a copy). Its ports, settings and working code fill the form.
- Name it, choose its category and icon, and say in one line what it does.
- Add its inputs, outputs and settings (below). Until you change the code, the starter code follows the form.
- Write the code. Verify checks the syntax (with the same parser that colours it) and that
run(ctx)orrender(ctx)is there. - Save block. The block page shows its definition, its code, the flows that use it and its versions; Download gives a zip with
block.json, the code and a README.
The code panel and the builder can be widened: drag the left edge of the code panel, or the line between the form and the code in the builder, and use the full-screen button for the whole window.
Ports: inputs and outputs
A connection goes from an output of one block to an input of another. Each port has an id (a short name used in the code: a letter, then letters, digits or _), a label, a type, and optionally the formats it reads or makes and a line on what it carries. Inputs can take several connections and be optional.
| Type | What travels | Typical formats |
|---|---|---|
| Files | Any files: the block receives a list of files (path, name, size). | png, jpg, mp4, mcap, bin … |
| Table | Tables: CSV with a header row (also TSV, or a JSON list of objects). Views get the rows; worker blocks get the files. | csv, tsv, json |
| Items | Records as JSON lines: one object per line, such as the decisions of an Agent. | jsonl |
| 3D scene | A scene as JSON in the format esasim.scene/1 (entities and runs of timed positions), with the 3D files it names. | json, glb, gltf, obj, stl, ply |
| Metrics | Named numbers in one JSON object, such as {"success_rate": 0.69}. | json |
| Text | Text or Markdown: reports, notes, logs. | md, txt, log |
| Model | A connection to an AI model, from an LLM block (inputs only): ctx.llm. | — |
| Anything | An input that accepts files of any type. | — |
| Live values (stream) | Inputs of views only: the live values of a worker block (its Live telemetry output wired straight in) or of a Telemetry block. The view reads them in ctx.input("port").stream. | — |
| Live telemetry | Values a worker block sends while it runs. Every worker block has this output; you do not add it. | — |
A table, metrics, a 3D scene and text are files too: they can go into an input of type Files. When both ends list formats and none is in common, the editor warns (for example, a block making csv wired into an input reading png, jpg).
Python packages
A worker block can use any Python package. Before it runs, the worker installs the ones that are missing on that computer, with pip, into its data folder (python-packages, one set per version of Python, and pip's download cache in pip-cache), never into the system's Python or your user profile; later runs reuse them. Which ones:
- those the code imports, found by themselves:
import numpyinstalls numpy,from PIL import Imageinstalls Pillow,import cv2installs opencv-python-headless,from numba import cudainstalls numba andnumba-cuda[cu12](the CUDA target with its CUDA libraries); - those listed under Python packages in the block's definition, for names that differ from the import, extras or versions:
torch>=2.3,numba-cuda[cu12],pillow>=10,<12. Only names, extras and versions: no pip options or URLs.
The first run on a worker shows the installation in the log and takes as long as the download. The worker's Install the Python packages blocks need setting (on by default) can turn it off; the block then reports which packages are missing. GPU code needs the GPU driver on the worker; CUDA libraries come with packages such as numba-cuda[cu12] or cupy-cuda12x, so the CUDA toolkit is not needed. Blocks run in Docker install their packages in the container at each run, unless the image already has them.
PyTorch on an NVIDIA GPU. List torch[cuda] (with a version if you need one: torch[cuda]>=2.4) when the block must use the GPU. On Windows, pip's ordinary PyTorch has no CUDA, so the worker installs PyTorch built for CUDA 12.6 from PyTorch's own package index instead (about 2.5 GB, once per computer) and replaces a PyTorch without CUDA it installed before. On Linux the ordinary PyTorch already has CUDA, and a PyTorch already in the worker's Python is kept as it is, such as one installed with a Tenstorrent card's PyTorch/XLA support. A plain torch keeps working as before.
A block has at most 60 settings.
CUDA with Numba. A block that uses Numba prints a CUDA check in its log before it starts: the Numba, numba-cuda, NumPy and Python it uses, the GPUs the task was given (CUDA_VISIBLE_DEVICES), whether the NVIDIA driver answers, and whether kernels compile. When the driver does not answer, the line says why. Check that nvidia-smi works on the computer; on Windows, if the worker runs as a service and the driver still does not answer, set the service to log on as your own account (services.msc → EsaSim Worker → Log On) and restart it. With numba-cuda, NumPy is kept below 2.5, the first version numba-cuda 0.30 does not work with.
Settings
Settings appear in the block's inspector and reach the code as ctx.params.
| Type | Shown as |
|---|---|
| text, long text, prompt | A text field (prompt: a larger box for instructions to a model). A text setting can instead pick one of the files wired to an input: with "files_from": {"port": "policy", "pattern": "*.pt"} the inspector lists the files of the datasets wired to that input that match, newest first, and an empty choice for the newest at each run. Add "by": "robot", "match": {"Unitree H1": "humanoid_h1_walk_*"} to narrow the list by another setting's choice. |
| number | A number, with an optional minimum and maximum the editor checks |
| choice | A list of options. When you rename an option, list the old names in "renamed": {"old option": "new option"}: flows saved with the old name keep working and show the new one. |
| yes / no | A switch |
| table | Rows of values with the columns you name (text, number, choice or colour), each row a card with Add and remove; for example the objects of the 3D scene block. Write the columns as name, speed:number, colour:color, kind:select(a|b). The code gets a list of rows: [{"name": "drone", "speed": 3.5}, …], at most 50 by default. |
| secret | An API key, pasted once in the block. It is stored encrypted and reaches the code only as an environment variable, ESASIM_SECRET_<KEY>, whose name is in ctx.params["<key>_env"]. With "profile_key": "huggingface", a block left without a key uses the Hugging Face token in the profile of whoever runs the flow (then ctx.params["<key>_from"] is "profile"). Worker blocks only. |
Where a worker block runs
"hardware" lists what the block can run on: cpu (any worker), nvidia, amd, intel, ascend, tenstorrent, apple. A run sends the block only to a worker that has one of them, and waits while none is online. When a setting chooses the accelerator, "hardware_by" narrows the list by its option:
"hardware": ["nvidia", "cpu"],
"hardware_by": {"param": "accelerator", "map": {"auto": ["nvidia"], "NVIDIA GPU": ["nvidia"], "CPU": ["cpu"]}}
Here the block goes to an NVIDIA worker unless the person running it picks CPU; options the map does not name keep the whole hardware list.
Worker blocks: def run(ctx)
def run(ctx):
frames = ctx.inputs.get("frames", []) # list of input files
for i, f in enumerate(frames):
data = open(f.path, "rb").read() # or f.read_text(), f.read_json()
ctx.progress(100 * (i + 1) // len(frames), f"{i + 1} of {len(frames)}")
ctx.telemetry("size", {"step": i, "value": len(data)})
ctx.write_text("table", "sizes.csv", "frames\n%d\n" % len(frames))
return {"frames": len(frames)} # becomes the "result" output
| In the code | What it is |
|---|---|
ctx.params["key"] | The block's settings |
ctx.inputs["port"] | The files connected to an input: .path, .name, .size, .read_text(), .read_json(), .from_node; from a parallel block .task; from a dataset .rel (its path in the dataset) and .kind |
ctx.output("port", "name.ext") | The path to write an output file (folders are made); ctx.write_json(port, name, data) and ctx.write_text(port, name, text) write it for you |
ctx.telemetry("channel", data) | Live values while the block runs (a number, a dict, a list, up to 60 KB), shown in the console's Live tab and by the views wired to the block; they reach the dashboard within about a second |
ctx.picture("channel", image) | A live picture (a file path, JPEG/PNG bytes, a PIL image or a numpy array; bgr=True for OpenCV), scaled down and sent as JPEG for the Images view |
ctx.messages("channel", wait=0) | Messages sent from the dashboard while the block runs (Controls and Camera views, or ctx.send in any view): a list of dicts with channel, data and t, oldest first; wait waits up to that many seconds for one. ctx.message("channel", timeout) returns the next one |
ctx.progress(pct, "note") | Progress on the grid and the run page |
ctx.log(...), print(...) | Lines of the run's log, in the editor's console |
ctx.task | .index, .count, .kind (run or combine) for blocks that run as parallel tasks |
ctx.llm | With an input of type Model: ctx.llm.chat(messages, system=...) returns .text and the tokens used |
ctx.folder | The task's folder on the worker |
ctx.time_left(), ctx.max_minutes | Seconds left before the run's time limit (Stop after) stops the block, and the limit in minutes; None without a limit. Long blocks use it to finish in time and keep their outputs |
The block runs with the worker's Python 3 (or in Docker, when the worker runs blocks there). Packages your code imports must be installed on the worker; Verify lists the ones outside Python's standard library. A worker runs code from projects only if its owner turned on Run code from projects in its Worker settings.
Views: function render(ctx)
function render(ctx) {
ctx.el.innerHTML = '<div class="viz-pad"><h3 class="viz-h"></h3><pre class="out"></pre></div>';
ctx.el.querySelector('.viz-h').textContent = ctx.params.title || 'Rows';
function draw() { ctx.rows('data').then(function (rows) { ctx.el.querySelector('.out').textContent = rows.length + ' rows'; }); }
ctx.on('update', draw);
draw();
}
ctx.el is the element to draw into; ctx.input("port") gives { files, stream }; ctx.rows, ctx.json and ctx.text read tables, JSON and text; ctx.files.text / json / buffer / url(file) read one file; ctx.on("update", fn) runs when new data arrives and ctx.on("resize", fn) when the view changes size; ctx.THREE and ctx.uPlot are three.js and uPlot; ctx.save(name, blob) offers a file the view made (a picture, a video) for download or a dataset; ctx.saveFiles([{ name, file }, { name, blob }], title) hands several at once (files of its inputs, or things it made), which the page saves into a dataset as one new version, in a folder, or downloads as one zip (the Video review view saves the kept videos this way). A large file can travel on the live stream in pieces of telemetry that the view puts back together, as Text to video sends whole videos while it runs; the page sends a view only the live items it does not have yet. ctx.send("channel", data) sends a message to the running blocks of the run (they read it with ctx.messages), and ctx.run.active, ctx.run.id and ctx.run.status tell the run shown (ctx.on("run", fn) when that changes). ctx.peer("topic", data) tells the other views on the same page something; they get it with ctx.on("peer", fn), fn(topic, data) (the 3D view's onboard cameras use it to show themselves in the main view). Views run in a sandboxed frame that cannot reach the dashboard or the network: data comes in through ctx. In the code panel, Preview runs the view with sample data.
Views that start runs
A view can start a run of its flow itself, so the View page works like an app: a prompt box and a Generate button, a form of settings and a Simulate button. Tick Starts runs in the view's definition (from an AI assistant: "permissions": ["run"]), then call ctx.run from a click:
button.addEventListener('click', function () {
ctx.saveState({ prompt: box.value }, true); // keep what was typed: the new run shows the view again
ctx.run({ params: { gen: { prompt: box.value, count: 4 } } }) // node id → { setting key: value }
.then(function (r) { status.textContent = 'Run #' + r.job + ' started'; },
function (err) { status.textContent = err.message; });
});
- Settings for this run only.
paramsgives values for settings of the flow's worker blocks: text, numbers (within their range), true or false, and the options of a choice. They go to this run and are written in its log; the flow keeps its own. Secrets, datasets, folders and tables cannot be set this way. - Only on a click.
ctx.runstarts a run only during a click or key press of the person looking at the view, who needs the runner role in the project. One run of the flow at a time: while one goes, it answers with the reason. - Where it runs. Like the last run of the flow: its placement or worker, and its Stop after. The first run is placed automatically.
- What comes back. The view then shows the new run: its result files and live data arrive as in any view, and
ctx.run.statusfollows it tosucceededorfailed.
The built-in Prompt view does all this for generative flows: wire the output of Text to image, Text to video or any block with a Prompt setting into it, and it shows a prompt box, more fields if you want them (number count 1 20 4 Images), a Generate button, the last prompts, and the pictures, videos or text the run makes.
The header of the code
Each time a block is saved, EsaSim writes a comment at the top of its code from the definition: the block's name, id and version, its workspace and author, every port with its type and formats, every setting with its default, and how to reach them in the code. You never edit it: change the definition and save, and the header follows. Built-in blocks carry the same header, with EsaSim as publisher.
# ── EsaSim block ──────────────────────────────────────
# Frame counter (ws-frame-counter) · version 3 · machine block, Python: def run(ctx)
# Counts frames and writes a CSV.
# Workspace: Acme Robotics · Author: Maya Chen
# Inputs ctx.inputs["id"] is a list of files: .path .name .size .read_text() .read_json()
# frames Files (png, jpg) Frames · several
# Outputs ctx.output("id", "name.ext") is the path to write
# table Table (csv) Counts
# Settings ctx.params["key"]
# min_size number Minimum size = 10
# ── end of EsaSim block header ─────────────────────────
Versions
Each save is a new version, kept with who saved it and when; the block page lists them, each one downloadable. Flows always use the latest version of a workspace block. A block's code can also be changed in one flow only (Edit code in the block's settings): that change stays in that flow.
Share a block with every EsaSim workspace
On a block's page, workspace admins can send its current version to the EsaSim library with a note for the reviewers: what the block is for, what it needs on the worker, and how it was tested. The EsaSim team reads the exact code and definition sent. Once approved, every workspace finds the block in its menu, credited to your workspace; a later version is published only after its own review. Secrets never travel: secret settings are empty in the library. You get an email with the decision.
Keep a copy on GitHub
On a block's page, link your GitHub account (or a personal access token: which permissions to give it), pick a repository, a branch and a folder, and Save to GitHub: block.json, the code and a README go in as one commit in your name. Tick Save every new version there too to commit each new version as it is saved. The link is yours, not the workspace's; tokens are stored encrypted and never shown again.
Under Profile > GitHub you link the account once and choose a default repository, branch and folder; there, Save my blocks by themselves commits every block you save, and you choose where projects go (a repository for each, or folders of one). A project brings the workspace blocks its blueprints use with it, without keys. Save everything now puts all blocks and projects of the workspace in one commit. Projects themselves live on GitHub and are saved there by themselves after every change: see Projects. The repository is laid out as <folder>/<workspace>/blocks/<block>/ and <folder>/<workspace>/projects/<project>/ (with project.json, a README and blueprints/*.json).
With A repository for each project (the default), each project gets a private repository of its own the first time it is saved, holding project.json, a README, blueprints/*.json and blocks/<block>/ for the workspace blocks its blueprints use, and nothing else. Saving opens a window that lists each step as it happens.
Bring projects back: GitHub and zip
A project's Settings has Update from GitHub: the blueprints in the repository replace the drafts of the project's blueprints (matched by blueprint, then by name; published versions are kept), new blueprints are added, and blueprints that are only in EsaSim stay. The workspace blocks the blueprints use come too: a block that is the same here is left alone, one that differs gets a new version (the window says which other projects use it), one that is missing is made. Keys pasted in blocks never travel: the blocks keep the keys they have here.
Export > Download as zip gives the same files as a zip, and Update from a zip takes them back. To start a project from either, open Projects > New project > Import a project and give a repository (and, when it holds several projects, the folder) or the zip.
Let an AI assistant build it
An assistant connected to EsaSim (see AI assistants) can read the block library, create and change blocks with save_block, wire them into flows and run them, with your access and within your role.
Limits
- Up to 300 blocks per workspace; code up to 300 KB; 12 inputs, 12 outputs and 30 settings per block.
- A block cannot change from a worker block to a view: make a new one.
- Deleting a block removes it from the menu; flows that use it show a problem until it is replaced. Past runs are kept.