Esacode

Docs / Build

Data and datasets

Connect where your data lives (folders on your workers, S3 buckets, Google Drive, Dropbox, OneDrive), upload and download files, browse them with a viewer for each kind, and send results from flows to datasets as they are made.

How data works in EsaSim

A dataset is a folder of files with versions: camera frames, videos, logs, recordings, tables, point clouds, models, documents. It lives in a connection, which is where the files are kept:

ConnectionWhere the files areWho reads them
S3 bucketAmazon S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO or Google Cloud Storage (HMAC keys)EsaSim, with the keys you give it
Google Drive, Dropbox, OneDriveA folder of the account you connect (My Drive or a shared drive; OneDrive or a SharePoint library)EsaSim, with the access you allow when you sign in
Folder on a workerA folder one of your workers shares, starting with each worker's own storageThe computer's worker; blocks that read it run there

EsaSim itself keeps no data files: it keeps the list of files of each version, their checksums, small image previews and the history of runs. Files you upload in the browser pass through EsaSim on their way to the worker, bucket or drive, and are not kept. Datasets made before 0.12.7 in EsaSim storage show Move to a worker on their page.

The data a project reads always stays where its owner keeps it, reached through a connection they set up; copies of a project never carry those connections or their keys. The results runs make stay on the workers that made them and open from there; only small ones (summaries and previews) come to EsaSim, within limits per file, run and workspace. A platform admin can have every result stored on EsaSim instead (Admin › Integrations). Trained policies go to GitHub when you publish a version of them (Projects › Publishing trained policies).

For every dataset EsaSim keeps an index: the list of files with their size, time, checksum and kind, a few facts (image size, log lines, table columns and rows) and small pictures of images. It also keeps which runs read each version and which run made it.

Connect where the data is

Under Data > Connections, owners and admins of the workspace choose Add a connection:

  • S3 bucket: the bucket, its region, the endpoint for services other than Amazon (for example https://<account>.r2.cloudflarestorage.com), an optional folder (prefix), and an access key and secret key that may list, read, write and delete objects in that bucket. EsaSim checks the keys before it keeps them; they are stored encrypted and never shown again.
  • Google Drive, Dropbox, OneDrive: an optional folder, then Sign in and allow EsaSim to read and write files. The connection keeps working until you remove it here or in the account's settings; Sign in again renews it.
  • The worker's storage: from worker 0.9 every computer shares a folder of its own (<data folder>/storage, or where esasim-worker storage --path puts it, with an optional size set by --max-gb). It appears as <worker> · Storage when you create a dataset, and it is writable. See Storage and shared folders.
  • Folder on a worker: in the worker's Worker settings, tick Share folders with EsaSim and add each folder with a name and its full path. Tick Writable (uploads, blocks) where EsaSim may write: uploads, deletions and results go only into such folders. Listing a folder turns sharing on. If you pair the same computer again, it becomes a new worker and its folders stay with the old one: the new worker's settings offer Move them to this worker, which brings the folders and their datasets over.

Google Drive, Dropbox and OneDrive need an app registered once for the whole platform by a platform admin, under Admin > Integrations, which shows the redirect address to register with each provider.

Create a dataset

Data > New dataset: a name, where the files are (a worker's own storage is chosen first, so the files stay on that worker), an optional folder inside the connection, which files (patterns such as *.png, *.jpg or *.mcap), and the project where it is listed. For storage and cloud drives, an empty folder means a new folder named after the dataset; for a folder on a worker, the whole shared folder. A folder that already holds files is indexed at once.

Upload and download

  • Upload on a dataset's page, or drop files or whole folders anywhere on it. Folders keep their structure; files go into the folder you are looking at, and a file with the same name is replaced. Files go up in parts, so large files pass any server limit (up to 20 GB each). When the last one has arrived, the dataset gets a new version.
  • For a folder on a computer, the files go to EsaSim first and the computer's worker writes them into the folder within a few seconds (worker 0.8 or later); the page shows its progress.
  • Download one file with its button, or select files and choose Download as zip (up to 4 GB). Files on a computer are fetched by its worker first; the page shows the progress, then the download starts.
  • Move to a worker, on the page of a dataset kept in EsaSim storage: choose a folder of one of your workers (its storage, or a shared folder marked Writable). The dataset keeps its name, project and history and points to that folder at once; the computer's worker writes the files there while it is online, a new version follows, and the copies on the EsaSim server are deleted once every file has arrived (all are kept if one fails).
  • Delete selected files of the latest version: they are removed where they are kept (Google Drive moves them to its bin) and a new version follows.

Browse and view

A dataset's page shows its folders and files at the chosen version, as a grid of pictures or a list, with search, filters by kind, and Every file, without folders. Open a file to view it; the arrow keys move to the previous and next file:

KindViewer
Images (PNG, JPG, WebP, GIF, BMP)Zoom with the wheel, drag to move, 100% or fit, a strip of the other pictures
Videos (MP4, WebM, MOV) and audio (WAV, MP3, OGG, FLAC, M4A)Player with seeking
Tables (CSV, TSV)Sortable columns, a filter, pages of 200 rows
JSON and JSON linesA tree that folds and unfolds
MarkdownFormatted, or as text
Text and logs (TXT, LOG, YAML, XML, TOML, INI)Line numbers, find, wrapping
PDFThe browser's viewer
3D models and point clouds (glTF, GLB, OBJ, STL, PLY)The 3D view, to turn around

Other files are downloaded. Large text and tables show their first megabytes. Files on a computer are fetched by its worker when you open them, and kept on EsaSim for a day.

Versions

Each upload, deletion, run that writes into the dataset, or Scan again (after files changed in the connection itself) makes a new version if something changed, with the counts of added, removed and changed files. A run always reads one fixed version, recorded under Read by; choose a version number in the Dataset block to repeat a run exactly.

Data in flows

  • The Dataset block brings a version into a flow. A dataset in a folder on a worker is read in place, so the blocks connected to it run on that worker. A dataset in a bucket or a drive is downloaded by the computer that runs the next block, before it starts, so any computer can use it (worker 0.8 or later).
  • Outputs go to, in the settings of any block that runs on a worker, sends its result files to a dataset in any connection: choose the connection, the dataset's name and folder, which outputs, As they are created (each file is sent as soon as it stops changing, while the block keeps working) and Move them (no copy stays in the task folder). When the block ends, the dataset gets a new version whose source is the run and the dataset versions it read. Blocks connected after it still receive the files.
  • Save as dataset copies the files it receives into a folder on the worker, as a new version.

Inputs and results of a run

A run's evidence is in its project's Results tab (the Runs tab lists the runs with their tasks and logs). For each run it shows the dataset versions it read, with some of their pictures and a link to browse them, and its result files, by block and output, in the same browser and viewers. Results under 5 MB are kept on EsaSim; files sent to a dataset are read there; the others stay in the task folder on the worker and are fetched when you open them.

A curation flow

  1. Dataset: last week's camera frames and logs, wherever they are.
  2. Agent with an LLM: looks at each image or log and labels it (keep what shows a failure worth simulating), with its reasoning.
  3. Select files: keeps the files labelled keep, with Outputs go to a dataset Curated failures in a bucket: each kept file arrives there as soon as it is chosen.

The same shape works for simulation results and training data: no project is fixed to one use, the blocks are the tools.

The tabs of Data

TabWhat it holds
DatasetsYour workspace's datasets, in every connection.
ProjectsThe results of runs, as folders: one per project you can open, then one per flow, then its runs.
EsaSim libraryData EsaSim offers to everyone for free: see The EsaSim library.
ConnectionsBuckets, cloud drives and the folders your workers share, with their status.

Security

  • Keys and sign-ins of connections are stored encrypted on the server and used only by EsaSim to reach the files; they are never sent to browsers or workers. Certificates are always checked.
  • Files are shown in the page with headers that keep the browser from running anything they contain.
  • Only the folders listed in a worker's Worker settings can be read, and only inside them: paths with .. are refused and symbolic links are not followed. EsaSim writes and deletes only in folders marked Blocks may write.
  • A computer's worker fetches a file of a task folder only inside that folder.