Dataset
A dataset of the workspace, at a fixed version: in EsaSim storage, a bucket, a cloud drive or a folder on a worker.
How it works
Choose a dataset and its version (latest, or a number to repeat a run exactly). A dataset in a folder on a worker is read in place, so the blocks connected to it run on that worker. A dataset in EsaSim storage, a bucket or a cloud drive is downloaded by the computer that runs the next block, before it starts (worker 0.8 or later). Narrow the files with a pattern such as *.png.
The version a run read is recorded on the dataset (lineage).
Outputs
| Output | Gives | What it is |
|---|---|---|
Filesfiles | Files |
Settings
| Setting | Default | What it does |
|---|---|---|
Datasetdataset |
empty | |
Versionversion |
latest | latest, or a version number such as 3 |
Only these filespattern |
* | Patterns separated by commas, like *.png, *.jpg |
At most (files)limit |
200000 | From 1 to 200000 |
Where it runs
Its settings are used by the blocks connected to it; see Blueprints.
More in Data
Download filesDownloads files or a whole dataset from the web (Mendeley Data, Zenodo, GitHub, any link), unpacks archives and keeps a sample of each folder.Files on a workerReads files from a folder on the worker the flow runs on.Filter tableKeeps the rows of a table that match a condition.Hugging Face datasetReads files of a dataset on the Hugging Face Hub at a fixed revision, all or a repeatable sample, and records the exact commit read.LibraryFiles from the EsaSim library: 3D models, pictures, videos, logs and tables, free for testing.MergePuts the outputs of several blocks together.