Esacode

Docs / Blocks / Data

Hugging Face dataset

Reads files of a dataset on the Hugging Face Hub at a fixed revision, all or a repeatable sample, and records the exact commit read.

Runs on a worker (Python) Data Version 1.2 Code you can open and adapt

How it works

Repository: owner/name or the dataset's link, such as imageomics/GZCD. Revision: a branch, a tag or a commit (default main); it is resolved to its commit, so the run's records say exactly which version of the data it read, and reading the same commit again gives the same files. Keep: patterns of the files to read (images/*.jpg); Also: files read every time and given apart (metadata, annotations, the card).

Most files: a random but repeatable sample (Seed), for a demo-sized set. Files are downloaded once into the worker's cache. Public datasets need no account; for a private one, paste a token with read access. Respect the dataset's licence: it is in the manifest.

Outputs

OutputGivesWhat it is
Files
files
FilesThe files read, in their folders
Other files
extra
FilesThe files of Also (metadata, annotations)
Manifest
manifest
Table
csv, json
manifest.csv (path, size, sha256) and source.json (repository, commit, link, licence)
Live telemetry
telemetry
Live telemetryWhat the block sends while it runs, for views such as Chart or 3D view.

Settings

SettingDefaultWhat it does
Repository
repo
empty owner/name or https://huggingface.co/datasets/owner/name
Revision
revision
main A branch, a tag or a commit
Keep
keep
*.jpg, *.jpeg, *.png Patterns of the files to read, separated by commas (images/*.jpg)
Also
also
empty Files to read every time, apart: metadata, annotations (final_metadata.json, README.md)
Most files
max_files
0 A random sample of at most this many files; 0 reads them all
From 0 to 1000000
Seed
seed
1
Kind
repo_type
dataset
One of: dataset, model
Token (private datasets only)
token
pasted in the editor, kept encrypted Empty: the Hugging Face token in the profile of whoever runs the flow (needed for private or gated datasets only)

Where it runs

On one of your computers with the EsaSim worker: Any worker. The worker installs the Python packages it needs when they are missing: huggingface_hub.

In its code, settings are ctx.params["key"], inputs ctx.inputs["id"] and outputs ctx.output("id", "name.ext"); see Custom blocks.

More in Data