Esacode

Docs / Blocks / Data

Download files

Downloads files or a whole dataset from the web (Mendeley Data, Zenodo, GitHub, any link), unpacks archives and keeps a sample of each folder.

Runs on a worker (Python) Data Version 1.3 Code you can open and adapt

How it works

Put one source per line: a direct link to a file or an archive (zip, tar, tar.gz), a Mendeley Data dataset page (data.mendeley.com/datasets/ID/VERSION), a Zenodo record (zenodo.org/records/ID) or a GitHub release asset.

The worker downloads them (into its cache, so a second run does not download again), unpacks the archives, and passes on the files that match Keep, with their folders: a folder name is often the label (Apple_Good, zebra). Most per folder keeps a random but repeatable sample of each folder (Seed), for a demo-sized set.

The manifest lists every file with where it came from and its checksum, and the licence you name is written next to the files, so they can be published again with the credit the licence asks for. Large datasets: for a zip on a server that sends parts of files (most data repositories do), only the files kept are read out of it, without downloading the rest (Read zips in place).

Case file and Cases take whole case folders (a patient, a scene: the folder that holds a file such as ct.nii.gz, with everything under it), a number of them at random or by name; Also keeps files such as meta.csv wherever they are.

Outputs

OutputGivesWhat it is
Files
files
FilesThe files kept, in their folders
Manifest
manifest
Table
csv, json
manifest.csv (path, folder, size, sha256, source) and source.json (links, licence, credit, totals)
Live telemetry
telemetry
Live telemetryWhat the block sends while it runs, for views such as Chart or 3D view.

Settings

SettingDefaultWhat it does
Sources
sources
empty One per line: a link to a file or an archive, a Mendeley Data dataset page, a Zenodo record or a GitHub release asset
Fetch only
fetch
empty Of the files of a dataset page, only those matching these patterns (such as *.zip); empty: all
Keep
keep
*.jpg, *.jpeg, *.png File patterns to keep, separated by commas
Only inside
inside
empty Keep only files under this folder of the archive (for example Good Quality_Fruits); empty: everywhere
Case file
case_file
empty A file that marks a case folder (such as ct.nii.gz): then Cases takes whole folders, with every file kept under them; empty: no cases
Cases
cases
empty How many case folders (a number, at random with Seed) or their names (s0011, s0040, s01*); empty: all
Also
also
empty Files kept wherever they are, besides Keep (such as meta.csv, LICENSE*)
Most per folder
per_folder
0 A sample of at most this many files from each folder; 0 keeps them all
From 0 to 100000
Most in all
max_files
0 0: no limit
From 0 to 1000000
Seed
seed
1 The same seed gives the same sample
Read zips in place
remote_zip
auto auto: read only the files kept out of a zip when the server allows it, else download it whole · remote: only in place · download: always the whole zip
One of: auto, remote, download
Licence
licence
empty The licence of the data (such as CC-BY-4.0), written with the files
Credit
credit
empty Who made the data and how to cite it, written with the files

Where it runs

On one of your computers with the EsaSim worker: Any worker. The worker installs the Python packages it needs when they are missing: requests.

In its code, settings are ctx.params["key"], inputs ctx.inputs["id"] and outputs ctx.output("id", "name.ext"); see Custom blocks.

More in Data