Download files
Downloads files or a whole dataset from the web (Mendeley Data, Zenodo, GitHub, any link), unpacks archives and keeps a sample of each folder.
How it works
Put one source per line: a direct link to a file or an archive (zip, tar, tar.gz), a Mendeley Data dataset page (data.mendeley.com/datasets/ID/VERSION), a Zenodo record (zenodo.org/records/ID) or a GitHub release asset.
The worker downloads them (into its cache, so a second run does not download again), unpacks the archives, and passes on the files that match Keep, with their folders: a folder name is often the label (Apple_Good, zebra). Most per folder keeps a random but repeatable sample of each folder (Seed), for a demo-sized set.
The manifest lists every file with where it came from and its checksum, and the licence you name is written next to the files, so they can be published again with the credit the licence asks for. Large datasets: for a zip on a server that sends parts of files (most data repositories do), only the files kept are read out of it, without downloading the rest (Read zips in place).
Case file and Cases take whole case folders (a patient, a scene: the folder that holds a file such as ct.nii.gz, with everything under it), a number of them at random or by name; Also keeps files such as meta.csv wherever they are.
Outputs
| Output | Gives | What it is |
|---|---|---|
Filesfiles | Files | The files kept, in their folders |
Manifestmanifest | Table csv, json | manifest.csv (path, folder, size, sha256, source) and source.json (links, licence, credit, totals) |
Live telemetrytelemetry | Live telemetry | What the block sends while it runs, for views such as Chart or 3D view. |
Settings
| Setting | Default | What it does |
|---|---|---|
Sourcessources |
empty | One per line: a link to a file or an archive, a Mendeley Data dataset page, a Zenodo record or a GitHub release asset |
Fetch onlyfetch |
empty | Of the files of a dataset page, only those matching these patterns (such as *.zip); empty: all |
Keepkeep |
*.jpg, *.jpeg, *.png | File patterns to keep, separated by commas |
Only insideinside |
empty | Keep only files under this folder of the archive (for example Good Quality_Fruits); empty: everywhere |
Case filecase_file |
empty | A file that marks a case folder (such as ct.nii.gz): then Cases takes whole folders, with every file kept under them; empty: no cases |
Casescases |
empty | How many case folders (a number, at random with Seed) or their names (s0011, s0040, s01*); empty: all |
Alsoalso |
empty | Files kept wherever they are, besides Keep (such as meta.csv, LICENSE*) |
Most per folderper_folder |
0 | A sample of at most this many files from each folder; 0 keeps them all From 0 to 100000 |
Most in allmax_files |
0 | 0: no limit From 0 to 1000000 |
Seedseed |
1 | The same seed gives the same sample |
Read zips in placeremote_zip |
auto | auto: read only the files kept out of a zip when the server allows it, else download it whole · remote: only in place · download: always the whole zip One of: auto, remote, download |
Licencelicence |
empty | The licence of the data (such as CC-BY-4.0), written with the files |
Creditcredit |
empty | Who made the data and how to cite it, written with the files |
Where it runs
On one of your computers with the EsaSim worker: Any worker.
The worker installs the Python packages it needs when they are missing: requests.
In its code, settings are ctx.params["key"], inputs ctx.inputs["id"] and outputs ctx.output("id", "name.ext"); see Custom blocks.