Esacode

Docs / Blocks / Processing

Curate and split

Removes the images the checks flagged (or a person rejected), balances the classes and splits train and validation, with a before/after report.

Runs on a worker (Python) Processing Version 1.0 Code you can open and adapt

How it works

Connect the images and the tables of the checks (quality, duplicates, labels: any table with path and flag). Every flagged image is removed, unless Remove names only some flags (such as duplicate, broken, blurry). A Review table (path, decision accept or reject, and optionally a new label) overrides the checks: accepted images stay, rejected ones go, relabelled ones move.

The rest is split into train and validation by class (Validation %), at most Most per class each (0: all), in a repeatable way (Seed), as train/<class>/… and val/<class>/… with manifest.csv. The report gives the images in, removed by reason, kept, and the classes before and after (curation.md, curation.json and CSV tables for charts); removed.csv lists every image removed and why.

Licence and credit files travel with the images.

Inputs

InputTakesWhat it is
Images
images
Files
jpg, jpeg, png, webp, bmp, tif
Images, in folders (the folder is the label) Takes several connections.
Flags
flags
Table
csv
Tables with path and flag (quality, duplicates, labels) Optional. Takes several connections.
Review
review
Table
csv
path, decision (accept or reject), optionally label Optional. Takes several connections.

Outputs

OutputGivesWhat it is
Curated
files
Filestrain/<class>/… and val/<class>/… with manifest.csv
Removed
removed
Table
csv
removed.csv: path, label, flags, reasons
Report
report
Table
md, json, csv
curation.md, curation.json, curation_by_reason.csv, curation_by_class.csv
Live telemetry
telemetry
Live telemetryWhat the block sends while it runs, for views such as Chart or 3D view.

Settings

SettingDefaultWhat it does
Remove
remove
empty Flags that remove an image, separated by commas (duplicate, broken, blurry, too_dark, label, outlier); empty: every flag
Validation (%)
val_pct
20
From 0 to 90
Most per class
max_per_class
0 0: all
From 0
Layout
layout
split/class
One of: split/class, as it was
Seed
seed
1

Where it runs

On one of your computers with the EsaSim worker: Any worker.

In its code, settings are ctx.params["key"], inputs ctx.inputs["id"] and outputs ctx.output("id", "name.ext"); see Custom blocks.

More in Processing