Find duplicates
Finds copies of the same picture (resized, recompressed, slightly cropped) and, with embeddings, images that show the same thing; keeps one per group.
How it works
Every image gets a perceptual hash (64 bits from the lowest frequencies of its 32 x 32 grey version): copies, resized or recompressed versions and small crops have hashes a few bits apart. Images within Distance bits are one group. With the embeddings of Embed images connected and Similarity set (0.97 for near copies, 0.9 for the same scene), images that look the same to the model join their groups too.
Each group keeps the largest image (then the sharpest, when a quality table is connected) and flags the others as duplicate, with which image was kept.
Inputs
| Input | Takes | What it is |
|---|---|---|
Imagesimages | Files jpg, jpeg, png, webp, bmp, tif | Images, in folders (the folder is the label) Takes several connections. |
Qualityquality | Table csv | quality.csv of Image quality check: the sharper image of a group is kept Optional. |
Embeddingsembeddings | Files npy, json | embeddings.npy and paths.json of Embed images Optional. |
Outputs
| Output | Gives | What it is |
|---|---|---|
Flagstable | Table csv | duplicates.csv: one row per image (path, group, keep, kept_image, flag, reason); flag is empty when the image is fine |
Groupsgroups | Metrics json | duplicate_groups.json: the images of each group |
Live telemetrytelemetry | Live telemetry | What the block sends while it runs, for views such as Chart or 3D view. |
Settings
| Setting | Default | What it does |
|---|---|---|
Distance (bits)distance |
6 | Hashes at most this many bits apart are the same picture From 0 to 20 |
Similarity (embeddings)similarity |
0 | Cosine similarity at or above which two images are the same; 0: hashes only From 0 to 1 |
Where it runs
On one of your computers with the EsaSim worker: Any worker.
The worker installs the Python packages it needs when they are missing: numpy</code>, <code>pillow.
In its code, settings are ctx.params["key"], inputs ctx.inputs["id"] and outputs ctx.output("id", "name.ext"); see Custom blocks.