Esacode

Docs / Blocks / Processing

Find duplicates

Finds copies of the same picture (resized, recompressed, slightly cropped) and, with embeddings, images that show the same thing; keeps one per group.

Runs on a worker (Python) Processing Version 1.0 Code you can open and adapt

How it works

Every image gets a perceptual hash (64 bits from the lowest frequencies of its 32 x 32 grey version): copies, resized or recompressed versions and small crops have hashes a few bits apart. Images within Distance bits are one group. With the embeddings of Embed images connected and Similarity set (0.97 for near copies, 0.9 for the same scene), images that look the same to the model join their groups too.

Each group keeps the largest image (then the sharpest, when a quality table is connected) and flags the others as duplicate, with which image was kept.

Inputs

InputTakesWhat it is
Images
images
Files
jpg, jpeg, png, webp, bmp, tif
Images, in folders (the folder is the label) Takes several connections.
Quality
quality
Table
csv
quality.csv of Image quality check: the sharper image of a group is kept Optional.
Embeddings
embeddings
Files
npy, json
embeddings.npy and paths.json of Embed images Optional.

Outputs

OutputGivesWhat it is
Flags
table
Table
csv
duplicates.csv: one row per image (path, group, keep, kept_image, flag, reason); flag is empty when the image is fine
Groups
groups
Metrics
json
duplicate_groups.json: the images of each group
Live telemetry
telemetry
Live telemetryWhat the block sends while it runs, for views such as Chart or 3D view.

Settings

SettingDefaultWhat it does
Distance (bits)
distance
6 Hashes at most this many bits apart are the same picture
From 0 to 20
Similarity (embeddings)
similarity
0 Cosine similarity at or above which two images are the same; 0: hashes only
From 0 to 1

Where it runs

On one of your computers with the EsaSim worker: Any worker. The worker installs the Python packages it needs when they are missing: numpy</code>, <code>pillow.

In its code, settings are ctx.params["key"], inputs ctx.inputs["id"] and outputs ctx.output("id", "name.ext"); see Custom blocks.

More in Processing