README
¶
memedup
Perceptual hashing and duplicate matching for images and video, in Go.
It does matching only. There is no storage, no database and no network. You keep your own index, decide which candidates a check runs against, and hand them to a match function. That last part matters: what counts as "in scope" for a duplicate check belongs to your application, not to a hashing library.
Install
go get github.com/painhardcore/memedup
Go 1.22 or newer. Video needs ffmpeg and ffprobe on PATH. Images need
neither.
Images
f, err := os.Open("meme.webp")
if err != nil {
return err
}
defer f.Close()
img, err := memedup.Decode(f) // jpeg, png, gif, webp
if err != nil {
return err
}
hash := memedup.PHash(img) // or memedup.DHash(img)
// Candidates come from your own store, already narrowed to whatever scope
// this check should cover.
matches := memedup.MatchImage(hash, candidates, 10)
for _, m := range matches {
log.Printf("%s is %d bits away", m.ID, m.Distance)
}
Two hashes are available and they are not interchangeable. DHash is a 64-bit
difference hash, cheap, with no DCT. PHash is a 64-bit DCT perception hash and
absorbs recompression slightly better. A threshold tuned for one is wrong for the
other, and mixing values from both in one index produces nonsense.
Neither returns an error. Hashing a decoded image has no failure mode, so both panic on a nil image and are otherwise total.
Videos
sig, err := memedup.VideoSignature(ctx, "clip.mp4", 3, memedup.PHash)
if err != nil {
return err
}
matches := memedup.MatchVideo(sig, candidates, memedup.VideoRule{
Threshold: 7,
Quorum: 2,
})
VideoSignature samples frames at proportional offsets i/(count+1), so three
frames give the 25%, 50% and 75% marks and the sample points scale with the clip
instead of sitting at fixed timestamps.
Matching runs in two stages. Length is checked first, with a tolerance of the wider of one second and 5%, which rejects most unrelated clips for free. Frames are then compared best-pair rather than positionally, because an added intro or a trim shifts every sample and a positional comparison would miss the match.
Set Quorum to 2 or more. A quorum of 1 fires on a single shared frame, and one
common intro or one black frame is enough to supply that.
Choosing a threshold
Pick it from your own data. Both hashes throw away high-frequency detail, which for memes is exactly the overlaid text, so two different jokes on the same template can sit a few bits apart. That, not collisions between unrelated pictures, is the false positive you will actually hit.
Treat a match as a candidate rather than a verdict. If you keep human decisions anywhere, plot distance against those decisions and read the threshold off the overlap instead of guessing.
Hash stability
Stored hashes are a wire format. Changing the resize filter, the channel sum or the bit order changes every value the package returns, which silently invalidates an existing index and every decision made from one.
Golden tests pin both formats. DHash is bit for bit identical to the duplicate
detection running in production at openmeme.dev, and
PHash was checked against a live index holding thousands of hashes. If those
tests fail, the code is wrong, not the constants.
PHash calls goimagehash instead of
reimplementing the DCT, for the same reason.
License
MIT. See LICENSE.
Documentation
¶
Overview ¶
Package memedup computes and compares perceptual hashes for images and video.
It does matching only. There is no storage, no database and no network here. Callers keep their own index and choose which candidates to compare against, because what counts as in scope belongs to the application, not to the hash.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DHash ¶
DHash is a 64-bit difference hash. It downscales to 9x8 with a Lanczos filter, converts to grayscale, then sets one bit per horizontal neighbour pair when the left pixel is brighter. Bits fill LSB-first in raster order.
Treat the filter and the bit layout as a wire format. Changing the resize filter, the channel sum, or the bit order changes every value this returns, which invalidates a stored index and every review decision made from one.
func Decode ¶
Decode reads a still image in any format this package supports. The gif, jpeg, png and webp decoders register themselves through the blank imports above.
Do not narrow this to one format. Decoding only JPEG admits every PNG and WebP without a hash, so those files skip duplicate detection and nothing says so.
func PHash ¶
PHash is a 64-bit DCT perception hash.
It panics if img is nil. Hashing a decoded image has no failure mode: the only error the underlying library reports is a nil image, and Decode never returns a nil image with a nil error. Returning an error here would make every caller handle a condition that a correct program cannot produce, so a nil image is treated as the programmer error it is.
It calls goimagehash instead of reimplementing the transform. A reimplementation that looked faithful but differed in the resize filter or the coefficient order would return different values for the same image, which invalidates a stored index.
Types ¶
type ImageCandidate ¶
ImageCandidate is one indexed still image to compare against.
Callers build this slice from their own store, having already applied whatever scope the check runs in, such as a channel, a source, or a publication target. Scope belongs to the application's data, so it never appears here.
type Match ¶
Match is one candidate that came within the threshold, and the distance that let it through. Recording the distance lets callers rank results and calibrate the threshold later against human verdicts.
func MatchImage ¶
func MatchImage(target uint64, candidates []ImageCandidate, threshold int) []Match
MatchImage returns every candidate within threshold bits of target.
This is a recall filter, not a verdict. Images built from a shared template differ only in overlaid text, which is the high-frequency detail both DHash and PHash discard, so a generous threshold pairs template siblings. Verify the results before acting on them.
func MatchVideo ¶
func MatchVideo(target Signature, candidates []VideoCandidate, rule VideoRule) []Match
MatchVideo returns candidates whose sampled frames agree with the target's under rule. Duration is checked first.
Frames are compared best-pair rather than positionally. Samples sit at proportional offsets, so an added intro or a trim shifts all of them and a positional comparison misses the match. The best pair still lines up on the footage the two clips share, and the quorum stops that from firing on a single incidental frame.
type Signature ¶
Signature is what a video is compared on: its length, and one perceptual hash per sampled frame.
func VideoSignature ¶
VideoSignature samples count frames from the video at path and hashes each one.
Frames are taken at proportional offsets i/(count+1), so sample points scale with the clip instead of sitting at fixed timestamps. Three frames give the 25%, 50% and 75% marks.
Requires ffprobe and ffmpeg on PATH. Frames are piped as MJPEG rather than written to temporary files.
Neither command is bounded on its own, so pass a ctx with a deadline unless the caller is happy to wait on a malformed file for as long as ffmpeg is.
type VideoCandidate ¶
VideoCandidate is one indexed video to compare against. Scope is the caller's to apply, as with ImageCandidate.
type VideoRule ¶
type VideoRule struct {
// Threshold is the maximum Hamming distance for one frame pair to count.
Threshold int
// Quorum is how many of the target's frames must find a match. A quorum of 1
// fires on a single shared frame, which one common intro or one black frame
// satisfies. Use 2 or more.
Quorum int
}
VideoRule sets how strict a video comparison is.
Source Files