1. Manuals
  2. Tumbleweed
  3. perl-Cavil-Matcher
  4. Cavil::Matcher(3pm)

Scroll to navigation

Cavil::Matcher(3) User Contributed Perl Documentation Cavil::Matcher(3)

NAME

Cavil::Matcher - Next-generation license pattern matcher for Cavil

SYNOPSIS

  use Cavil::Matcher;
  my $m = Cavil::Matcher::init_matcher;
  $m->add_pattern(1, Cavil::Matcher::parse_tokens('Permission is hereby granted $SKIP30 to deal'));
  my $matches = $m->find_matches('source/file.c');   # [[pattern_id, start_line, end_line], ...]

DESCRIPTION

"Cavil::Matcher" turns source files into license and keyword matches for Cavil <https://github.com/openSUSE/cavil>. It keeps the proven token-hash prefix-tree algorithm of its predecessor, Spooky::Patterns::XS, but stores the compiled patterns as immutable, memory-mapped segments described by a small manifest, so that adding or removing a pattern never rebuilds the whole cache and index workers share one physical copy of the data per host.

The tokenizer and hashing are a frozen C++ core, bit-for-bit compatible with the previous engine; the segment lifecycle is pure Perl (see Cavil::Matcher::Index and Cavil::Matcher::Manifest). For the design and rationale see docs/Architecture.md.

FUNCTIONS

init_matcher

  my $matcher = Cavil::Matcher::init_matcher;

Create a new matching engine (a "Cavil::Matcher::Engine").

parse_tokens

  my $tokens = Cavil::Matcher::parse_tokens($pattern_text);

Tokenize pattern text into the arrayref of token hashes that "add_pattern" expects. A $SKIP wildcard ($SKIP followed by a number n up to 99) is recognised here: each matches from one up to n arbitrary words (at least one, at most n - never a zero-word gap). A pattern may not begin or end with a skip.

Input must be text. This and "normalize" take the string with C-string semantics, so an embedded NUL byte terminates the input (everything after it on that call is ignored). That is fine for their intended use - curated pattern text and human-readable text - and matches the previous engine. To scan raw source bytes (which may contain NULs), use "add_pattern" + "find_matches", whose file reader is NUL-tolerant and reads past embedded NULs.

normalize

  my $rows = Cavil::Matcher::normalize($text);   # [[line, token, hash], ...]

Tokenize text, returning each token with its line number and hash. Text input only - see the NUL-handling note under "parse_tokens".

distance

  my $edits = Cavil::Matcher::distance($norm_a, $norm_b);

An approximate token-level edit distance between two "normalize" results. Note: for byte-for-byte parity it deliberately reproduces the previous engine (Spooky::Patterns::XS), including its off-by-one (it compares count-1 tokens), so it is not a strict Levenshtein distance and can report 0 for inputs that differ only in a single or trailing token. It is currently unused within Cavil; once the previous engine is retired it can be made a strict Levenshtein.

read_lines

  my $rows = Cavil::Matcher::read_lines($file, \%wanted_lines);   # [[line, value, text], ...]

Return the requested lines of a file as raw (undecoded) bytes. Note: %wanted_lines is consumed - each line found is deleted from the hash (an early-exit optimization), so pass a fresh hash if you need to reuse it.

The returned text for a single physical line is capped at 1 MiB, to bound memory on pathological input (e.g. a minified or binary file that is one enormous line). "find_matches" numbers such a line the same way, so a match reported past the first 1 MiB of a single line can point at text this call does not return. Real, line-wrapped source is unaffected; this only matters for degenerate multi-megabyte lines.

init_hash

  my $hash = Cavil::Matcher::init_hash($seed1, $seed2);

Create a streaming 128-bit content hash (a "Cavil::Matcher::Hash").

init_bag_of_patterns

  my $bag = Cavil::Matcher::init_bag_of_patterns;

Create a tf-idf "closest match" model (a "Cavil::Matcher::Bag").

Cavil::Matcher::Engine

The matching engine returned by "init_matcher".

  $engine->add_pattern($id, $tokens);

Add a pattern (from "parse_tokens") to the in-memory delta segment.

  my $matches = $engine->find_matches($file);

Scan a file and return the resolved matches as "[[pattern_id, start_line, end_line], ...]".

Write the in-memory patterns to a compiled segment file / replace the engine's state with a single mmapped segment file. "dump" fully verifies the compiled segment before publishing it, so a bad segment is never written.

"load" (like "attach") validates the segment's structure but does not re-checksum its whole payload: a compiled segment is an immutable, already-verified cache, and re-checksumming hundreds of megabytes on every open is almost the entire cost of a load. Structure is always validated, so a malformed file is still rejected and scanning is always memory-safe; only on-disk corruption of an otherwise well-formed file goes undetected here (use "verify" for that).

Memory-map an additional compiled segment into the active set. Returns false (without dying) on a missing or structurally invalid file. Trusts the payload checksum like "load".
Fully check a segment file on disk — structure and whole-payload checksum — and return true iff it is intact. This is the corruption check that the scan path ("load"/"attach") deliberately skips for speed; run it on demand (an fsck for a cache directory) when you need to prove a segment has not rotted on disk.
Drop these pattern ids from results before overlap resolution.
Record, and read back, the manifest generation this engine was built from. Reading it back from the engine itself (rather than re-reading the index) is race-free, so a report can record exactly the generation it scanned with even if the index is updated concurrently.

Cavil::Matcher::Hash

Streaming content hash from "init_hash": add($bytes), "hash128" ("[hi, lo]"), "hex" (32-char string) and "hash64".

Cavil::Matcher::Bag

tf-idf closest-match model from "init_bag_of_patterns": set_patterns(\%id_to_text), "best_for($text, $count)", dump($file), load($file) and verify($file). "dump" and "load" return true on success; a failed "load" (missing/truncated/malformed file) leaves the model unchanged. Like the engine, "load" validates structure but trusts the payload checksum of this immutable, already-verified cache; "verify" is the on-demand full check (structure and CRC) for an fsck.

SEE ALSO

Cavil::Matcher::Index, Cavil::Matcher::Manifest, <https://github.com/openSUSE/cavil>.

COPYRIGHT AND LICENSE

Copyright (C) SUSE LLC. This is free software, licensed under GPL-2.0-or-later.

2026-08-05 perl v5.44.0