REQUIREMENTS¶

• The ability to compile XS extensions.

  This means a working C compiler, a linker, a "make" program etc. If you built perl from source you will have these already and they will be used automatically. If your perl was built in some other way, for example you may have installed it using your Operating System's packaging mechanism, you will need to ensure that the appropriate tools are installed.

• Module "File::Find"

  This is a core module now for a while.

OPTIONAL MODULES¶

• Test::More

  Required if you want to run Code::DRY's own tests.

• Test::Output

  Optional if you want to run Code::DRY's own tests.

"scan_directories($minlength, $ignoreContent, $regexAccept, $regexIgnore, @array_of_pathnames_of_directories)"¶

Scans the given directories in @array_of_pathnames_of_directories recursively for file names matching the regexp $regexAccept, if it is defined. If those file names also do not match against the regexp $regexIgnore (unless $regexIgnore is undefined) they are included in the analysis.

If $regexAccept and $regexIgnore both are "undef", all file names will be considered for analysis.

If either of $regexAccept and $regexIgnore is not a ref of type 'Regexp', it is expected to be a pattern string that will be converted into a regexp with "qr{}xms".

The parameter $ignoreContent can be used to avoid duplication reports for content matching this regex. If $ignoreContent is not a ref of type 'Regexp', it is expected to be a pattern string that will be converted into a regexp with "qr{}xms".

The parameter $minlength is interpreted in units of lines when being positive. Otherwise its absolute value is interpreted in units of bytes.

All repetitions with a minimum length of $minlength will be reported by the "report" callback function.

"find_duplicates_in($minlength, $ignoreContent, @array_of_pathnames_of_files)"¶

Reads files for the given file names composing a long string, which is then analysed for repetitions.

The parameter $minlength is interpreted in units of lines when being positive. Otherwise its absolute value is interpreted in units of bytes.

The parameter $ignoreContent can be used to avoid duplication reports for content matching this regex. If $ignoreContent is not a ref of type 'Regexp', it is expected to be a pattern string that will be converted into a regexp with "qr{}xms".

All repetitions with a minimum length of $minlength will be reported by the "report" callback function.

"set_reporter(sub{ CODE BLOCK })"¶

Set custom code to report duplicates of a code fragment. The callback is invoked with position information for the copies found during analysis.

The supplied code has to accept two scalars and an array reference.

The first parameter is the required minimum length of duplicates to be reported.

The second parameter contains a string describing the units for minimum length ('lines' or 'bytes').

The referenced array (third parameter) contains one entry with an anonymous array reference for each copy found. Copies are reported in the order of the processing of the files and then in the order of positions. Each copy is represented by this position information as an array entry:

1. filename

2. line number at start of copy (starting with 1). This is the line number of the first line completely contained in the copy.

3. line number at end of copy. This is the line number of the last line completely contained in the copy.

4. offset from start of file at start of copy (starting with 0) clipped to the next completely contained line.

5. offset from start of file at end of copy clipped to the last completely contained line.

6. offset from start of file at start of copy (starting with 0) (used in 'bytes' mode)

7. offset from start of file at end of copy (used in 'bytes' mode)

The default reporter is like this:

    XXX insert code when stable

"set_default_reporter"¶

Resets the reporter callback function to the default shown above.

"report_dupes($minlength, $copies, $length, $index_in_suffix_and_lcp_array)"¶

This function builds a data structure with position information for the duplication copies described by the input parameters. The entries in the suffix array from $index_in_suffix_and_lcp_array to "$index_in_suffix_and_lcp_array + $copies -1" will give the offsets in the string where the copies start. Each has a length of $length characters. With these values file names and line numbers are retrieved and stored in the structure. Then the reporter callback function is called with the minimum length of this scan $minlength and this structure. See also function set_reporter().

enter_files($ref_to_array_of_pathnames_of_files)¶

Reads the files given by the pathnames. Any files with length zero are skipped (and removed from the filename array). Offset arrays for file and line end positions are built. The content of all files is concatenated. Currently the content must not be valid Perl (but this might change when parsing gets involved in a future release).

build_suffixarray_and_lcp($textstring_to_analyse)¶

The XS routine calls the sais-lite function to construct the suffix and longest common prefix arrays for the complete string to analyse.

clip_lcp_to_fileboundaries(@array_with_endoffset_positions_of_files)¶

The XS routine limits lcp values that cross files according to given file end positions. The accumulated end positions have to be sorted in ascending order. Internally a binary search is done in order to find the right file end position. @Code::DRY::fileoffsets contain the needed offset when "enter_files" has been used before.

"reduce_lcp_to_nonoverlapping_lengths"¶

The XS routine limits lcp values that overlap with the preceeding entry. This is necessary to avoid overlap of the reported duplicates.

offset2fileindex($offset)¶

This function uses binary search to get the index of the respective file entry for this offset.

offset2filename($offset)¶

This function uses binary search to get the index of the respective file entry for this offset. It returns the filename for this entry.

offset2line($offset)¶

This function uses binary search to get the index of the respective file entry for this offset. Then it again uses binary search to get the max offset of the respective line and returns the line number for this entry.

"offsetAndFileindex2line($offset, $fileindex, \$nextcompleteLine, \$lastcompleteLine)"¶

This function uses binary search to get the max offset of the respective line belonging to file $fileindex and returns the line number for this entry.

If the third parameter is defined, it is expected to be a reference to a scalar. It will be set to the line number of the next line unless $offset points to the start of a line. Then it will be set to the current line number.

If the fourth parameter is defined, it is expected to be a reference to a scalar. It will be set to the line number of the previous line unless $offset points to the end of a line or there is no previous line. Then it will be set to the current line number.

"clearData"¶

This function clears all resources that were used for a scan.

get_len_at($index)¶

This XS function returns the lcp value at index $index or ~0, if the index is out of range.

get_offset_at($index)¶

This XS function returns the offset value from the suffix array at index $index or ~0, if the index is out of range.

get_isa_at($offset)¶

This XS function returns the index number from the suffix array where the $offset is found or ~0, if the offset is out of range.

set_lcp_to_zero_for_shadowed_substrings($index)¶

This XS function sets all prefix lengths to zero for those entries where the suffix is contained in another longer (or more leftward) suffix.

"get_concatenated_text($offset, $length)"¶

This function returns the given text portion of the internal concatenated string at offset $offset with a length of $length.

get_line_offsets_of_fileindex($fileindex)¶

This function returns the array reference of the line end offsets for the file at index $fileindex.

"get_next_ranked_index" not yet implemented¶

This XS function returns the next index number of the sorted lcp values or ~0, if there are no more entries left.

"reset_rank_iterator" not yet implemented¶

This XS function resets the iterator of the sorted lcp values.

"get_size"¶

This XS function returns the size of string (in 8-bit characters) used by the "build_suffixarray_and_lcp" function.

"get_lcp"¶

This XS function returns a reference of a copy of the lcp array from the "build_suffixarray_and_lcp" function.

"get_sa"¶

This XS function returns a reference of a copy of the suffix array from the "build_suffixarray_and_lcp" function.

"__get_text"¶

Internal function

"__free_all"¶

Internal function

Output messages¶

Duplicates are reported (as per default reporter) in the following format:

        1 duplicate(s) found with a length of 8 (>= 2 lines) and 78 bytes reduced to complete lines:
        1.  File: t/00_lowlevel.t in lines 57..64 (offsets 1467..1544)
        2.  File: t/00_lowlevel.t in lines 74..81 (offsets 1865..1942)
        =================
        ...<duplicated content>
        =================

Error messages¶

This module can die with the following error messages:

• "cannot open file $file: $!\n";

  The opening of a file for read access failed.

• "Error building suffix array:$!\n"

  The XS code could not allocate enough memory for the combined file content.