About¶

LZO is an algorithm that de/compresses data. It is tuned for speed (especially decompression speed) and trades the size of the compressed file for it to some degree. There are variants with slow compression (yet still very fast decompression) available though. See the algorithm parameter below.

This plugin has been written for dd_rescue and uses the plugin interface from it. See the dd_rescue(1) man page for more information on dd_rescue.

Compression or decompression¶

The lzo dd_rescue plugin (subsequently referred to as just ddr_lzo which reflects the variable parts of the filename libddr_lzo.so) choses compression or decompression mode automatically if one of the input/output files has an [lt]zo suffix; otherwise you may specify compr[ess] or decom[press] parameters on the command line.
The parameter opt[imize] will tell ddr_lzo to do an optimization pass after compression. This might speed up decompression by a few percent when creating compressed data with high compression levels and large block sizes.

The plugin also supports the parameter bench[mark] ; if it's specified, it will output some information about CPU usage and resulting compression or decompression bandwidth. (For small files, the numbers become meaningless due to jitter and limited time resolution -- ddr_lzo will skip the output if the numbers are very tiny.)

De/compression algorithm¶

The lzo plugin supports a number of the (de)compression algorithms from liblzo2. You can specify which one you want to use by passing algo=XXX , where XXX can be lzo1x_1, lzo1x_1_15, lzo1x_999, lzo1x_1_11, lzo1x_1_12, lzo1y_1, lzo1y_999, lzo1f_1, lzo1f_999, lzo1b_1 ... lzo1b_9, lzo1b_99, lzo1b_999, lzo2a_999. Pass algo=help to get a list of available algorithms. Consult the liblzo documentation for more information on the algorithms. Note that only the first three are supported by lzop (it can decompress the first five though, as they're all handled by the same decompression routine).
The default (lzo1x_1) is a good choice for fast compression and very fast decompression and ensures compatibility with lzop. For higher compression you might want to chose lzo1x_999, which is very slow but lzop compatible or lzo2a_999, which is twice as fast, but not compatible with lzop. Note that it's better to use other compression algorithms (e.g. lzma with a low preset) if you are interested in good compression, so there's rarely a reason to change the default algo, as you can get marginal gains only in exchange for signifiantly higher CPU and memory ocnsumption with LZO.

Debugging¶

The debug flag will cause the ddr_lzo to output information about blocks and other internal data. It's meant for debugging purposes.

Finally there is also a flags=XXXX parameter. This sets the flags field in the header (default is 0x03000403) and is used for testing only. It is not sanity checked and you can easily set values that will break decompression or cause ddr_lzo to abort. Really only use for development purposes when you know meaning of the various bits.

Error recovery¶

On compression, when input bytes can't be read, ddr_lzo will encode holes in the compressed output file -- these will be skipped over on decompression.

On decompression, erroneous blocks can be detected by the checksums (most often) or by the decompressor. The lzo plugin tries to continue in that case if the block header that specifies de/compressed lengths is intact. It will then result in a block being skipped over (hole) and the decompression will be continued with the next block. This avoids corrupt data to end up in the output file (or preexisting, potentially good data there being overwritten).
The behaviour can be modified by specifying the nodisc[ard] option. When given, the decompressor's output (filled up with zeros if too short for the block) will be written to the output file. Even if we know that the data is incorrect, with some luck, parts of the block may actually be valid.

When the block headers are corrupt, your situation is desperate, as you will have lost the remainder of the file. To recover pieces after such a block header corruption, ddr_lzo supports the search option. With it, the plugin will search the input file (starting from the position given in dd_rescue with -s) for data that looks like a block header and if a valid looking header is found, it will start decompressing from that position. (If you can't find the data you look for, you might actually study the output generated with the debug flag.)

Blocksize considerations¶

When decompressing, the (soft) block size chosen in dd_rescue must be sufficient (at least half the size of the blocksize used when compressing); if you chose too small blocks, ddr_lzo will warn and exit.
For compression, the chosen (soft)blocksize in dd_rescue will determine the size of blocks to be fed to the lzo??_?_compress() routines. Larger block sizes will typically result in slightly better compression ratios, though the returns on increasing the block size quickly diminish after 64k.
The default from dd_rescue (128kiB) is a good choice. It is NOT recommended to increase the block size too much -- when an lzo file gets corrupted, at least one block will be lost; larger blocks result in larger damage. Also, blocks larger than 16MiB will not work well with the error tolerance features of ddr_lzo. Also note that blocks larger than 256kiB need recompilation of lzop if you want to be able to use lzop to process the .lzo files; blocks larger than 64MiB prevent decompression even with a recompiled lzop.

Maturity¶

The plugin is new as of dd_rescue 1.43. Do not yet rely on data saved with ddr_lzo as the only backup for valuable data. Also expect some changes to ddr_lzo in the not too distant future. (This should not break the file format, as we're following lzop ....)
Compressed data is more sensitive to data corruption than plain data. Note that the checksums (adler32 or crc32) in the lzop file format do NOT allow to correct for errors; they just allow a somewhat reliable detection of data corruption. (Ideally, a 32bit checksum just misses 1 out of 2^32 corruptions; on small changes, crc32 comes a bit closer to the ideal than adler32. You may pass the crc32 option to use crc32 instead of adler32 checksums at the expense of some speed -- unfortunately the crc32 polynomial for lzop/gzip/... is not the crc32c polynomial that has hardware support on many CPUs these days.) Also note that the checksums are NOT cryptographic hashes; a malicious attacker can easily find modifications of data that do not alter the checksums. Use MD5 or better SHA-256/SHA-512 for ensuring integrity against attackers. Use par2 or similar software to create error correcting codes (Reed-Solomon / Erasure Codes) if you want to be able to recover data in face of corruption.

Security¶

While care has been applied to check the result of memory allocations ..., the decompressor code has not been audited and only limited fuzzing has been applied to ensure it's not vulnerable to malicious data -- be careful when you process data from untrusted sources.