Downloaders / Sync
Curl has it. Wget supports it.
aria2c –split=5 –max-connection-per-server=5 http://www.savioko.com/100M.iso
A program to control aria2c xml-rpc in terminal lainme/aria2cmd · GitHub
GUI (The best ones)
RSync & ZSync
ZSync block size
Default is 2K (2048) or 4K (4096), depending on file size. The maximum value allowed was 128KB (both in rsync and zsync).
I will use 65536 (64K). This is not a must, but it’s logical to be most efficient.
RSync Optimized Packers
Option –rsyncable in gzip.
When using with Tar, you can do even this:
GZIP="–rsyncable" tar -czf something.tgz /something
Parallel GZip = Pigz
It also supports –rsyncable parameter.
pigz --best --rsyncable --recursive --keep --blocksize 64
pigz --decompress testing.gz
I will use 64K as block size. Pigz minimal block size is 32K.
BZIP2 & Parallel BZIP2
By architecture (working in blocks), both of them are good for rsyncing.
Block size: bzip2 compresses large files in blocks. The flags -1 through -9 specify the block size to be 100,000 bytes through 900,000 bytes (the default) respectively.
Anything in parallel Parallel xargs, but executes in parallel xjobs utility
Courgette (Google Chrome on Windows uses it): Courgette algorithm is a differential compression algorithm designed for incremental update of Google Chrome. Unlike other diff algorithms such as xdelta, zdelta and bsdiffwhich handle general binary files, the Courgette algorithm was designed to difference executable files supported by x8086 architecture. It achieves much smaller diff size.