Downloaders / Sync

Downloaders / Sync

Metalink support

Curl has it. Wget supports it.

Aria2

All-new 1.16 series debuted - aria2 aria2 - CLI Metalink/BitTorrent Client tatsuhiro-t/aria2 · GitHub

aria2c –split=5 –max-connection-per-server=5 http://www.savioko.com/100M.iso

A program to control aria2c xml-rpc in terminal lainme/aria2cmd · GitHub

GUI (The best ones)

YAAW by binux TheBear | Free Security & Utilities software downloads at SourceForge.net

Aria2c-GUI

Others

mulk.1

RSync & ZSync

zsync cm3/azsync/recursive-zsync at master · outofjungle/cm3 · GitHub cm3/azsync at master · outofjungle/cm3 · GitHub zsync: client-side rsync over HTTP - azure sky

ZSync block size

Default is 2K (2048) or 4K (4096), depending on file size. The maximum value allowed was 128KB (both in rsync and zsync).

I will use 65536 (64K). This is not a must, but it’s logical to be most efficient.

RSync Optimized Packers

GZip

Option –rsyncable in gzip.

When using with Tar, you can do even this:

GZIP="–rsyncable" tar -czf something.tgz /something

Parallel GZip = Pigz

It also supports –rsyncable parameter.

Parallel gzip Pigz – parallel gzip OMG | Kitchen Soap

Usage

Compress:

pigz --best --rsyncable --recursive --keep --blocksize 64

Decompress:

pigz --decompress testing.gz

I will use 64K as block size. Pigz minimal block size is 32K.

BZIP2 & Parallel BZIP2

By architecture (working in blocks), both of them are good for rsyncing.

Block size: bzip2 compresses large files in blocks. The flags -1 through -9 specify the block size to be 100,000 bytes through 900,000 bytes (the default) respectively.

Parallel BZIP2 (PBZIP2)

Others

Anything in parallel Parallel xargs, but executes in parallel xjobs utility

Courgette (Google Chrome on Windows uses it): Courgette algorithm is a differential compression algorithm designed for incremental update of Google Chrome. Unlike other diff algorithms such as xdelta, zdelta and bsdiffwhich handle general binary files, the Courgette algorithm was designed to difference executable files supported by x8086 architecture. It achieves much smaller diff size.

date 01. Jan 0001 | modified 29. Dec 2023
filename: Downloaders & Sync