How to quickly pack small files?

Question

linux

Good evening.
Help to deal with the situation:
There is a directory in which there are a huge number of small files.
How can I pack these files as quickly as possible?
With the expectation that the files accumulated over the year are archived in tar.gz in 3 days

Answer 1

If the performance of archiving rests on the processor, then I recommend the pigz archiver , it can use all the processor cores.

Answer 2

7zip rar and compression and multithreading.

Answer 3

If speed is important, then you need to select the fast compression mode of a specific algorithm, or initially select fast compression algorithms, such as LZ4, LZOP, ZSTD. In the latter, the compression ratio is high relative to algorithms of this class.
And in what container to pack, TAR (preferably), ZIP (if without compression) - is not so important.
That is, I would suggest TAR + LZ4 if you need the highest write speed to the drive and a combination of TAR + ZSTD if you want to achieve a compression ratio close to GZIP.
Alternatively, deduplication programs such as bup, zpaq, BackupPC can be used instead of classic archivers. They will allow you to save only changes in files, without duplicating information with each impression of information.