I
I
Ivan2016-08-15 15:58:06
macOS
Ivan, 2016-08-15 15:58:06

How to transfer a large number of small files?

Greetings!
Situation:
There is a folder with a large number (more than 400 thousand) of small files.
The maximum size of a file in a folder is about 3 mb, while the average "in the hospital" is about 100-200 kb per file.
Two questions:
1) How to transfer this folder to an external hdd as painlessly as possible?
The standard system tools (Mac OS) are choking.
They suggested such a thing as pipe. I googled + compiled a command that, it seems, should do what is required:

tar cf - "/Volume1/Folder/TargetFolder" | (cd "/Volume2/Folder/" && tar xf -)
(please check more :)
But it does not show at what stage which process (there are 3 of them, sort of) is.
I want to implement pvit, but it doesn't work. Need help :)
2) How to split a folder of 400 thousand files into several folders of 10 thousand files?
Basically, I need some help :)

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Alexey Cheremisin, 2016-08-15
@johnnywoody

So, if in the terminal this directory can be viewed using ls and it does not fall out with an error, then everything is still fixable.
Well, you don’t need to use tar for such an operation.
Simple enough
yes, and tar in your case can be made simpler (all characters -matter .!!!)
Well, you can split it into a group of files like this.
First, we take a list of all files with the command

cd /Volume2/Folder
find . > myfiles.txt
split -l 100000 myfiles.txt myfiles.

We get several files myfiles.aa myfiles.ab and so on, 100000 lines each.
then apply copying to each file
Tested on OSX :-)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question