R
R
ruboss2015-11-20 15:05:02
linux
ruboss, 2015-11-20 15:05:02

How to arrange fast reading of random sections in a 400 GB file?

It is necessary to read blocks of bytes of a given size from the file as quickly as possible.
I created a 400GB file and wrote 4 billion records of 100 bytes into it.
Then I try to randomly get 100 bytes from anywhere in the file.
SSD 400 gb, 56gb ram, linux ubuntu, ext4. The code is minimal in java.
Used the RandomAccessFile class. But for seek each time it does it not from the current location but from the beginning of the file.
The result is 1000 random reads of blocks of 100 bytes in 7 seconds.
How can you speed up random reading?

Answer the question

In order to leave comments, you need to log in

5 answer(s)
N
nirvimel, 2015-11-20
@ruboss

The author should have pointed out right away that this is not a physical server, but an azure microsoft hypervisor .
It explains a lot... No, it explains everything!
And it would be better to remove the word SSD if you yourself did not hold this "SSD" in your hands (and no one held it in your hands, because it's just a virtual machine under a hypervisor).

A
alexxandr, 2015-11-20
@alexxandr

Change language to C

L
lega, 2015-11-20
@lega

But for seek, each time it does it not from the current location, but from the beginning of the file

seek just sets the address and doesn't do any io operations, so it has no effect.
Most likely the SSD is slow, you can check it with tools. Also, when you read only 100 bytes, the minimum block is read from the device itself (4kb, 16kb, ...)

S
Saboteur, 2015-11-20
@saboteur_kiev

"I created a 400GB file and wrote 4 billion records of 100 bytes into it."
This phrase is not very clear. Do you just have a binary file, or a database?
If it's just a binary file, then what does it mean "wrote 4 billion records into it"?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question