M
M
mad_god2015-11-26 23:30:06
Python
mad_god, 2015-11-26 23:30:06

How to increase the speed of reading files from disk?

Reading small (100KB) files from disk in a loop and adding their contents to a list for further processing does not raise hard disk usage above 3MB per second. Is this the limit or can I increase the reading speed?
CrystalDiskMark shows 40 MB, but this is on large files, and small ones, and even fragmented ones, can be read more slowly.
It makes no sense to use several processes, since this only loads the reading head of the hard drive, as I understand it.
But 3 MB? It's somehow implausible.
code like this

while(count<2000):
      try:
        data = open(path+str(num)+".html").read()
        lst.append(data)
      except:
        pass

Answer the question

In order to leave comments, you need to log in

4 answer(s)
R
Roman Kitaev, 2015-11-26
@deliro

This is plausible.

M
mad_god, 2015-11-27
@mad_god

Oh yes) SSD would oh how it would speed up. I compare the IOPS of SSD and SATA-2 hard drives and salivate. But there is no such money. Yes, and I have 370 GB of files.
OS - WIN 7 x64. HDD - Hitachi SATA-2 2TB.

A
asd111, 2015-11-27
@asd111

If there is a lot of RAM - RAMdisk . The 4GB version is free. All data stored on the Ram disk is loaded into RAM, but you can work with them like regular files on your hard drive.
If there are 3-4 thousand rubles. then you can take a 60-120 GB SSD.

A
alpy, 2015-11-28
@alpy

open does not return the contents of the file, but a File Object, so you do not actually read the data from the disk. Change to
lst.append(data.read())

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question