Z
Z
Zerstoren2014-02-19 18:58:13
big data
Zerstoren, 2014-02-19 18:58:13

How to organize the storage of large text data?

There is a system for parsing emails for their subsequent submission to the system. Because There may be problems with parsing, we decided to keep the original letters.
Another moment is with attachments in letters, i.e. Letter size can reach up to 50 and 100MB. It is impossible to extract attachments from a letter.
The file system is considered last.
The question arose, what is the best software to use to store this data?
Be sure that he was able to receive files from the network (the software will be on a separate server).
It is desirable that there are some labels for future data retrieval.
It's good if there is a connection with nodejs.
It's great if the data is also archived.

Answer the question

In order to leave comments, you need to log in

4 answer(s)
T
Timur Shemsedinov, 2014-02-20
@MarcusAurelius

The file system is considered last.

Just the file system is the best way to store files. Or do you have a prejudice against her? Not cool enough? Get ZFS. Minimum processing, maximum efficiency, file systems were developed for this.
Why? Do you store them in MIME BASE64? Just what you need - make a folder for each letter and save all attacks from it to a folder. Of course, you need to keep an index of files and letters in the DBMS, with additional fields that are not in the file system. To do this, it is better to take a relational file and store file names there in UTF-8, and you can keep renamed files on disk, for example, in the ID of a record in the database, you can use HEX.

Z
zxmd, 2014-02-19
@zxmd

You can of course try mongo gridfs but... do you want that?

D
Dan Ivanov, 2014-02-19
@ptchol

Elliptics/Casandra

Y
Yaroslav Eremin, 2014-02-20
@YaroslavEremin

Use Mongo

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question