A
A
Areso2018-06-09 09:00:41
Data storage
Areso, 2018-06-09 09:00:41

Where to store the dataset for an open source project?

I have a project. Simplistically, it is divided into 2 parts by functionality: - collects
data
- makes some kind of selection from this data
The project is open source and is available on GitHub /
The spider that collects data into a dataset is slow (400 records per minute), and the dataset itself, after the end of the spider will weigh more than 2 gigabytes.
I have a suggestion that this dataset might be of interest to someone on its own. Perhaps someone will figure out what to do with it, and be able to reuse it, and not collect data for ~ 100 days by my spider again.
Where should I store this information so that it is available to anyone who wants it and does not abuse GitHub?
There are options with clouds (Yandex.Disk, Google Drive, OneDrive, Mail.ru Cloud, Mega), but I have everything packed there (create a new one - they will delete it in a year for not using it), and, in my opinion, this is not the best the best option for storing datasets.

Answer the question

In order to leave comments, you need to log in

5 answer(s)
A
athacker, 2018-06-09
@areso

And why "non-use"? Or the dataset will not be updated by your software?
You can make a script that will upload/delete a small file from the public cloud every few days on a schedule. You can definitely send files to the same Ya.Disk with a curl. Then the activity on the account will be permanent :-)

P
pfg21, 2018-06-09
@pfg21

make a NAS at home from an old computer, stick a screw or two into it, pile up a web ftp on it, etc. server.
NAS to use for personal storage, server.

R
Roman Mirilaczvili, 2018-06-09
@2ord

There are options with clouds (Yandex.Disk, Google Drive, OneDrive, Mail.ru Cloud, Mega), but I have everything packed there (create a new one - they will delete it in a year for not using it), and, in my opinion, this is not the best the best option for storing datasets.
It just makes sense to store and distribute a public dataset in the cloud.

A
andrew-nii, 2018-07-03
@andrew-nii

An immutable dataset for users can be distributed by torrents. fast, simple, hashcheck.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question