Collections List c# maximum length?

Question

C++ / C#

Good afternoon!

Please tell me how to solve the issue of processing more than a trillion values?

Storage takes place in NoSql LiteDb

If necessary, I extract data and I need to process it, but I wondered if I would have more than a quadrillion data? The server generates packets like all the rules using the uint value is enough for a long time.

But when receiving data from the database, if an overflow occurs and I cannot pull it into the collection, who knows what to do?

At least a hint where to look ...

I save the data to the database

Curr c = p.curr.Dequeue();

                    lock (p.locker)
                    {
                        using (var db = new LiteDatabase(@"MyData.db"))
                        {
                            var col = db.GetCollection<Curr>("currency");
                            col.Insert(c);
                        }
                    }

Requesting data from the database

List<Curr> d;
            lock (p.locker)
            {
                using (var db = new LiteDatabase(@"MyData.db"))
                {
                    ILiteCollection<Curr> currDat = db.GetCollection<Curr>("currency");
                    d = currDat.FindAll().ToList();

                }
            }

Actually a collection
List<Curr> d;

How to bypass overflow or to make page reading who can be knows?

Z. You need to pass the test task in the evening ... :)

Answer 1

When requesting data, do not do ToList - leave it in IQueryable for further selections.
Then it will be possible to make a selection + pagination, through
d.Where(...).Take(...).Skip(...) , whereIQueryable<Curr> d

Answer 2

0) Everything depends on the available RAM.
1) If you want to show them all to the user, then he does not need these trillions of your values, he does not even need a thousand.
2) If possible, do all the data processing logic on the SQL server.
3) Pagination in SQLiteSELECT * FORM t LIMIT сколько, начиная_от_куда

Answer 3

For show: the maximum length is limited by the indexer type, i.e. 2147483647 (int32).

Answer 4

How much memory do you have