How to work with a shared list in a multi-user application?

Question

Python

I wrote a simple example (both files have the same code, only different ID), if you run both programs, everything works, but...

1.py

import os
import pickle
import random
import time

if not os.path.isfile('file'):
    with open('file', 'wb') as file:
        pickle.dump([], file)
    file.close()

ID = 1

while True:
    f = open('file', 'rb')
    li = pickle.load(f)
    f.close()
    print(li)
    x = random.randint(100, 999)
    for i in li:
        if i[0] == ID:
            i[1] = x
            break
    else:
        li.append([ID, x])

    f = open('file', 'wb')
    pickle.dump(li, f)
    f.close()
    time.sleep(1)

2.py

import os
import pickle
import random
import time

if not os.path.isfile('file'):
    with open('file', 'wb') as file:
        pickle.dump([], file)
    file.close()

ID = 2

while True:
    f = open('file', 'rb')
    li = pickle.load(f)
    f.close()
    print(li)
    x = random.randint(100, 999)
    for i in li:
        if i[0] == ID:
            i[1] = x
            break
    else:
        li.append([ID, x])

    f = open('file', 'wb')
    pickle.dump(li, f)
    f.close()
    time.sleep(1)

... my gut tells me that this should not be done .. with a large number of users and with a large list, the data will partially overwrite each other (especially until the packet reaches the client to the server and back). I do not ask for a ready-made solution, just explain briefly how this is solved? What should be studied?

Answer 1

The entry point to the application (the python script being launched) must be one: when entering or starting, the user enters his id (or it is generated based on some indirect signs). A database must be used to store data. Moreover, given that you immediately wanted a multi-user application, you need to raise a full-fledged database server (simple sqlite will not work).
If you are not familiar with sql, I recommend starting with getting to know mongodb and how to interact with it through python. If you are still familiar, then it would be better to raise a mysql / postgress / or whatever you know there, arm yourself with the pyodbc library and begin to comprehend the world of full-fledged industrial programming.
UPD:having answered, I decided to investigate whether it is possible to run the standard sqlite3 in multi-process mode - it turned out there is a crutch.
Connecting to the database blocks other connections and they wait until the file is freed with a timeout delay. so if connections are constantly opened and buried, then several processes can work with one base at once (theoretically, you can put it in a picke connection, but this is an evil idea).
Here's the code, in theory, should cope with a couple of dozen users (and most importantly: no server):

import sqlite3
import os
import sys
import random
import time
from datetime import datetime

TIMEOUT_DELAY=10

CURRENT_DIRECTORY = os.path.normpath(
    os.path.relpath(
        os.path.dirname(__file__),
        os.getcwd())
    )
DB_PATH = os.path.normpath(os.path.join(CURRENT_DIRECTORY, './db/lists.db'))



def initBase():
    with sqlite3.connect(DB_PATH) as connect:
        cursor = connect.cursor()
        cursor.execute("""
CREATE TABLE lists(
    userid INTEGER primary key,
    value INTEGER
)
""")
        connect.commit()



if not os.path.exists(DB_PATH):
    os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
    initBase()


ID = None
if len(sys.argv) > 1:
    ID = int(sys.argv[1])
else:
    ID = int(datetime.now().timestamp() * 1000) % 10000 + 10000
    print(f'ID не передан. сгенерирован идентефикатор {ID}')


while True:
    with sqlite3.connect(DB_PATH, timeout=TIMEOUT_DELAY) as connect:
        cursor = connect.cursor()
        cursor.execute("""SELECT userid, value FROM lists""")
        print(cursor.fetchall())
        x = random.randint(100, 999)
        cursor.execute("""
INSERT INTO lists(value, userid)
VALUES (?, ?)
ON CONFLICT(userid) DO UPDATE SET
value=?
WHERE userid=?
""", [x, ID]*2)
        connect.commit()
    time.sleep(1)

opened 3 consoles and ran
python3 __main__.py
python3 __main__.py 1
python3 __main__.py 2
seems to work):

But it’s better not to do this anyway, but still use a normal database server