Getting 429 status_code even though I sleep(30) before every request and use a proxy in every thread?

Question

Python

Hello!

Wrote a script for parsing some data from instagram.
Since many posts need to be processed at once, to parse each post, I create a thread inside which requests to https://www.instagram.com/graphql/query/ are made using requests.Session. For each thread, an individual public proxy is used (from proxy import Proxy), also, before making the next request, the thread falls asleep for 25 - 30 seconds, but everything is useless, in the middle of the script, I always get a response with a status of 429 - TooManyRequests, although with each proxy is made no more than 2 requests per minute.

Moreover, everything looks as if requests.Session ignores the passed proxy and knocks from my IP - for example, I have 20 threads created, as soon as the first thread receives 429 TooManyRequests, the rest of the threads immediately receive the same response, with the same status code !

Here is a piece of code:

proxies = get_proxy_list()
    random.shuffle(proxies)

    idx_counter = 0
    for code in shortcodes:
        proxy = proxies[idx_counter]
        idx_counter += 1

        session = requests.Session()
        session.proxies.update(proxy)
        humanize_session(session)

        threading.Thread(target=new_load_data, args=(session, code, )).start()

In the new_load_data method, only the following code is significant:

sleep(random.randint(25, 30))
print(f'Запрос для {code}')
response = session.get(QUERY_ROUTE, params={'query_hash': QUERY_HASH,
                                            'variables': json.dumps(load_data_vars)})

What am I doing wrong?

Answer 1

This is not enough .

Answer 2

1. you can check through any public checker which IP the site sees.
2. Perhaps your cookies are rummaging between all the threads, and you are stupidly fired at them :)