A
A
Alons2020-09-01 02:19:03
Python
Alons, 2020-09-01 02:19:03

What to do if you need to send a very long request to api?

Preface = error description

Got more than 8190 bytes (25569123) when reading Status line is too long.

I'm making an api, I need to accept a large json object, process it and return it. But in trying to pass that object to the api as the error suggests, I'm being told that my request is too large. To be honest, I'm at a loss and do not understand what I can do to implement my plan or go in some acceptable alternative way.
api I built on aiohttp. I tried other frameworks but in general the problem is the same, not to mention the accompanying problems of other frameworks ....
PS: Please take into account that I'm a beginner ... if it turns out that I'm asking some nonsense. ..

async def handle(request):
    data = json.loads(request.query['data'])
    brand = json.loads(request.query['brand'])
    ip = request.query['ip']
    final_data = launch_processors(brand, data)
    
    return web.Response(text=json.dumps(final_data, ensure_ascii=False))

Answer the question

In order to leave comments, you need to log in

3 answer(s)
V
Vladimir Korotenko, 2020-09-01
@Alons

It is better to transfer large amounts of data via POST. There is a limit on the size of the request body of several megabytes (depending on the settings, it can be up to 2 gigabytes).

B
bkosun, 2020-09-01
@bkosun

First you need to look at the source:

class HeadersParser:
    def __init__(self,
                 max_line_size: int=8190,
                 max_headers: int=32768,
                 max_field_size: int=8190) -> None:
...

https://github.com/aio-libs/aiohttp/blob/5f0a59fd3...
class HttpParser(abc.ABC):

    def __init__(self, protocol: BaseProtocol,
                 loop: asyncio.AbstractEventLoop,
                 max_line_size: int=8190,
                 max_headers: int=32768,
                 max_field_size: int=8190,
...

https://github.com/aio-libs/aiohttp/blob/5f0a59fd3...

A
Alons, 2020-09-01
@Alons

As a result, what happened ...
I passed the post data but used params = out of ignorance, but I should have data =.

requests.post("http://localhost:8080/find", data=payload)

This certainly immediately increased the amount of data being passed, but, as it turned out, there is still a limiter by default.
It is set to client_max_size=1024**2
Changes to client_max_size=1024**8 were quite enough for the volume I needed.
web.Application(client_max_size=1024**8)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question