S
S
swed2014-07-08 16:39:33
Python
swed, 2014-07-08 16:39:33

What technologies to choose for a specific task?

Good afternoon.
It is necessary to write a service with approximately the following functionality:
The database contains addresses that provide some information in xml/json. Several times a day, you need to go through them, parse, leading to a single view, and subsequently distribute them to requesting clients.
Each client has its own list of addresses. The total final weight of what will be given is 0.5-3mb.
I will implement in python.
Most of all, the question of how best to store the final data is of interest:
1. A regular xml file
2. Perhaps you should look towards a database like Redis or Mongo, and return it as json from there
3. Is there any other option?
Well, I’m thinking of doing the project itself on such a stack of technologies:
Database with addresses and users - PostgreSQL
Framework - Tornado
Job queue for parsing - RabbitMQ
I didn't deal with asynchrony before, I worked only with django/flask, so I would like to hear the opinion of more experienced ones, is my choice optimal, maybe there are more interesting options.
Thanks in advance for your replies.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Alexey Cheremisin, 2014-07-08
@leahch

I work (3 years already) with a tornado clone written under twisted - cyclone.io
Everything is the same as in tornado, plus the whole twisted technology stack, which is very convenient. There is no need to be afraid of asynchrony, but the brain will have to be twisted a little. It is very convenient to work with deferred requests like comet/websockets in tornado/cyclone, well, in addition to this, twisted has libraries for working with mongs and rabbits.
Choose the database that is more convenient, you can mongo, you can postgres. And it is logical to store the results in the same database. I am for mongo.
If the result is needed only once, for return to the client, then you can not store it in the database at all, but simply send it to the rabbitmq queue, as the client came for the result - give it away.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question