What is the best way to implement this architecture?

Question

Software design

I want to write my own content aggregator in go. The architectural idea is as follows: when a request is made to the server from a specific user, a python script is included that parses content from sources and updates the database, then the user is given the result in the form of updated content.

So, how can this be done? Namely, how to run a python script when requesting content?

Answer 1

When the user waits for the aggregator to do its job, this is not good. It is more correct that the first one wants some content and can immediately get it, straight from the database, and ideally from the cache of frequently requested content.
To do this, the aggregator needs to regularly perform its work, regardless of visitors, in the background. Of course, the aggregator pipeline must know in advance where to get the content from or get a list of sources at the beginning, go through them and enter the data into the database. Then regularly update from already known sources, regardless of the entry of users.
Well, if the database is empty and you desperately need to give out content, then there remains a bad option - to let the user wait until the content is downloaded, processed by the aggregator and received back. In this case, when trying to get content, you can display a message that says, "Sorry, ss, come back a little later" or "Wait 5 seconds, I'm fast, fast." And when the fresh content has already been processed, then send back the content via SSE / WebSocket. Or short polling is just that the client will periodically make requests to the API with the hope of getting content.
Here you can read about the ways of interaction
. Therefore, I see such an architecture when working with a user:

Client -> API -> cache/DB (read)
             \
              MQ
                 \
               [Aggregator]
                      \
                      DB (write)

Here, [Aggregator] can mean both a monolithic download-processing mechanism, where everything is in one, and a microservice architecture.
For me, writing business logic in Go is not very convenient and it is better to implement it in higher-level languages. So in Go I would implement a mechanism for downloading content and extracting the necessary parts, and in Python / Ruby / Perl, etc. - the logic of the aggregator itself (mixing, compositing content).