W
W
web_dev2021-07-30 14:36:17
Performance evaluation
web_dev, 2021-07-30 14:36:17

Apache Airflow benchmark for simple frequently DAGs/tasks?

Taurus Hello, I am considering Apache AIrflow as one of the components in my project.
Can't find an approximate benchmark. Therefore, I have a number of questions.
Let's take for example a car with 8 cpu and 32 GB of RAM.
And Doug with such tasks. "curl REST" => "extract content" => "save to DB" => "send email"

- Is Apache AIrflow suitable for frequent tasks that will be executed every second?
- How many parallel tasks can be executed by such a computer per second?
- Approximately how much memory does such a DAG require?
- Will Apache AIrflow do well if for example 1000 Dags are called every 2-3 seconds?

Share your experience. Thank you!

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question