Answer the question
In order to leave comments, you need to log in
What architecture to put in the PhantomJS parser?
Good afternoon, colleagues.
We plan to develop a parser for about 10 sites on casperjs. Now the question arose with the architecture of the entire system.
I briefly describe the process:
1. The server receives a POST\GET request with a set of input data
2. The server processes the input data, makes it suitable for ALL scripts run in casperjs
3. The server must run all 10 parsing scripts synchronously
4. Scripts after successful parsing transfer the processed data back to the server in a unified format
Questions:
item 3 - how to run all 10 scripts synchronously? bash script?
p.4 - how to write the result to mysql as quickly as possible? put json in file then read from file?
The task is to make the most fault-tolerant service + speed should be on top. The average running time of a parsing script is 10 seconds.
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question