How to get more performance out of ruby?

Question

Ruby on Rails

There is a task of background processing of a large amount of data:
10 thousand records per second are fed into the input, processing takes place and then all these 10 thousand records are sent to the database.
While I'm making a decision on one machine (the option of several servers is not yet considered).
Now I'm thinking about two options:
1) launching several threads, but then there is a danger of blocking the thread
2) parallel launching of several instances of the handler
Which option will be faster? Has anyone experienced ruby ​​performance gains?

Answer 1

It may well happen that the bottleneck will be the DBMS. Try to simply save this array to the database without processing, and if this does not work out, then it makes no sense to optimize Ruby, you need to look for a more nimble DBMS, I know that, for example, in industry, where the task of saving huge arrays of information from sensors is also used key-value DBMS, which have much more speed for insertion than relational ones.

Answer 2

You can't make Ruby faster. At the very least, a lot of people are constantly working on this, so it’s unlikely that we will succeed here.
Do you have rails or not? What is your ORM? What base? 10000 what records? What kind of processing is going on? What should happen in the database with these 10000 ?
What conditions? Need to fully process the first 10,000 in 1 second and proceed to the next? Or will the queue roll? If queue, then with priority\sequence? management? What is the server resource?
Don't suffer from premature optimization. Set a problem, solve it in the simplest and fastest way. If you run into numbers, then ask yourself questions.
Usually indexing, proper table engine and some https://github.com/zdennis/activerecord-importor a manually composed request solves the problem. If not, and the conditions are not very strict, then queues.
Well, then only the specifics.

Answer 3

https://github.com/resque/resque
Rails 4.2.0 will also have its own queue, so for now you can use resque
and then switch to the built-in queue.
Active Job, ActionMailer #deliver_later
The headline feature for Rails 4.2 is the brand new Active Job framework, and its integrations. Active Job is an adapter layer on top of queuing systems like Resque, Delayed Job, Sidekiq, and more. You can write your jobs to Active Job, and they'll run on all these queues with no changes.