H
H
Hanry6542022-02-11 03:21:19
Node.js
Hanry654, 2022-02-11 03:21:19

How to optimize the search for the desired phrase in json objects?

Now it will be absurd, and you will write why I need this, BUT there is a web socket through which an array with data arrives every 2 seconds

[
  {
    "id": 345,
    "descripton": "test follow maybe test test"
  },
  // ... 740 length
]

And my list of phrases, for 1 million lines
[
  "test test test test test",
  "test test test test test",
  "test test test test test",
  "test test test test test",
  "test test test test test",
  "test test test test test"
  // ... 1mln length


When new websocket data arrives, I check it
for (object of wsData) {
  for (myPhrase of myPhrasesList) {
    // Если есть совпадение, вызываю функцию отправки сообщения в тг и отдельный запрос на сайт
  }
}


So, with a load of 1 million lines, sending a message to tg and a separate request to the site takes a lot of time than with a load of 10k, with 10k lines, everything works perfectly.
upd
what I meant, the search itself is fast and the function is also called in a matter of seconds, but the execution of the function is slow, and can take up to 40 seconds

. So, how to optimize all this crap, and so that everything works smartly and nicely? At the same time, so that everything does not fall off under load, because as I understand (well, my guess) the data comes every 2 seconds, and the load grows more and more and this is bad because the past data does not have time to be checked when new ones arrive. Are there solutions?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
Vladlen Hellsite, 2022-02-11
@Hanry654

// Это должно быть вне обработчика закэшировано
const myPhrasesList = new Set([
  "test test test test test",
  "test test test test test",
  "test test test test test",
  "test test test test test",
  "test test test test test",
  "test test test test test"
  // ... 1mln length
]);

for (object of wsData) {
    if (!myPhrasesList.has(object.description)) {
       continue;
    }

    this.notification(object, object.description).then()
}

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question