How to correctly launch ~2200 objects with a delay, each of which has its own endpoint for the request?

Question

Java

There is an online game that has a market. There are 9 products on the market with different characteristics. A total of 29 unique requests for data on all products for a single country market. There are 74 countries in total. Total 2146 requests.
Requests should be delayed, at least 200-250ms between each. So it will take 7-9 minutes to get everything up and running.
When the application starts, my CollectorManager collects all collectors (2146 pieces) into one set (so that there are no duplicates, in the future I would like to hang additional Subscribers on specific collectors), subscribes to them a service that will write data to the database and starts the collection process data.

PROBLEM.I noticed that for some specific products (country, product type, Country, Industry, Quality) the collectors stop working. At the start, there may be 1-2-3 calls with the expected delay and that's it. Not a single call all night. For other markets they work stably. After stopping the application in debbug, I see that all 2146 collectors are in the set, but some stupidly do not work (there are no logs from them and no data in the database).
Then I started running each collector in the MangerCollector itself because I thought the randomDelay() in executorService.schedule(new MarketplaceTask(), randomDelay(), MILLISECONDS) was not working properly:

CollectorManager.java

@EventListener(ApplicationReadyEvent.class)
public void start() {
  log.info("Let's get party started!");
  Set<MarketplaceCollector> collectorSet = marketplaceCollectorProviders.stream()
    .flatMap(provider -> provider.getCollectors().stream())
    .peek(this::subscribeOfferDBSubscriber)
    .collect(Collectors.toSet());
  collectors.addAll(collectorSet);
  collectors.forEach(this::runWithDelay);
}

private void subscribeOfferDBSubscriber(MarketplaceCollector marketplaceCollector) {
  marketplaceCollector.subscribe(marketplaceOfferDBSubscriber);
}

private void runWithDelay(Collector collector) {
  try {
    long delay = 200;
    Thread.sleep(delay);
    collector.collect();
  } catch (InterruptedException e) {
    e.printStackTrace();
  }
}

MarketplaceCollector.java

@Override
public void collect() {
  executorService.schedule(new MarketplaceTask(), 1, MILLISECONDS);
}

Unfortunately, this did not bring the desired effect. I haven’t worked with multithreading before, so I don’t even know which way to row in order to get it all started.

UPDATE . So, I noticed that the problem appears when some Collector tries to get data from the server. I see the log before the line var responseEntity = restTemplate.exchange(erepublikHost + MARKET_URL, POST, requestEntity, MarketplaceResponse.class); , and no further response. That is, I don’t know whether the request went to the server or not and what returned. It seems that the request has gone further and this task has been put on an eternal pause :(

public MarketplaceResponse getOffers(MarketplaceRequest request) {
        var authentication = authenticationService.getAuthentication();
        var requestEntity = new HttpEntity<>(requestString(request, authentication), headersComposer.getHeaders());
        var restTemplate = restTemplateBuilder.build();
        var responseEntity = restTemplate.exchange(erepublikHost + MARKET_URL, POST, requestEntity, MarketplaceResponse.class);
        return responseEntity.getBody();
}

UPDATE2 : " Resolved " the issue by creating a collection of RestTemplats . At least now I know where to look for the problem. Either I'm catching a deadlock , or there's another reason why restTemplare.exchange() doesn't return a result for some queries. What does it return for others... Even if it's the same RestTemplate .

Answer 1

The `peek` method is an intermediate operation. One of the peculiarities is that such operations are lazy , and you have no guarantee when they will be executed or if they will be executed at all.
The ExecutorPool approach is healthier, but for example in the MarketplaceCollector I didn't see the initialization of `final ScheduledExecutorService executorService`.
In addition, if you have a console application, then when the main thread ends, the JVM will go to complete all the child ones and beat the task, which is scheduled in 3 minutes. Make sure it lives long enough.
The third point may be that the service you are accessing considers such a number of requests with such an interval to be redundant and temporarily stops responding. This also requires verification.