Show an example of warming up the cache for your project?

Question

linux

Hello.
I have a web application which contains many unique pages. As often happens, a 10% pool of the most popular pages quickly formed, and it is these pages that are famously processed by memcached. The problem is that the project is dedicated to a very narrow area of ​​commerce and each visit is really very important and I want everything to load quickly even when accessing less popular pages. It is not yet pleasant to see sanctions from Google and Yandex about the fact that the site is slow.
In the application engine itself there is no way to "send all pages to the cache" and I'm trying to find an example script or sequence of actions to be able to load all the pages of the project into the cache at any time. Can you tell us how you do it on your projects and perhaps the pros / cons of such a solution?

Answer 1

It's simple:
1 Take your access.log for a week, for example
2 make the top 100 or 1000 most popular requests
3 remove unnecessary requests - form login, captcha, etc.
4 what is left you write down as a list of links that can be given to curl
5 run for some time with a curl cycle through the list of links from point 4
6 profit!

Answer 2

Look in this direction for search keys in Google "curl warm up script url"
https://gist.github.com/jaseclamp/27c4b74e535ddd749bbe
In principle, they are similar for any platform. For example, I warm up Sharepoint by hanging the script to restart the server. After any reboot, the addresses of portals and reports are called.
Just pay attention to the authorization and download of all content.

Answer 3

It would be nice to find out what exactly slows down - the processor or the disk. Perhaps the easiest thing to do is to upgrade your computer or optimize your CMS.
And so - you were rightly advised to simply download the entire site.