How to create a sitemap for 5 million pages?

Question

linux

Hey! There is a website with 5 million pages. You need to make a sitemap, you are interested in a server-side solution in the console, preferably a multi-threaded one. All that I found on github is one thread and very slow work. Are there any options?

Answer 1

https://www.sitemaps.org/protocol.html

You can provide multiple Sitemap files, but each Sitemap file that you provide must have no more than 50,000 URLs and must be no larger than 50MB (52,428,800 bytes). If you would like, you may compress your Sitemap files using gzip to reduce your bandwidth requirement; however the sitemap file once uncompressed must be no larger than 50MB. If you want to list more than 50,000 URLs, you must create multiple Sitemap files.

Answer 2

EE so it's generally easy
https://www.mydefile.ru/sitemap.xml

Answer 3

It seems to me that parsing a site using external links is very resource-intensive and crooked.
It is logical to generate a sitemap by hardcoding the site structure / requests to the database, at the same time you can correctly fill in the fields lastmod, changefreq, etc.