How to make robots.txt, sitemap.xml on a forum?

Question

Search Engine Optimization

robots.txt, sitemap.xml how to do it on the forum?

https://programmatron.com/

Answer 1

Usually a text editor is used for this.
Letters are printed in it in the right order and must be saved at the end of the work.
The order of the letters is indicated in the instructions https://yandex.ru/support/webmaster/controlling-ro...

Answer 2

I recommend the simplest and most effective method: crawl the site with a crawler, Screaming Frog SEO Spider is best. Upon completion of the scan, I recommend deleting unnecessary pages and generating a sitemap as shown in the screenshot: https://cln.sh/hehGMn
In the settings, leave Response Codes only at 2xx (see screenshot https://cln.sh/dy1Rx5 ), in Last Modified set the current date and export, it will take a little time, then add the map to robots.txt
. I also have a ready-made online solution for personal needs for splitting the sitemap into categories and by the number of URLs. Knock if you need to.
If you promote the site yourself, subscribe to my author's telegram channel - SEO specialist(I post a lot of relevant first-hand information).