What are the approaches to remote firmware update on devices and OpenWRT in particular?

Question

Firmware

I am making firmware for wifi routers, which will be installed on devices, devices will be distributed to users. There is a big issue with updating the firmware when a new version of the system is released - there will be hundreds and potentially thousands of devices, so walking around and putting it by hand is not an option.
What are the approaches / methods for remotely updating firmware for devices, is it interesting in general and specifically for openwrt?
How to make sure that all this does not fall and, in case of some problems, rolls back with minimal user participation?

Answer 1

Ну, как-то так плюс Ansible