Linux, (non)block devices, what are the potential pitfalls?

Question

linux

linux has the ability to create logical partitions in files. And in the future, use them like regular logical partitions located on a block device.
For example: after installing the OS, many settings have been made and / or subsequent resizing of partitions is not possible. It is required to physically limit the size of the directory to 3TB. We create a file.hdd with a size of 3TB. Next, mkfs.ext4 / file.hdd and mount it. Everything is working. The size is limited.
Or, for example, we can make 4 files of 1 TB. and mdadm combine them into RAID5 and get the same 3TB but in 4 files.
Question: what pitfalls can we catch using files as storage devices? In all instructions, this method is recommended only for experiments, but not for production. And why?" is not explained.

Answer 1

"Why" is a very simple answer. Slowly. 1.5-2 times less iops than a normal disk.
At the same time, there will no longer be such an answer to the question "why" on ssd. And there such advice is a consequence of the inertia of thinking.

Answer 2

Specifically in your case, you just need to apply disk quotas.
Using files with partitions inside is a common practice for hypervisors. But:

• It's a little slower.
• It is possible to get unreliable storage if there is a completely crazy chain of caches.