How to properly configure the network for servers (services) in containers?

Question

LXC

Introductory:

There is a server in the data center.
On it, you need to cock several independent web servers belonging to different owners.

For isolation reliability, it is supposed to use the
LXC container technology for the Linux version and Jails for the FreeBSD version of the "parent" server. The

server will have one real "white" address.
all servers.

It is assumed that port 80 will enter nginx, which will already determine by name which of the internal servers to transfer to.

Internet recipes look different

Someone will recommend creating a fictitious additional network interface of a local type (the same as 127.0.0.1 |) and attaching 192.168.0.x addresses to it, respectively, one address = one web server inside the container.

Someone recommends setting the address 192.168.0.x from inside the container to exactly the same interface on which the real external white address is attached outside.

But that's not all.

The first incoming nginx, which will multiply requests to internal servers, also wants to be put into a container. That is, he will need to make a redirect

But that's not all.

Containers from the inside want to sometimes have direct access to the Internet for updates,
and sometimes they didn’t have access (they don’t need it, it’s enough to listen to the local address on port 80 for the web server to work).
That is, you need NAT inside containers that can be forwarded and disabled.

What would be the concept with interfaces inside containers, with addresses on them, and what would be the connection of internal containers with the external Internet, and what kind of connection of internal containers with the first "incoming" main nginx would you suggest?

Answer 1

one internal bridge to route everything you need to it,
well, forward port 80 to one virtual machine,
and turn it on and off with a firewall rule

Answer 2

FreeBSD:
/etc/rc.conf file

cloned_interfaces="lo1"
ipv4_addrs="192.168.0.1/24"

firewall_enable="YES"
firewall_type="/etc/firewall"

file /etc/firewall

add 1040 allow ip from any to any via lo1
nat 1 config log if em0 reset same_ports deny_in redirect_port tcp 192.168.0.2:80 80 redirect_port tcp 192.168.0.2:443 443 
add 10130 nat 1 ip from any to any via em0
add 65534 deny all from any to any

Here:
em0 - interface looking at the Internet
lo1 - interface looking at the Jail network
192.168.0.1 - address of the external machine (host) on lo1
192.168.0.2 - address of the web server in the Jail, also on lo1
When you create a Jail , do not forget to specify that your address will be exactly on lo1
For example:
PS
The deny_in parameter in the nat configuration will close your server from all unknown incoming.
If you need ssh don't forget to add its port like redirect_port
Or before rule 10130 add skip of this rule for port 22.

Answer 3

sisn's answer is only good if you trust containers.
if there is a potential malware inside that can take the address and port of someone else's jail, then I would rely on vnet in FreeBSD.
And this means epair, not lo1
epair - emulates a direct cable , one end of which is connected to jail, the other end is connected to the parent host.
Well, you can’t do without a bridge here - after all, traffic needs to be steered somewhere further.
shawndebnath.com/articles/2016/03/27/freebsd-jails...

Answer 4

FreeBSD
epair+bridge seems to be the better way. however, in version 10 it was definitely not ready for production.
In version 11 you will have to recompile the kernel to enable it.
While the method with lo1 is ready from the box.