Kernel_Killer
November 6th, 2006, 22:54
I figured this should get it's own topic, instead of trying to find it through my blog, or the 5.4 version's thread. Enjoy! Now confirmed to be a working How-To for FreeBSD 6.3 as well.
To start out, make sure you have your source tree installed for FreeBSD. If you do not have this, run /stand/sysinstall, go to Configure, Distributions, and then "src". After you have your system preped and ready for jails to be added, start planning out your virtual network.
In this tutorial, we are going to use our main interface as a switch. Each jail will be in the same collision domain, but they can be subnetted, or even put as their own networks. For now, we'll stick with the same network, and after this tutorial, you shouldn't have a problem changing the network to your needs.
First I'm going to set the IP to 10.0.0.86, and then run a jail off the main interface.
# ifconfig em0 10.0.0.86 255.255.255.255
# ifconfig em0 inet alias 10.0.0.88 netmask 255.255.255.255
# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=8<VLAN_MTU>
inet6 fe80::204:5aff:fe6f:1d0c%dc0 prefixlen 64 scopeid 0x1
inet 10.0.0.86 netmask 0xffffff00 broadcast 192.168.115.255
inet 10.0.0.88 netmask 0xffffffff broadcast 192.168.115.181
ether 00:04:5a:6f:1d:0c
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
Ok. So now we have our main interface at 10.0.0.86 and our aliased IP at 10.0.0.88. Since we don't want any inetd services listening on the aliased IP, we need to add a line to the /etc/rc.conf file.
inetd_flags="-wW -a 10.0.0.86"
Now to make the jail. First make the directory you want the jail in.
# mkdir /usr/jail/server1
Then we make the virtual system.
# cd /usr/src
# make world DESTDIR=/usr/jail/server1
# cd etc
# make distribution DESTDIR=/usr/jail/server1
# cd /usr/jail/server1
# ln -sf /dev/null kernel
Once this is all done your jail is built and ready to be set. First, a few minor details to the setup.
# touch /usr/jail/server1/etc/fstab
# cat /etc/resolv.conf > /usr/jail/server1/etc/resolv.conf
So now we are ready to configure the jail for the first time. We start the jail with the jail command, like so:
# jail /usr/jail/server1 jail1.prison.com 10.0.0.88 /bin/sh
You will then be dropped to a new shell inside the jail. First, set your root password with 'passwd' or with 'sysinstall'. Also setup a user account to use when using SSH to enter. Make sure to also run 'newaliases' while in this environment, otherwise you will have issues with your jail starting with sendmail. Next, add the SSH enable line into your /etc/rc.conf (in the jail).
sshd_enable="YES"
Once you have this done, type 'exit' to leave the jail. Now, we need to mount the proc and dev filesystems for our jail.
# mount -t procfs proc /usr/jail/server1/proc
# mount -t devfs dev /usr/jail/server1/dev
NOTE: If you are having trouble with SSH finding a console, make sure this is mounted
and now to start it all up:
# jail /usr/jail/server1 jail1.prison.com 10.0.0.88 /bin/sh /etc/rc
Now you should be able to ssh to your new jail, and start configuring the services you wish to run inside.
To double check that your jail is running, run 'jls'
# jls
JID IP Address Hostname Path
1 10.0.0.88 jail1.prision.com /usr/jail/server1
Well, now we have a fully functioning jail. Now we need to set it up so we don't have to start it manually. We are going to add a few more values to the rc.conf. You can probably put this anywhere in the rc.conf, but in good practice, you should put it behind main host specific entries (i.e. hostname, ifconfig, defaultroute, etc).
ifconfig_em0_alias0="inet 10.0.0.88 netmask 0xffffffff"
jail_enable="YES"
jail_list="jail1"
jail_socket_unixproute_only="YES"
jail_jail1_rootdir="/usr/jail/server1"
jail_jail1_hostname="jail.prison.com"
jail_jail1_ip="10.0.0.88"
jail_jail1_exec_start="/bin/sh /etc/rc"
jail_jail1_devfs_enable="YES"
jail_jail1_devfs_ruleset="devfsrules_jail"
Here we set the jail config name inside rc.conf, and the settings for that jail "jail_jail1*". We also set the global setting, "jail_socket_unixproute_only", to only allow TCP/IP to be used inside the jail.
Now, if we want to administer the jail without using SSH, we can always use the jexec command.
jexec <JID> <command>
# jexec 1 /bin/sh
This will drop you right into a Bourne in the jail (If it doesn't, run jls, and make sure the JID is correct). Now that we have the environment setup, you might want to start installing some ports. Before you go and drop the ports collection into your jail, why not make it to where you don't have to use all of that space? Drop out of your jail shell, if you are in one, to setup the ports from the main to the jail. Since symlinks do not work in a jail (and we don't want them to), login with your root user, or use sudo to create a ports dir, and mount a nullfs copy of the ports.
# mkdir /usr/jail/server1/usr/ports
# mount_nullfs /usr/ports /usr/jail/server1/usr/ports
If for any reason that you need to find processes in the jail, do a "ps ax | grep J" in a shell of the main system.
Now you should be all ready to go with your jail, and many more to come. Be sure that your network service applications in the jail are listening to the aliased IP, otherwise you might have issues connecting to the server with those services. In this sort of setup, you shouldn't have too much of a problem.
To start out, make sure you have your source tree installed for FreeBSD. If you do not have this, run /stand/sysinstall, go to Configure, Distributions, and then "src". After you have your system preped and ready for jails to be added, start planning out your virtual network.
In this tutorial, we are going to use our main interface as a switch. Each jail will be in the same collision domain, but they can be subnetted, or even put as their own networks. For now, we'll stick with the same network, and after this tutorial, you shouldn't have a problem changing the network to your needs.
First I'm going to set the IP to 10.0.0.86, and then run a jail off the main interface.
# ifconfig em0 10.0.0.86 255.255.255.255
# ifconfig em0 inet alias 10.0.0.88 netmask 255.255.255.255
# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=8<VLAN_MTU>
inet6 fe80::204:5aff:fe6f:1d0c%dc0 prefixlen 64 scopeid 0x1
inet 10.0.0.86 netmask 0xffffff00 broadcast 192.168.115.255
inet 10.0.0.88 netmask 0xffffffff broadcast 192.168.115.181
ether 00:04:5a:6f:1d:0c
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
Ok. So now we have our main interface at 10.0.0.86 and our aliased IP at 10.0.0.88. Since we don't want any inetd services listening on the aliased IP, we need to add a line to the /etc/rc.conf file.
inetd_flags="-wW -a 10.0.0.86"
Now to make the jail. First make the directory you want the jail in.
# mkdir /usr/jail/server1
Then we make the virtual system.
# cd /usr/src
# make world DESTDIR=/usr/jail/server1
# cd etc
# make distribution DESTDIR=/usr/jail/server1
# cd /usr/jail/server1
# ln -sf /dev/null kernel
Once this is all done your jail is built and ready to be set. First, a few minor details to the setup.
# touch /usr/jail/server1/etc/fstab
# cat /etc/resolv.conf > /usr/jail/server1/etc/resolv.conf
So now we are ready to configure the jail for the first time. We start the jail with the jail command, like so:
# jail /usr/jail/server1 jail1.prison.com 10.0.0.88 /bin/sh
You will then be dropped to a new shell inside the jail. First, set your root password with 'passwd' or with 'sysinstall'. Also setup a user account to use when using SSH to enter. Make sure to also run 'newaliases' while in this environment, otherwise you will have issues with your jail starting with sendmail. Next, add the SSH enable line into your /etc/rc.conf (in the jail).
sshd_enable="YES"
Once you have this done, type 'exit' to leave the jail. Now, we need to mount the proc and dev filesystems for our jail.
# mount -t procfs proc /usr/jail/server1/proc
# mount -t devfs dev /usr/jail/server1/dev
NOTE: If you are having trouble with SSH finding a console, make sure this is mounted
and now to start it all up:
# jail /usr/jail/server1 jail1.prison.com 10.0.0.88 /bin/sh /etc/rc
Now you should be able to ssh to your new jail, and start configuring the services you wish to run inside.
To double check that your jail is running, run 'jls'
# jls
JID IP Address Hostname Path
1 10.0.0.88 jail1.prision.com /usr/jail/server1
Well, now we have a fully functioning jail. Now we need to set it up so we don't have to start it manually. We are going to add a few more values to the rc.conf. You can probably put this anywhere in the rc.conf, but in good practice, you should put it behind main host specific entries (i.e. hostname, ifconfig, defaultroute, etc).
ifconfig_em0_alias0="inet 10.0.0.88 netmask 0xffffffff"
jail_enable="YES"
jail_list="jail1"
jail_socket_unixproute_only="YES"
jail_jail1_rootdir="/usr/jail/server1"
jail_jail1_hostname="jail.prison.com"
jail_jail1_ip="10.0.0.88"
jail_jail1_exec_start="/bin/sh /etc/rc"
jail_jail1_devfs_enable="YES"
jail_jail1_devfs_ruleset="devfsrules_jail"
Here we set the jail config name inside rc.conf, and the settings for that jail "jail_jail1*". We also set the global setting, "jail_socket_unixproute_only", to only allow TCP/IP to be used inside the jail.
Now, if we want to administer the jail without using SSH, we can always use the jexec command.
jexec <JID> <command>
# jexec 1 /bin/sh
This will drop you right into a Bourne in the jail (If it doesn't, run jls, and make sure the JID is correct). Now that we have the environment setup, you might want to start installing some ports. Before you go and drop the ports collection into your jail, why not make it to where you don't have to use all of that space? Drop out of your jail shell, if you are in one, to setup the ports from the main to the jail. Since symlinks do not work in a jail (and we don't want them to), login with your root user, or use sudo to create a ports dir, and mount a nullfs copy of the ports.
# mkdir /usr/jail/server1/usr/ports
# mount_nullfs /usr/ports /usr/jail/server1/usr/ports
If for any reason that you need to find processes in the jail, do a "ps ax | grep J" in a shell of the main system.
Now you should be all ready to go with your jail, and many more to come. Be sure that your network service applications in the jail are listening to the aliased IP, otherwise you might have issues connecting to the server with those services. In this sort of setup, you shouldn't have too much of a problem.