elmore
May 25th, 2004, 00:38
Getting Started with CARP Part I
I've recently setup CARP and I thought I'd post a few notes for those of you that are interested in it. CARP is brand new to OpeBSD 3.5 and it stands for the Common Address Redundancy Protocol. Essentially, CARP allows for failover between two computers. When CARP is setup in conjunction with pfsync stateful failover is also possible with pf. This is my first experience with CARP so don't look at this how-to as the end all be all, I'm quite confident that this setup is probably not optimal in many places, as CARP gets documented better, and gets more refined, I'll post revisions to this guide as well.
In this example we'll be setting up the following:
A dual firewall setup which provides load balancing and redundancy between the firewalls for a webserver which sits behind the firewalls.
I used the following documents for reference when setting this up, I encourage you to fully read this text:
http://www.countersiege.com/doc/pfsync-carp/
http://www.openbsd.org/cgi-bin/man.cgi?query=carp&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
http://www.openbsd.org/cgi-bin/man.cgi?query=pfsync&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
http://www.openbsd.org/cgi-bin/man.cgi?query=pf&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
OK let's get started, the following is a crude network diagram of what we'll be setting up in this example:
INTERNET
|
|
CARP0 10.0.0.1/24
|
|
CARP1 10.0.0.1/24
|
|
10.0.0.2/24---10.0.0.3/24
FW1----pfsync----FW2
10.1.1.2/24---10.1.1.3/24
|
|
CARP3 10.1.1.1/24
|
|
CARP4 10.1.1.1/24
|
|
10.1.1.4/24
Webserver
That's a little more crude than I'd like but I'll try and explain it better here.
10.0.0.1/24 is the public virtual IP we'll be using when we create our CARP groups.
10.0.0.2/24 and 10.0.0.3/24 are the physical IP addresses of each firewall respectively.
10.1.1.2/24 and 10.1.1.3/24 are the physical internal interfaces of each firewall respectively.
10.1.1.1/24 is the private virtual IP we'll use when we create our internal CARP groups.
10.1.1.4/24 is the IP address ofthe webserver.
OK, here's one thing I'm not quite sure of so I'll just let everyone know right off the bat: I'm not sure that CARP group 1 and CARP group 3 are needed. However the CARP(4) manpage says that for this type of setup two CARP groups are needed.
In order to set up an ARP balanced virtual host, it is necessary to con-
figure one virtual host for each physical host which would respond to ARP
requests and thus handle the traffic. In the following example, 2 hosts
are configured to provide balancing and failover for the IP address
192.168.1.10.
Though Ryan McBride's page does not reflect this in his examples which have similar setups.
If anyone know for sure if these extra groups are needed please let me know so I can update this doc. and corresponding setups I have which use this scheme.
Now here's what you'll need to do to get this running:
On Firewall 1:
edit /etc/sysctl.conf and insert the following line:
net.inet.carp.arpbalance=1
edit /etc/pf.conf and insert the following rules:
Macros:
$carp_group="{ 10.0.0.2/24, 10.0.0.3/24, 10.0.0.1/24 }"
Rules:
#Filter Rules for Carp / pfstnc
pass in log quick on $ext_if proto pfsync from $carp_group
pass in log quick on $ext_if proto carp from $carp_group keep state
*If you have a default deny for outgoing traffic you need to add a corresponding Macro and rules for the internal carp groups as well.
Next create the follwing files in /etc;
hostname.carp0
hostname.carp1
hostname.carp3
hostname.carp4
hostname.pfsync0
Add the following to hostname.carp0:
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 1 pass somegoodphrasehere
Add the following to hostname.carp1:
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 2 advskew 100 passgoodphrasehere
Add the following to hostname.carp3
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 3 pass somegoodphrasehere
Add the following to hostname.carp4
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 4 advskew 100 pass somegoodphrasehere
Please note that the pass must be the same on each vhid (Virtual Host) i.e. - vhid 1 pass on firewall 1 must match vhid 1 pass on firewall 2.
Add the following to hostname.pfsync0
up syncif sis1
Where sis0 is the external network interface of your computer.
*Note it may not be a good idea to pass pfsync information over your external interface. A better option maybe to do this over your internal interface or as in Ryan McBride's examples a 3rd NIC can be added to transmit this information.
On Firewall 2 you do the exact same thing except for the follwowing:
in /etc/hostname.carp0
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 1 advskew 100 pass somegoodphrasehere
in /etc/hostname/carp1
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 2 pass somegoodphrasehere
in /etc/hostname.carp3
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 3 advskew 100 pass somegoodphrasehere
in /etc/hostname.carp4
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 4 pass somegoodphrasehere
Now that you have the second firewall done update the /etc/mygate file on your webserver to point to 10.1.1.1 for the default route.
Reboot the webserver and both firewalls.
When the boxes come back up you should have a nice new CARP setup.
To test simply halt firewall 1 and check the connection to your webserver. Once verified that you can still gain access to it restart firewall 1. Next halt firewall 2. Once verified that you can still gain access to your webserver restart firewall 2.
Next open two terminals 1 to each of the firewalls and run tcpsump -i sis0.
where sis0 is your external interface.
watch for incoming web traffic to make sure that traffic is being evenly split between your two firewalls.
Next run pfctl -s state to verify that your states are synced accross your firewalls.
Once satisfied you should be good to go.
In part 2 we'll setup CARP and pfsync on two Soekris 4801's for a redundant home firewall setup.
I've recently setup CARP and I thought I'd post a few notes for those of you that are interested in it. CARP is brand new to OpeBSD 3.5 and it stands for the Common Address Redundancy Protocol. Essentially, CARP allows for failover between two computers. When CARP is setup in conjunction with pfsync stateful failover is also possible with pf. This is my first experience with CARP so don't look at this how-to as the end all be all, I'm quite confident that this setup is probably not optimal in many places, as CARP gets documented better, and gets more refined, I'll post revisions to this guide as well.
In this example we'll be setting up the following:
A dual firewall setup which provides load balancing and redundancy between the firewalls for a webserver which sits behind the firewalls.
I used the following documents for reference when setting this up, I encourage you to fully read this text:
http://www.countersiege.com/doc/pfsync-carp/
http://www.openbsd.org/cgi-bin/man.cgi?query=carp&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
http://www.openbsd.org/cgi-bin/man.cgi?query=pfsync&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
http://www.openbsd.org/cgi-bin/man.cgi?query=pf&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
OK let's get started, the following is a crude network diagram of what we'll be setting up in this example:
INTERNET
|
|
CARP0 10.0.0.1/24
|
|
CARP1 10.0.0.1/24
|
|
10.0.0.2/24---10.0.0.3/24
FW1----pfsync----FW2
10.1.1.2/24---10.1.1.3/24
|
|
CARP3 10.1.1.1/24
|
|
CARP4 10.1.1.1/24
|
|
10.1.1.4/24
Webserver
That's a little more crude than I'd like but I'll try and explain it better here.
10.0.0.1/24 is the public virtual IP we'll be using when we create our CARP groups.
10.0.0.2/24 and 10.0.0.3/24 are the physical IP addresses of each firewall respectively.
10.1.1.2/24 and 10.1.1.3/24 are the physical internal interfaces of each firewall respectively.
10.1.1.1/24 is the private virtual IP we'll use when we create our internal CARP groups.
10.1.1.4/24 is the IP address ofthe webserver.
OK, here's one thing I'm not quite sure of so I'll just let everyone know right off the bat: I'm not sure that CARP group 1 and CARP group 3 are needed. However the CARP(4) manpage says that for this type of setup two CARP groups are needed.
In order to set up an ARP balanced virtual host, it is necessary to con-
figure one virtual host for each physical host which would respond to ARP
requests and thus handle the traffic. In the following example, 2 hosts
are configured to provide balancing and failover for the IP address
192.168.1.10.
Though Ryan McBride's page does not reflect this in his examples which have similar setups.
If anyone know for sure if these extra groups are needed please let me know so I can update this doc. and corresponding setups I have which use this scheme.
Now here's what you'll need to do to get this running:
On Firewall 1:
edit /etc/sysctl.conf and insert the following line:
net.inet.carp.arpbalance=1
edit /etc/pf.conf and insert the following rules:
Macros:
$carp_group="{ 10.0.0.2/24, 10.0.0.3/24, 10.0.0.1/24 }"
Rules:
#Filter Rules for Carp / pfstnc
pass in log quick on $ext_if proto pfsync from $carp_group
pass in log quick on $ext_if proto carp from $carp_group keep state
*If you have a default deny for outgoing traffic you need to add a corresponding Macro and rules for the internal carp groups as well.
Next create the follwing files in /etc;
hostname.carp0
hostname.carp1
hostname.carp3
hostname.carp4
hostname.pfsync0
Add the following to hostname.carp0:
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 1 pass somegoodphrasehere
Add the following to hostname.carp1:
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 2 advskew 100 passgoodphrasehere
Add the following to hostname.carp3
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 3 pass somegoodphrasehere
Add the following to hostname.carp4
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 4 advskew 100 pass somegoodphrasehere
Please note that the pass must be the same on each vhid (Virtual Host) i.e. - vhid 1 pass on firewall 1 must match vhid 1 pass on firewall 2.
Add the following to hostname.pfsync0
up syncif sis1
Where sis0 is the external network interface of your computer.
*Note it may not be a good idea to pass pfsync information over your external interface. A better option maybe to do this over your internal interface or as in Ryan McBride's examples a 3rd NIC can be added to transmit this information.
On Firewall 2 you do the exact same thing except for the follwowing:
in /etc/hostname.carp0
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 1 advskew 100 pass somegoodphrasehere
in /etc/hostname/carp1
inet 10.0.0.1 255.255.255.0 10.0.0.255 vhid 2 pass somegoodphrasehere
in /etc/hostname.carp3
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 3 advskew 100 pass somegoodphrasehere
in /etc/hostname.carp4
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 4 pass somegoodphrasehere
Now that you have the second firewall done update the /etc/mygate file on your webserver to point to 10.1.1.1 for the default route.
Reboot the webserver and both firewalls.
When the boxes come back up you should have a nice new CARP setup.
To test simply halt firewall 1 and check the connection to your webserver. Once verified that you can still gain access to it restart firewall 1. Next halt firewall 2. Once verified that you can still gain access to your webserver restart firewall 2.
Next open two terminals 1 to each of the firewalls and run tcpsump -i sis0.
where sis0 is your external interface.
watch for incoming web traffic to make sure that traffic is being evenly split between your two firewalls.
Next run pfctl -s state to verify that your states are synced accross your firewalls.
Once satisfied you should be good to go.
In part 2 we'll setup CARP and pfsync on two Soekris 4801's for a redundant home firewall setup.