Routing multiple network interfaces inside a Xen PV CentOS VPS

1) Comment the "GATEWAY=" line in the /etc/sysconfig/network file.

Let us first set some symbolical names. Let $IF1 be the name of the first interface and $IF2 the name of the second interface. Then let $IP1 be the IP address associated with $IF1 and $IP2 the IP address associated with $IF2. Next, let $P1 be the IP address of the gateway at Provider 1, and $P2 the IP address of the gateway at provider 2. Finally, let $P1_NET be the IP network $P1 is in, and $P2_NET the IP network $P2 is in.

2) Creates two additional routing tables, say T1 and T2. These are added in /etc/iproute2/rt_tables. 

echo 1 T1 >> /etc/iproute2/rt_tables
echo 2 T2 >> /etc/iproute2/rt_tables

Eg:

================================
[root@test ~]# cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
1 T1
2 T2
================================

3) Then you set up routing in these tables as follows:

       ip route add $P1_NET dev $IF1 src $IP1 table T1
       ip route add default via $P1 table T1
       ip route add $P2_NET dev $IF2 src $IP2 table T2
       ip route add default via $P2 table T2

4) Next you set up the main routing table. Note the `src' arguments, they make sure the right outgoing IP address is chosen.

         ip route add $P1_NET dev $IF1 src $IP1
         ip route add $P2_NET dev $IF2 src $IP2

5) Then, your preference for default route:

         ip route add default via $P1

6) Next, you set up the routing rules. These actually choose what routing table to route with. You want to make sure that you route out a given interface if you already have the corresponding source address:

         ip rule add from $IP1 table T1
         ip rule add from $IP2 table T2

7) Put the above commands in the /etc/rc.local file.

Example:
########################
ip route add 210.61.156.0/24 dev eth0 src 210.61.156.253 table T1
ip route add default via 210.61.156.254 table T1
ip route add 59.125.26.0/24 dev eth1 src 59.125.26.34 table T2
ip route add default via 59.125.26.254 table T2

ip route add 210.61.156.0/24 dev eth0 src 210.61.156.253
ip route add 59.125.26.0/24 dev eth1 src 59.125.26.34

ip route add default via 210.61.156.254

ip rule add from 210.61.156.253 table T1
ip rule add from 59.125.26.34 table T2
#########################

8) Reboot the VM and check if eth0 and eth1 are responding to ping requests.

Reference:

http://www.tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.rpdb.multiple-links.html

http://www.linuxhorizon.ro/iproute2.html