How to configure a Linux router with multiple ISP

Assume you have a PC with Fedora Core 4 installed, acting as a router.

You have a local network and 2 ISPs.

Your local network
Network: 192.168.1.0/24
Router's IP for this network: 192.168.1.254

ISP 1
Network: 10.0.1.0/24
Router's IP for this network: 10.0.1.1
Gateway: 10.0.1.254

ISP 2
Network: 10.0.2.0/24
Router's IP for this network: 10.0.2.1
Gateway: 10.0.2.254

Configuring Interfaces
First, we configure interface eth0 for the local network:
# ifconfig eth0 192.168.1.254/24

Then we configure interface eth1 for ISP 1:
# ifconfig eth1 10.0.1.1/24

Next we configure interface eth2 for ISP 2:
# ifconfig eth2 10.0.2.1/24

That's all for interface configuration, next we'll setup routing.

Configuring Routing Tables
When configuring multiple ISP, we need a separate routing table for each of them.

We'll use the number 101 for the name of ISP 1's routing table.
And 102 for the name of ISP 2's routing table.

ISP 1 Routing Table
Make sure we start with empty table
# ip route flush all table 101

Add loopback network
# ip route add 127.0.0.0/8 dev lo table 101

Add local network
# ip route add 192.168.1.0/24 dev eth0 table 101

Add the network of ISP 1
# ip route add 10.0.1.0/24 dev eth1 table 101

Add default gateway for ISP 1
# ip route add default via 10.0.1.254 dev eth1 table 101

ISP 2 Routing Table
Make sure we start with empty table
# ip route flush all table 102

Add loopback network
# ip route add 127.0.0.0/8 dev lo table 102

Add local network
# ip route add 192.168.1.0/24 dev eth0 table 102

Add the network of ISP 2
# ip route add 10.0.2.0/24 dev eth2 table 102

Add default gateway for ISP 2
# ip route add default via 10.0.2.254 dev eth2 table 102

And we all set for ISPs routing tables.

Fixed or Balanced?
Next, we'll need to decide, how we will be using those ISPs?

Do we need split access, that is some of the computers on our local network will be using ISP 1, and some other use ISP 2?

Or we just want to use both ISP, utilizing it's connection and balance the load of them?

Case A: Split Access
Assume we want some of the computers on our local network to use ISP 1.
Their IP addresses are: 192.168.1.11 through 192.168.1.13

And some other will use ISP 2.
Their IP addresses are: 192.168.1.21 through 192.168.1.23

Thus we need to define ip rules, to classify the packets by source address.

Rules for ISP 1
# ip rule add from 192.168.1.11 table 101
# ip rule add from 192.168.1.12 table 101
# ip rule add from 192.168.1.13 table 101

Rules for ISP 2
# ip rule add from 192.168.1.21 table 102
# ip rule add from 192.168.1.22 table 102
# ip rule add from 192.168.1.23 table 102

That's it for ip rules, we just have one last step to do and all will work as expected, jump to SNAT below.

* Edit *
I forgot that this one also needs default gateway on the main table for packets originating from the router itself, here it is:

# ip route add default via 10.0.1.254

Case B: Load Balance
For this we don't need ip rules, we only need to setup default gateway on the main routing table so packets will be using each ISP in a balanced way.

Remove the existing default gateway
# ip route del default

And add a load balanced gateway
# ip route add default nexthop via 10.0.1.254 weight 1 nexthop via 10.0.2.254 weight 1

The key is to use same values for 'weight' parameter. You could guess what would happen if you put more weight on one gateway than the other.

SNAT
We need to map local ip addresses to public ip address that's been assigned by our ISPs.

So all packets that will be going out through ISP 1 on interface eth1 will be using 10.0.1.1 as their source ip address.

And all packets that will be going out through ISP 2 on interface eth2 will be using 10.0.2.1 as their source ip address.

For ISP 1
# iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to 10.0.1.1

For ISP 2
# iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to 10.0.2.1

And we all set.