How to use a routed network for a VPS

This article is about how to configure a VPS host and VPS guests to use a routed network without spoiling IP addresses. Also a section is included to use BGP to announce the IP you have to the routers you use. A configuration like this is also used for the server where this site is located (in the Coloclue network). I do not include how to install a VPS host, only the network part is covered.

Note 1: All examples are based on Debian, but should also work on other Linux distributions.
Note 2: The examples are based on IPv4, but should also apply to IPv6 with small changes.


 

Configuring the VPS host

Install packages if you want to use vlans:

apt-get install vlan

Now create a new interface in /etc/network/interfaces, this interface is used for all virtual machines to connect to. The used IP will be the gateway for your virtual machines.

auto vlanbr

iface vlanbr inet static

address 10.0.0.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
bridge_maxwait 0

Prepare your VPS host for forwarding and NAT for outgoing connections:

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -A INPUT -d 255.255.255.255/32 -i vlanbr -j ACCEPT
iptables -A INPUT -s 10.0.0.0/24 -i vlanbr -j ACCEPT
iptables -A INPUT -s 10.0.0.0/24 -i eth0 -j ACCEPT
iptables -A FORWARD -s 10.0.0.0/24 -i vlanbr -o eth0 -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -d 10.0.0.0/24 -o eth0 -j LOG
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -d 255.255.255.255/32 -o vlanbr -j ACCEPT
iptables -A OUTPUT -d 10.0.0.0/24 -o vlanbr -j ACCEPT
iptables -A OUTPUT -d 255.255.255.255/32 -o eth0 -j ACCEPT

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

Per IP that should point to your VPS you will do something like:

/sbin/route add -net 172.16.10.10 netmask 255.255.255.255 gw 10.0.0.10

In this example 172.16.10.10 should be replaced with your public ip you want to route to your VPS. 10.0.0.10 is the internal IP for your VPS.

Per VPS you also want a line like this if you want outgoing connections from that VPS to use the public IP:

iptables -t nat -I POSTROUTING -s 10.0.0.10/32 -o eth0 -j SNAT --to-source 172.16.10.10

In this example the IP addresses have the same meaning as the previous example. eth0 is the interface used for all outgoing connections.

This is everything that is needed for routing IP addresses to your VPS guests from the VPS host. Now we will first at BGP on the VPS host and after that we will configure the important sections on the VPS guest.

BGP on the VPS host

Install required package (bird6 is used for IPv6, not covered besides the installation):

apt-get install bird bird6

Configure bird. You can base it on the example below, comment lines start with # and are used to explain the line just above it.

log syslog { debug, trace, info, remote, warning, error, auth, fatal, bug };
router id 172.16.25.25;
# Replace 172.16.25.25 with your public IPv4 address for the VPS host. If only a private IP is available check that it is uniq in your environment.

function is_owned_by_me()
prefix set owned_by_me_space;
{
owned_by_me_space = [ 172.16.10.10/30{30,32}, 172.16.20.20/31{31,32} ];
# Replace 172.16.10.10/30 and 172.16.20.20/31 with the prefixes that should be routed to you. Also mention the largest and biggest announcements that are allowed within { and }. The first example allows a /30, /31 and /32 to be announced for this range. In the last example only /31 and /32 are allowed.
if net ~ owned_by_me_space then return true;
return false;
}

filter ebgp_import {
if ( is_owned_by_me () ) then accept;
reject;
}

template bgp ebgp {
local as 65151;
# Define a local as, if you don't have an AS from a RIR use something that is uniq in your network. If needed ask what you need to use at your network supplier/network admin.
import all;
export filter ebgp_import;
source address 172.16.25.25;
# Replace 172.16.25.25 with your public IPv4 address for the VPS host. If only a private IP is available check that it is uniq in your environment. This should be the address used for communication between the host and the routers.
next hop self;
}

protocol bgp dcg1 from ebgp {
neighbor 172.16.25.254 as 8283;
# Replace 172.16.25.254 with the IP of the first router that is your uplink
}
protocol bgp dcg2 from ebgp {
neighbor 172.16.25.253 as 8283;
# Replace 172.16.25.253 with the IP of the second router that is your uplink
}

protocol kernel {
learn; # Learn all alien routes from the kernel
persist; # Don't remove routes on bird shutdown
scan time 20; # Scan kernel routing table every 20 seconds
import all; # Default is import all
export all; # Default is export none, changed to all
}
# This pseudo-protocol watches all interface up/down events.
protocol device {
scan time 10; # Scan interfaces every 10 seconds
}
protocol direct {
interface "eth0";
}

After this restart bird (/etc/init.d/bird restart).

The host is now configured. Now go to the VPS guests.

The VPS guests network configuration

For the installation configure everything to use the internal IP (10.0.0.10 in the example). After that add the following to /etc/network/interfaces and bring up the new interface.

auto lo:0
iface lo:0 inet static
address 172.16.10.10
netmask 255.255.255.255

Per public/external IP add an interface with the IP.

Now run the following commands and it should work:

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

echo 1 > /proc/sys/net/ipv4/ip_forward

Now everything should be configured.

PowerDNS scripts

The last few years I did create multiple scripts that could be used with PowerDNS auth. All scripts are tested with PowerDNS auth 3.2.

For help installing and using the scripts I strongly suggest mailing to info@sinnerg.nl and asking for a quote for the problems you have.

Cleaning “old” domains where you are the slave from PowerDNS:

<?php
// Script created by Mark Scholten (www.mscholten.eu) for SinnerG BV (www.sinnerg.nl)
// Distribution is allowed if you don't change this copyright notice
// Changing this code is allowed if you don't change this copyright notice (at least for the parts created by Mark Scholten)
// Asking money for this script is allowed, however if you didn't change it don't say you created it (if you want to donate money, please donate it to PowerDNS)
// Mark Scholten and SinnerG BV provide this script "as is" and without any warranties, it is possible that there are errors in this script
 
// This script requires that the column failed is added to the domains table in the powerdns database, without it it will not work. int(5) should really be enough for this column
 
mysql_connect(  'localhost', // host
'user', //username
'pass'); //pass
mysql_select_db('pdns');
$test = 0; // change to 0 to delete records/domains after $checks tests that failed, if set to something else it will not delete anything
$checks = 7; // define the number of checks before deleting records, this has to be a number. Setting it to 0 will clean it immediately after 1 failed check
$verbose = 0; // set to 1 to be verbose (output domainnames that are deleted/are to be deleted (depending on the $test setting)
 
function is_stil_active($domain,$server){
$axfr = shell_exec("dig AXFR ".$domain." @".$server."");
$explode = explode("XFR size:",$axfr);
if(isset($explode['1'])){
return TRUE;
}else{
sleep(1);
$axfr = shell_exec("dig AXFR ".$domain." @".$server."");
$explode = explode("XFR size:",$axfr);
if(isset($explode['1'])){
return TRUE;
}else{
return FALSE;
}
}
}
$timestamp = time()-(24*3600); // get the timestamp for 24 hours ago
 
mysql_query("UPDATE domains SET `failed`=0 WHERE `type`='SLAVE' AND `last_check` > ".$timestamp." AND `failed` NOT LIKE '0'");
 
$sql3 = "SELECT `id`,`name`,`master`,`failed`,`account` FROM domains WHERE `type`='SLAVE' AND `last_check` < ".$timestamp;
$query = mysql_query($sql3) or die(mysql_error());
$dump = '';
if(mysql_num_rows($query) == FALSE){
}else{
while($record = mysql_fetch_object($query)){
if(!is_stil_active($record->name,$record->master)){
if($test === 0){
mysql_query("UPDATE domains SET `failed`=failed+1 WHERE id='".$record->id."'") or die(mysql_error());
if($record->failed >= $checks){
mysql_query("DELETE FROM records WHERE domain_id='".$record->id."'");
mysql_query("DELETE FROM cryptokeys WHERE domain_id='".$record->id."'");
mysql_query("DELETE FROM domains WHERE id='".$record->id."'");
mysql_query("DELETE FROM domainmetadata WHERE domain_id='".$record->id."'");
//mysql_query("DELETE FROM dnssec WHERE domain_id='".$record->id."'"); // a table we use internally for the dnssec data (with some scripts it can be requested remote (read only) by our domain management UI)
}
}
if($verbose === 1){
echo $record->name."
";
}
$dump .= $record->name." - ".$record->master." - ".$record->account." - ".$record->failed."\r\n";
}elseif($record->failed != 0){
mysql_query("UPDATE domains SET `failed`=0 WHERE id='".$record->id."'") or die(mysql_error());
}
 
}
if($verbose === 1){
echo "Done
";
}
if($dump != ''){
//mail("mark@streamservice.nl","AXFR failed for the following domain names",$dump);
}
}
?>

Required for this to work is:

ALTER TABLE `domains` ADD `failed` INT( 5 ) NOT NULL

 

We also have a script to put all DNSSEC related information in a database, this will automatically sign domains:

<?php
// Start config
$mysqlhost = "127.0.0.1"; //mysql server
$mysqluser = "user"; // mysql user
$mysqlpass = "pass"; // mysql pass
$mysqldaba = "pdns"; //mysql database
$pdnssec = "/usr/bin/pdnssec"; // pdnssec
// End config
 
$mysqli = mysqli_init();
if(!$mysqli){
die('FATAL ERROR: mysqli_init failed');
}
if(!$mysqli->real_connect($mysqlhost, $mysqluser, $mysqlpass, $mysqldaba)){
die('FATAL ERROR: mysqli->real_connect failed');
}
$query = $mysqli->query('SELECT id,name,changed FROM `domains` WHERE `type` NOT LIKE "SLAVE"') or die($mysqli->error);
if($query->num_rows == "0"){
}else{
while($row = $query->fetch_array(MYSQLI_ASSOC)){
exec($pdnssec." show-zone ".$row['name']." 2>&1", &$output, $retval);
$dnssec = 0;
$ds = 1;
$dnskey = 1;
unset($ds);
unset($dnskey);
foreach($output as $line){
if($line == "Zone has NSEC semantics"){
$dnssec++;
}elseif($line == "Zone is not presigned"){
$dnssec++;
}elseif($line == "keys:"){
$dnssec++;
}else{
$expl = explode(' ',$line);
if($expl[0] === "DS"){
$expl2 = explode('DS',$line,3);
$ds[] = trim($expl2[2]);
unset($expl2);
}elseif($expl[0] === "KSK"){
$expl2 = explode('DNSKEY',$line,3);
$dnskey[] = trim($expl2[2]);
unset($expl2);
}
unset($expl);
}
}
unset($output);
if($dnssec !== 3){
exec($pdnssec." secure-zone ".$row['name']." 2>&1", &$output, $retval);
$secure = 0;
foreach($output as $line){
if($line == "Zone ".$row['name']." secured"){
$secure++;
}
}
unset($output);
if($secure !== 1){
exec($pdnssec." secure-zone ".$row['name']." 2>&1", &$output, $retval);
$secure = 0;
foreach($output as $line){
if($line == "Zone ".$row['name']." secured"){
$secure++;
}
}
unset($output);
if($secure !== 1){
exec($pdnssec." secure-zone ".$row['name']." 2>&1", &$output, $retval);
$secure = 0;
foreach($output as $line){
if($line == "Zone ".$row['name']." secured"){
$secure++;
}
}
unset($output);
if($secure !== 1){
exec($pdnssec." secure-zone ".$row['name']." 2>&1", &$output, $retval);
$secure = 0;
foreach($output as $line){
if($line == "Zone ".$row['name']." secured"){
$secure++;
}
}
unset($output);
}
}
}
exec($pdnssec." show-zone ".$row['name']." 2>&1", &$output, $retval);
$dnssec = 0;
foreach($output as $line){
if($line == "Zone has NSEC semantics"){
$dnssec++;
}elseif($line == "Zone is not presigned"){
$dnssec++;
}elseif($line == "keys:"){
$dnssec++;
}else{
$expl = explode(' ',$line);
if($expl[0] === "DS"){
$expl2 = explode('DS',$line,3);
$ds[] = trim($expl2[2]);
unset($expl2);
}elseif($expl[0] === "KSK"){
$expl2 = explode('DNSKEY',$line,3);
$dnskey[] = trim($expl2[2]);
unset($expl2);
}
unset($expl);
}
}
unset($output);
}
if(is_array($ds) && is_array($dnskey)){
$mysqli->query('DELETE FROM `dnssec` WHERE `domainid` LIKE '.$row['id'].'');
foreach($ds as $record){
$mysqli->query('INSERT INTO `dnssec` (`domainid`,`type`,`record`) VALUES ("'.$row['id'].'","DS","'.$record.'")');
}
foreach($dnskey as $record){
$mysqli->query('INSERT INTO `dnssec` (`domainid`,`type`,`record`) VALUES ("'.$row['id'].'","DNSKEY","'.$record.'")');
}
//echo $row['name']." secured and key stored.\r\n";
}
}
}
?>

This script requires an additional table, you can create this table with:

CREATE TABLE IF NOT EXISTS `dnssec` (
`id` bigint(255) NOT NULL auto_increment,
`domainid` int(255) NOT NULL,
`type` varchar(10) default NULL,
`record` varchar(255) NOT NULL,
PRIMARY KEY (`id`),
KEY `domainid` (`domainid`,`type`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;

To rectify all domains you can use the script below. For every change you do on the domains and records table you need to update the domains table and increase changed with 1. The script you can use is listed below:

<?php
// Script created by Mark Scholten (www.mscholten.eu) for SinnerG BV (www.sinnerg.nl)
// Distribution is allowed if you don't change this copyright notice
// Changing this code is allowed if you don't change this copyright notice (at least for the parts created by Mark Scholten)
// Asking money for this script is allowed, however if you didn't change it don't say you created it (if you want to donate money, please donate it to PowerDNS)
// Mark Scholten and SinnerG BV provide this script "as is" and without any warranties, it is possible that there are errors in this script
 
// This script assumes that there is an additional column in the domains table that contains the number of changes since the last time this script did run, we run this script every minute so the number is low (normally 0)
// If this isn't done there might be problems with domains not resolving, a default value for this column of 1 is probably the easiest option so you only need to update it (+1) when you update/remove/add a record
 
// Start config
$mysqlhost = "127.0.0.1"; //mysql server
$mysqluser = "user"; // mysql user
$mysqlpass = "pass"; // mysql pass
$mysqldaba = "database"; //mysql database
$pdnssec = "/usr/bin/pdnssec"; // pdnssec
// End config
 
$mysqli = mysqli_init();
if(!$mysqli){
die('FATAL ERROR: mysqli_init failed');
}
if(!$mysqli->real_connect($mysqlhost, $mysqluser, $mysqlpass, $mysqldaba)){
die('FATAL ERROR: mysqli->real_connect failed');
}
$query = $mysqli->query('SELECT id,name,changed FROM `domains` WHERE `changed` NOT LIKE "0"') or die($mysqli->error);
if($query->num_rows == "0"){
}else{
while($row = $query->fetch_array(MYSQLI_ASSOC)){
$output = passthru($pdnssec." rectify-zone ".$row['name'], $retval);
$mysqli->query("UPDATE `domains` SET `changed` = `changed`-".$row['changed']." WHERE `id` = ".$row['id']." LIMIT 1");
}
}
 
// To start ordering rectify-zone to all domain names without another system being the master: UPDATE `domains` SET `changed` = +1 WHERE `master` IS NULL;
// ALTER TABLE `domains` ADD `changed` INT( 5 ) NOT NULL;
?>

For this to work the following is required:

ALTER TABLE `domains` ADD `changed` INT( 5 ) NOT NULL

I hate spam

Some people like to spam, I don’t like it (I hate it). They also try that here on this blog.

Spam comments will be changed and/or deleted, when I’m in doubt I’ll change the comment and at least remove the links.

nl.php.net updated

After some time without updating the system running nl.php.net it was time for an update. Last night I updated it and now it is running a current PHP/Apache/Debian version. The update did take around 10 minutes and restarting the system was the only downtime.

Debian upgrade problems

The last time I updated a system from Debian Lenny to a more recent Debian version I discovered a problem. This was from Debian Lenny to Debian Squeeze and at the end I upgraded to Debian Wheezy. This was caused by tar being an old version and not the most recent version, this is something that shouldn’t happen. The solution I used was installing a more recent tar version.

The problem:

server:~# apt-get dist-upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
You might want to run `apt-get -f install’ to correct these.
The following packages have unmet dependencies:
dpkg-dev: Depends: base-files (>= 5.0.0) but 5lenny6 is installed
Recommends: libalgorithm-merge-perl but it is not installed
libc6: Breaks: locales-all (< 2.13) but 2.11.3-2 is installed
libc6-amd64: Depends: libc6 (= 2.11.3-2) but 2.13-27 is installed
libc6-i686: PreDepends: libc6 (= 2.11.3-2) but 2.13-27 is installed
locales-all: Depends: glibc-2.11-1
E: Unmet dependencies. Try using -f.
server:~# apt-get -f install
Reading package lists… Done
Building dependency tree
Reading state information… Done
Correcting dependencies… Done
The following packages were automatically installed and are no longer required:
(…)

Use ‘apt-get autoremove’ to remove them.
The following extra packages will be installed:
base-files
The following packages will be REMOVED:
lib64gcc1 lib64gomp1 libc6-amd64 libc6-i686 locales-all
The following packages will be upgraded:
base-files
1 upgraded, 0 newly installed, 5 to remove and 340 not upgraded.
1 not fully installed or removed.
Need to get 67.5kB of archives.
After this operation, 16.7MB disk space will be freed.
Do you want to continue [Y/n]? y
Get:1 http://debian.apt-get.eu squeeze/main base-files 6.0squeeze4 [67.5kB]
Fetched 67.5kB in 0s (2072kB/s)
tar: unrecognized option `–warning=no-timestamp’
Try `tar –help’ or `tar –usage’ for more information.
dpkg-deb: error: subprocess tar returned error exit status 64
dpkg: error processing /var/cache/apt/archives/base-files_6.0squeeze4_i386.deb (–unpack):
subprocess dpkg-deb –control returned error exit status 2
Errors were encountered while processing:
/var/cache/apt/archives/base-files_6.0squeeze4_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
server:~# aptitude full-upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
Reading extended state information
Initializing package states… Done
Writing extended state information… Done
Reading task descriptions… Done
The following packages are BROKEN:
(…)
The following NEW packages will be installed:
(…)
The following partially installed packages will be configured:
dpkg-dev
The following packages are RECOMMENDED but will NOT be installed:
(…)

334 packages upgraded, 72 newly installed, 24 to remove and 0 not upgraded.
Need to get 234MB/234MB of archives. After unpacking 244MB will be used.
The following packages have unmet dependencies:
libfont-freetype-perl: Depends: perlapi-5.10.0 which is a virtual package.
locales-all: Depends: glibc-2.11-1 which is a virtual package.
libc6-i686: PreDepends: libc6 (= 2.11.3-2) but 2.13-27 is installed.
libept0: Depends: libapt-pkg-libc6.7-6-4.6 which is a virtual package.
libuuid-perl: Depends: perlapi-5.10.1 which is a virtual package.
libcap-dev: Depends: libcap2 (= 1:2.19-3) but 1:2.22-1 is installed.
libc6-amd64: Depends: libc6 (= 2.11.3-2) but 2.13-27 is installed.
libmudflap0-dev: Depends: gcc-4.1-base (= 4.1.2-25) but 4.1.2-29 is to be installed.
libc6: Breaks: locales-all (< 2.13) but 2.11.3-2 is installed.
The following actions will resolve these dependencies:

Remove the following packages:
lib64gcc1
lib64gomp1
libc6-amd64
libc6-i686
libept0
libmudflap0-dev
linux-image-2.6-486
linux-image-2.6-686-bigmem
locales-all

Keep the following packages at their current version:
libcap-dev [1:1.10-14 (now)]
libfont-freetype-perl [Not Installed]
libuuid-perl [Not Installed]
linux-base [Not Installed]
linux-image-2.6.32-5-486 [Not Installed]
linux-image-2.6.32-5-686-bigmem [Not Installed]

Leave the following dependencies unresolved:
defoma recommends libfont-freetype-perl
linux-image-2.6.26-2-686-bigmem recommends libc6-i686
libc6 recommends libc6-i686
Score is 190

Accept this solution? [Y/n/q/?]y
The following NEW packages will be installed:
(…)
The following packages will be upgraded:
(…)
The following partially installed packages will be configured:
dpkg-dev
The following packages are RECOMMENDED but will NOT be installed:
(…)
328 packages upgraded, 66 newly installed, 34 to remove and 1 not upgraded.
Need to get 178MB/178MB of archives. After unpacking 61.9MB will be used.
Do you want to continue? [Y/n/?] y

(…)

Fetched 178MB in 12s (14.0MB/s)
Extracting templates from packages: 100%
Preconfiguring packages …
tar: unrecognized option `–warning=no-timestamp’
Try `tar –help’ or `tar –usage’ for more information.
dpkg-deb: error: subprocess tar returned error exit status 64
dpkg: error processing /var/cache/apt/archives/patch_2.6-2_i386.deb (–unpack):
subprocess dpkg-deb –control returned error exit status 2
tar: unrecognized option `–warning=no-timestamp’
Try `tar –help’ or `tar –usage’ for more information.
dpkg-deb: error: subprocess tar returned error exit status 64
dpkg: error processing /var/cache/apt/archives/base-files_6.0squeeze4_i386.deb (–unpack):
subprocess dpkg-deb –control returned error exit status 2
Errors were encountered while processing:
/var/cache/apt/archives/patch_2.6-2_i386.deb
/var/cache/apt/archives/base-files_6.0squeeze4_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install.  Trying to recover:
dpkg: dependency problems prevent configuration of dpkg-dev:
dpkg-dev depends on base-files (>= 5.0.0); however:
Version of base-files on system is 5lenny6.
dpkg: error processing dpkg-dev (–configure):
dependency problems – leaving unconfigured
Errors were encountered while processing:
dpkg-dev
Reading package lists… Done
Building dependency tree
Reading state information… Done
Reading extended state information
Initializing package states… Done
Writing extended state information… Done
Reading task descriptions… Done

Like you can see there was an error while upgrading the Debian system. The first thing I did try didn’t work:

cd /usr/bin

mv dpkg dpkg-old

mv dpkg-deb dpkg-deb-old
mv dpkg-divert dpkg-divert-old
mv dpkg-maintscript-helper dpkg-maintscript-helper-old
mv dpkg-split dpkg-split-old
mv dpkg-query dpkg-query-old
mv dpkg-statoverride dpkg-statoverride-old
mv dpkg-trigger dpkg-trigger-old

scp other_server_with_debian_lenny_and_same_kernel:/usr/bin/dpk* ./

chmod a+x dpk*

This didn’t solve the problems. After that I did download and install a newer tar version. This is something you can copy (please change version number/kernel type when needed):

mkdir /tmp/tar

cd /tmp/tar

wget http://ftp.br.debian.org/debian/pool/main/t/tar/tar_1.26-4_i386.deb

ar x tar_1.26-4_i386.deb

tar xzf data.tar.gz

cp -p /bin/tar /bin/tar-old

cp -p bin/tar /bin/tar

This way tar was installed and upgrading was possible again without really big problems. Below was what I did next.

nano /etc/apt/sources.list # Change it to Debian Wheezy

apt-get clean

apt-get update

apt-get install tar

apt-get dist-upgrade # Did give a problem

apt-get -f install # Didn’t really work

aptitude full-upgrade # Multiple times, till there where no packages to update/upgrade

That was how it was solved.