[poetics5]

I promised to release these last year, but lost my editor shortly after. Hope you can use it anyway, pardon any mistakes. I'll post more of the inprogress tuts from last year this week.

Lates,
Sid

Setting up drbd on vps.net ubuntu

‎Thursday, ‎June ‎25, ‎2009, ‏‎4:22:59 PM

Using vps.net's vps cloud allows you setup a backend and scale up on the fly,
unfornately vps.net's cloud system prevents you from modifying the current partition tables. In order to get around this we'll setup drbd using loopback devices instead of actual partions. Other than this limitation creating a high avaibility san on vps.net is pretty straight forward.

So lets get rocking

Create the filesystem

first things first we'll need to create the folders for everything we'll be doing

mkdir -p /srv/fs /srv/data

and now we'll answer the question of how do we create new partitions without being able to edit the actual partion table, drum roll please... loopback devices.

A big thank you to James Withall at vps.net for recommending I look into loopback devices.


creating loopback devices for drbd
we'll be creating 2 devices for drbd, 1 for drbd meta data and 1 for drbd data. Our meta data disk will be 256mb and our data disk will be 7.5gb. So lets create our devices.


Create the meta data file

dd if=/dev/zero of=/srv/fs/meta.img bs=1024 count=250000

Create the data file

dd if=/dev/zero of=/srv/fs/data.img bs=1024 count=7325000

The next step is to create our loopback device.

Loopback device
We'll need to connect our LOFS at system startup, so I created a script to handle this for you. Place this script at: /etc/init.d/vpsdrbdlofs

Create File via Nano

sudo nano /etc/init.d/vpsdrbdlofs

Here is the script

#!/bin/bash

#

#Startup script to create LOFSs for drbd on ubuntu / vps.net 

#

#Author: Sid Sidberry <greg@halfgray.com> http://himynameissid.com

#

#Description: This script attaches files from the file system to loopback 

#	devices for use as drbd partitions. 

#	Two files are required, 1 for drbd meta-data and 1 for drbd data 

#


#your partion files

DRBD_METADATA_SRC="/srv/fs/meta.img"

DRBD_FILEDATA_SRC="/srv/fs/data.img"


#loopback devices

DRBD_METADATA_DEVICE="/dev/loop6"

DRBD_FILEDATA_DEVICE="/dev/loop7"


#losetup

LOSETUP_CMD=/sbin/losetup


#make sure the src files exist

[ -x $LOSETUP_CMD ] || exit 0

[ -e "$DRBD_METADATA_SRC" ] || exit 0;

[ -e "$DRBD_FILEDATA_SRC" ] || exit 0;


#includes lsb functions

. /lib/lsb/init-functions


function connect_lofs

{


	log_daemon_msg "Connecting loop devices $DRBD_METADATA_DEVICE, $DRBD_FILEDATA_DEVICE"


	$LOSETUP_CMD $DRBD_METADATA_DEVICE $DRBD_METADATA_SRC

	$LOSETUP_CMD $DRBD_FILEDATA_DEVICE $DRBD_FILEDATA_SRC

}



function release_lofs

{


	log_daemon_msg "Releasing loop devices $DRBD_METADATA_DEVICE, $DRBD_FILEDATA_DEVICE"


	$LOSETUP_CMD -d $DRBD_METADATA_DEVICE

	$LOSETUP_CMD -d $DRBD_FILEDATA_DEVICE

}


case "$1" in


    start)

    	connect_lofs

    ;;


    release)

    	release_lofs

    ;;


	*)

        echo "Usage: /etc/init.d/vpsdrbdlofs {start|release}"

        exit 1

    ;;


esac


exit 0

Make the script executable and add to startup

sudo chmod +x /etc/init.d/vpsdrbdlofs

sudo update-rc.d vpsdrbdlofs defaults

Now your loopback devices will be attached at startup BEFORE our drbd resources are started, and we're ready to setup our server.

Configure fstab for our drbd file system

If your on vps.net your fstab loks something similar to:

proc            /proc   proc    defaults                        0 0


/dev/sda1       /       ext3    relatime,errors=remount-ro      0 1


/dev/sda2       none    swap    sw                              0 0

add an entry for drbd

/dev/drbd0      /srv/data ext3  noauto                          0 0

afterwards your fstab looks like

proc            /proc   proc    defaults                        0 0


/dev/sda1       /       ext3    relatime,errors=remount-ro      0 1


/dev/sda2       none    swap    sw                              0 0


/dev/drbd0      /srv/data ext3  noauto                          0 0

Configure /etc/hosts

We'll need to map our servers to there internal ips, the simplest way to do this is via the /etc/hosts file.

TODO

Install the userspace portion of drbd

sudo apt-get install drbd8-utils

enable the drbd module

check if drbd module is enabled

sudo cat /etc/modules | grep drbd

If found you should see

drbd

If not found add it

sudo echo 'drbd' > /etc/modules

or you can use nano

sudo nano /etc/modules

add

drbd

load drbd module

sudo modprobe drbd

Configure drbd

sudo nano /etc/drbd.conf

common { 


  protocol C;


  handlers {  


  	 pri-on-incon-degr "echo 'DRBD: primary requested but inconsistent!' wall; /etc/init.d/heartbeat stop";


     outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater";  


  } 


  startup {


 	wfc-timeout	60;


    degr-wfc-timeout 120;    # 2 minutes.


  }


  disk {


    on-io-error   detach;


    fencing    resource-only;  


  }


  net {


  	max-buffers 2500;#buffer a max of 10mb of data before writing to disk


  	ko-count	4;


  }


  syncer {


    rate 5M;


    cram-hmac-alg "sha1";


    shared-secret "strongpassword+app_cluster@NASTUT2009";


    al-extents 257;


  } 


}


resource web0{


 on fuel61{                


   device     /dev/drbd0;        #


   disk       /dev/loop7;         #data partition on server 1


   address    10.0.0.22:7788; #server 1 ip


   meta-disk  /dev/loop6[0];      #meta-data partition on server 1


  }


 on fuel62{                


   device    /dev/drbd0;         #


   disk      /dev/loop7;          #data partition on server 2


   address   10.0.0.23:7788;  #server 2 ip


   meta-disk /dev/loop6[0];       #meta-data partition on server 2


  }


}

create drbd meta data for our resource

sudo drbdadm create-md web0

after creating our meta data on both machines start drbd by running

sudo drbdadm up all

check the drbd status

cat /proc/drbd

Quote

version: 8.0.11 (api:86/proto:86)

GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43

0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0

resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0

act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0

Both servers should be secondary with the data being inconsistent. So the next step is to set our master.

On your master server run

sudo drbdsetup /dev/drbd0 primary -o

sudo drbdadm primary all

This will set the master server as the drbd primary and do the initial sync

Now check the status of drbd

cat /proc/drbd

Quote

version: 8.0.11 (api:86/proto:86)

GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43

0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---

ns:340704 nr:0 dw:0 dr:340704 al:0 bm:20 lo:312 pe:0 ua:311 ap:0

[>....................] sync'ed: 4.7% (6991/7324)M

finish: 0:48:42 speed: 2,400 (2,660) K/sec

resync: used:2/31 hits:21583 misses:22 starving:0 dirty:0 changed:22

act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0

You'll now see the master server is the primary and it's data is considered up to date. Our secondary servers data is still inconsistent

The next step is to create the file system

sudo mkfs.ext3 /dev/drbd0

check the status of drbd again

cat /proc/drbd

Quote

version: 8.0.11 (api:86/proto:86)

GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43

0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---

ns:1616496 nr:0 dw:192500 dr:1424036 al:91 bm:87 lo:312 pe:1 ua:311 ap:0

[==>.................] sync'ed: 19.8% (5883/7324)M

finish: 0:33:28 speed: 2,984 (2,332) K/sec

resync: used:1/31 hits:101907 misses:187 starving:0 dirty:0 changed:187

act_log: used:0/127 hits:48034 misses:146 starving:0 dirty:55 changed:91

Now our devices are syncing the file systems. when done you should see:

cat /proc/drbd

version: 8.0.11 (api:86/proto:86)


GIT-hash: b3fe2bdfd3b9f7c2f923186883eb9e2a0d3a5b1b build by phil@mescal, 2008-02-12 11:56:43


 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---


    ns:7641396 nr:0 dw:192500 dr:7448936 al:91 bm:458 lo:0 pe:0 ua:0 ap:0


	resync: used:0/31 hits:477879 misses:557 starving:0 dirty:0 changed:557


	act_log: used:0/127 hits:48034 misses:146 starving:0 dirty:55 

Now that the drbd file system is in sync and we're ready to setup our servers to use drbd's filesystem


Heartbeat configuration

To add drbd control to heartbeat add the following to your haresources file

drbddisk::web0 Filesystem::/dev/drbd0::/srv/data::ext3::rw

Heartbeat mysql setup is covered in detail in our ha mysql setup and ha active / active mysql tuts

If you need help setting up heartbeat read the heartbeat installation tut.

[Barry]

Thanks, I was thinking of setting up a DRDB enabled NFS server, this will help me get it up and running at lot easier :)

[poetics5]

@Barry: There's a tut covering that from last year. I'll post it later this week.

[Barry]

Sounds even better!
I'll be able to separately verify your steps :)

[poetics5]

Good to go and thanks for the twitter follow.

[Shammyh]

Very interesting post. Thanks a lot for this!

I looked at DRBD a long time ago as a potential HA/Failover solution, but thought it was impossible since as you mentioned VPS.net doesn't support editing the partition table, but this certainly seems like a very clever way of getting it to work.

First chance I get, I'll give this a run-through and see how it goes.

[poetics5]

I just posted the complete tut: "Creating a high availability san with s3 backups on vps.net". It's split in 2 due to forum limits and time. They'll show up once approved.

[Shammyh]

Very much looking forward to that! I'll keep an eye out on the forums for any posts by you.

I still haven't tested this particular DRBD config yet (though I have done it on non-vps.net servers), but the problem for me is finding a way to do master-master replication in a meaningful way for load balancing purposes, not just HA.

Also, have you had any experience with keepalived on VPS.net? I've been looking for a solution to allow me to pair HAProxy machines together with instant fail-over should one of them go down. Nick Nelson a while back said he a had a copy of some keepalived binaries that worked in the cloud... but I have yet to hear of any follow up on that.

[poetics5]

If you know how to do crm configs for heartbeat you can actually setup a CLUSTERIP that should adapt on failover, but don't quote me on that.

I usually use wackamole / spread for master / master as all I need is ip failover. IF you need services to also be controlled on failover then you'll need to learn heartbeats CRM configuration.

I'll be looking into CLUSTERIP again in the future, so can keep you updated if I find something new.

[poetics5]

Pardon, just realized nothing I said answered the question. For an active master / master you'll need to use a clustered file system on top of DRBD I believe. IF your thinking about mysql you can either use haproxy or mysql proxy to balance, but you'll have to separate reads / writes at the app level.

[Shammyh]

That was the problem I ran into with DRBD, it requires a clustered file system, which I must admit I have absolutely no experience with, and I have no idea if that would work with VPS.net, although I think the setup here is relatively file system agnostic (assuming you don't mind backups and other dashboard features not working), but i'm not sure how far that extends.

I took a look at wackamole a ways back when I ran across keepalived for IP fail-over, but I was under the impression that any public IP changes on a VM required a reboot? Meaning that those sort of setups wouldn't work? (unless I'm totally missing something)

I mean, the best solution would be Anycast IPs, but I'm not sure how feasible that is on VPS.net's end.

And thanks for the feedback!

This post has been edited by Shammyh: 02 April 2010 - 05:08 AM

[poetics5]

I'm used wackamole / spread successfully on vps.net. You just need to purchase additional ips. So instead of using the ip's the come with the machines, you'd use the purchased ips which are allowed to float between your VMs. I've also confirmed that CLUSTERIP works on vps.net.

Let me look through last years notes. I started a clustered file system, but stopped when the editor left.

[Shammyh]

Interesting, maybe I'll give it a try tomorrow...

Yea, I love the addon IPs, pretty much use them exclusively for outward facing services. But can they "float" between VMs instantly?

And love to hear whatever you've got about clustered file systems, since as I mentioned, I'm a complete novice with them.

[Barry]

poetics5 said:

Pardon, just realized nothing I said answered the question. For an active master / master you'll need to use a clustered file system on top of DRBD I believe. IF your thinking about mysql you can either use haproxy or mysql proxy to balance, but you'll have to separate reads / writes at the app level.

Hi,

I've read about Tungsten which includes several components for replication and proxying, but I've never set it up.
This would allow you to use a standard MySQL setup and handle the synchronization on top of MySQL (different approach)

[poetics5]

@Barry thanks for the link. I'll check it out.

@Shammyh yes the ips can "Float". I've almost narrowed my FS down to 2 choices, I'll let you know when I do.

My posts still haven't appeared, @ADMIN HELP!

[poetics5]

Started a discussion on Distributed File System options here on the forums.

It's been 12+ hours, so I guess the 2 part tut isn't getting approved. I'll post to the blog (being redesigned, so may be a week or 2)

[Shammyh]

That's pretty strange. I wonder why a Mod wouldn't approve it? And if you could drop a message in this thread (so I get an email alert) when you post it to your blog, I would be most grateful.

And good looks on the DFS thread.

[poetics5]

Uploaded all the either tut folder from last year to the forums: http://www.vps.net/f...d-tuts-the-dump

Do the poll and what not.

[Serg]

Hi there,

Thanks for the great tutorial!

I am looking for a way to roll out drbd on a vps (can't partition) and found your method with the loopbacks..
Official site however says loop devices are not recommended to use http://www.drbd.org/...-configure.html
Have you had any issues with this setup, deadlock problems?

Kind regards,

Sergii

poetics5, on 01 April 2010 - 10:49 AM, said:

I promised to release these last year, but lost my editor shortly after. Hope you can use it anyway, pardon any mistakes. I'll post more of the inprogress tuts from last year this week.


Lates,

Sid


Setting up drbd on vps.net ubuntu



‎Thursday, ‎June ‎25, ‎2009, ‏‎4:22:59 PM


Using vps.net's vps cloud allows you setup a backend and scale up on the fly,

unfornately vps.net's cloud system prevents you from modifying the current partition tables. In order to get around this we'll setup drbd using loopback devices instead of actual partions. Other than this limitation creating a high avaibility san on vps.net is pretty straight forward.


So lets get rocking


......

Heartbeat mysql setup is covered in detail in our ha mysql setup and ha active / active mysql tuts


If you need help setting up heartbeat read the heartbeat installation tut.

[catoseceazy]

Hello amazing thread we have going there! mars venus coach