Nginx Scaling + High Availability Tips

[poetics5]

Since the tutorial blog is being delayed due to me writing more and more tuts I'll start releasing some of the smaller / helpful items here.

Here are some tips for setting up a high availability nginx server, there's a tut that covers all this from a-z, but hopefully this will help for now. Everything except the wackamole and server per cluster advice are good for any setup.

• Don't use the fair module, use ey balancer (maxconn) module
• separate servers by cluster, not domain
• use wackamole, not heartbeat (unless your using crm)
• setup round robin dns for your nginx servers
• serve your page cache directly from memcached

use ey, not fair
Don't use the fair balancer module, use ey balancer. the ey balancer module adds maxconn support that behaves like haproxy's maxconn. It also creates a request queue. Fair balancer will only return 500 errors

separate hosts by server / cluster, not domain

If you have multipule backends,

Ex: server1 / apache + mod+php, server2 / apache + passenger

Let the destination server handle the host by host stuff. Create one default server and one server with the domains used on the second server.

so following the example if server1 gets the most traffic, it's server entry would look like

server {

		listen 80;
		#default server

		server_name  default;

		location / {

		    proxy_pass http://server1;

		}

}

and server2's entry would look like

server {

		listen 80;
		#default server

		server_name  .himynameissid,com .babygotback.com .oldpeoplecantrunfast.info;

		location / {

		    proxy_pass http://server2;

		}

}

this makes configuration dead simple and keep the config light weight for nginx

Use wackamole, not heartbeat

Most people use heartbeat in ver 1 configuration mode. If your using crm with heartbeat you can skip this. If not use wackamole for fail over between your servers. setup is extremely easy, you can group ips, and i love it. This allows you to setup as many frontend servers as you'd like knowing they'll quickly and easily failover when needed.

Setup round robin dns
As mentioned above using wackamole you can setup multiple nginx entry servers. clusterip is a good option if your using heartbeat with crm, without it failover can cause a good portion of your requests to go unanswered. Either way you should setup round robin dns for your entry servers / reverse proxies. Round robin dns allows you to share your requests across multiple ips. While not true load balancing it definitely helps.

Serve your page cache directly from memcache

First off the number reason to do this is the coolness factor. Secondly it's fast as hell. Third, in a decent setup nginx is separated from the hosting file system. *edit* forgot to mention that this will also reduce load on the application servers.

On our old test servers we were able to consistently get our page cache to serve in under 400ms. So instead of caching a page to the file system, store it in memcached. For the key use HOST:REQUEST_URI

then setup nginx to serve the page cache

	server {

		listen 80;
		#default server

		server_name  default;

		#set the key for the varnish upstream
		set  $proxy_key  $host:$request_uri; 
		#set the key for memcached using the host and uri
		set  $memcached_key  $proxy_key;
		
		#check the memcached page cache if dynamic content
		location / {
			log_not_found  off;
			#when not found in memcache use proxy
			error_page 403 404 422 500 501 502 503 = @cluster; 
			#go directly to backend if request isn't using get

		    if ($request_method != "GET"){

		   		proxy_pass http://cluster;

				break;

			}
			#check if the page is in the memcached page cache   	

			memcached_pass   memcached;

			default_type     text/html;

		}
		

		# pass to  application cluster

		location @cluster {

		    internal;

		    proxy_pass http://cluster;

		}

	}

The tut covering all of this will be on the tut blog once I stop being a mysql nazi.

Greg

[Ditlev]

Thanks for all your tut's lately! Much appreciated!

[Shammyh]

Just found this post while looking into something else. Very useful stuff there, and thanks for sharing.

Any chance you could elaborate on the setup you used for roundrobin dns? i understand how it works, but i've never tried to host it myself.

any idea if vps.net's hosted dns supports this?

[nka]

can this setup be used directy on cPanel server ?

It may only failover HTTP(s) to the 2nd server, until the "master" is back online.

[nka]

can't edit my post.

But, what about MySQL Cluster ?

I'm trying to build this (well, I didn't begin yet lol) :
- Master server with cPanel with HTTP, MX, MySQL
- Slave server with HTTP failover (or load balanced too!), second MX and MySQL (Cluster?)

My only fear are that server become unsynced or slow cause of that setup (mysql cluster are slow?).