It seemed to make sense, but I was having a hard time wrapping my head around its ramifications. I was looking at some of the docs on the Varnish site and at this great Varnish config walkthrough when the metaphorical lightbulb went on. By adding the cookie to the hash it’s effectively creating a per-session cache.

Hmm. An interesting tradeoff. On one hand it’s filling up my available cache with duplicate copies of the same content because the hash identifying the cached content is cookie-specific. On the other, it is delivering content from cache that wouldn’t normally be cached because of the cookie.

I see three primary places we might start building on that seed:

• High Performance Drupal group
• Pressflow Launchpad (or its associated wiki)
• Project Mercury Launchpad

Those of you out there running Drupal in large scale environments, let’s start sharing configs and techniques so we can all do better.

The Problem:

HALP! The site, it is sinking!

A large school district wanted to replace their existing outdated static web site with a modern CMS. They chose Drupal as their platform. The new site was successful.

Too successful.

The average traffic of 5 hits/sec jumped to over 100 hits/sec and the server went into a swap death spiral.

Fear not! Help is on the way in to form of a couple of technological superheroes …

Pressflow and Varnish to the rescue!

The mutually-complimentary combination of these two tools can vastly increase the number of users your site can serve. Here’s the what, why, and how:

Pressflow

What:

Pressflow is “a derivative of Drupal core providing enhanced performance, scalability, and data integrity”. Basically, some really smart guys at Four Kitchens and elsewhere back-ported a bunch of Drupal 7 performance enhancements to the Drupal 6 (and even Drupal 5!) code base.

Why:

The most expensive thing a web site can do is have to fire up the entire Apache/PHP stack, pull something from the database, and render it. It takes a lot of time, processor cycles, and memory to do it. It’s slow. It ties up sessions waiting for the query to return and render the response. Whenever you can, push static content - images, CSS, JS, static files, etc. - into some sort of cache. Preferably in memory, and preferably as far out on the “edge” (as close to the requesting client browser) as possible. If we can avoid pulling something from disk (or the database), absolutely do it! If we can avoid even touching Apache/PHP (and its associated overhead), do it. The Pressflow changes help make the output more cache-friendly so that more and more of the site’s content can live in and be served from cache. With it we can free up those web server sessions and resources for serving content that does need to be dynamic.

How:

Pressflow adds the following features to Drupal.

• Support for database replication
• Support for Squid and Varnish reverse proxy caching
• Optimization for MySQL
• Optimization for PHP 5

All four are admirable additions that can help sites scale, but the second one is the primary reason I chose to bring the site up on Pressflow. It makes Drupal more cache-friendly, allowing us to store and serve more content from cache, speeding up the site and increasing the number of users who can be served.

My experience has been that Pressflow is also 100% compatible with Drupal core - I’ve switched sites back and forth between Pressflow and Drupal with no changes to the database or modules. Copy the /sites directory over and point the webroot at Pressflow, you’re done. Win!

So this is all well and good, but we need to actually have a cache in front of the server for this to do much good. Enter our second technology superhero:

Varnish

What:

Varnish is an HTTP accelerator and caching reverse proxy. Varnish is all about speed. It stores as much content as it can in the fastest place possible - RAM in this case - and bypasses the expensive process of making a request to Apache.

Why:

Pressflow structures the Drupal content to be more cache-friendly, but we still need something to actually cache the content.

How:

Varnish sits in front of Apache, accepts incoming connections from browsers and, if possible, fulfills the requests from its cache. If it can’t, it passes the request on to the underlying Apache/PHP stack. It then takes the response from Apache and forwards it on to the requesting browser. If the response from Apache is cacheable, Varnish stores it in RAM for fulfilling future requests.

Setting up Pressflow

Installing Pressflow is just like installing Drupal. Grab the tarball from Four Kitchens, unzip it, do the usual Drupal setup. You’re probably going to want to make sure Cacherouter is installed and properly configured (including the config array added to settings.php). Point your Apache vhost at the Pressflow docroot. That’s it for Pressflow. It’s Drupal, really. Just tweaked.

Setting up Varnish

1. Download and compile Varnish
2. Configure your Apache vhosts to listen on an alternate port (8080, for example)
3. Start varnishd

Configuring Varnish

Now the fun really begins. Varnish is amazingly-configurable. The VCL syntax is, even for a sysadmin-turned-developer like me, clear and relatively easy to understand. The default.vcl is well-commented. I learned a lot just reading through it. I’m going to post excerpts from my current Varnish config showing what I modified.

Please note - I am not a Drupal core hacker. I’m experimenting my way through this mostly through a “cut and try” methodology. This config info is based primarily on the work of others - Josh Koenig, Iskra/ekes and Narayan Newton. I don’t completely understand yet what a few of these configs do, exactly, other than they seem to make a positive difference in the hit rates on my systems and they didn’t seem to break anything. I’ll also include the configs I commented out because they did break something in the hopes that we can figure out why and what we should do to improve them.

By default, Varnish is set up to pass any requests with a cookie on to the backend (Apache) un-cached. We’re playing it safe and not risking sending an authenticated user outdated content by not sending cached content if we see a cookie. It’s a sane and conservative way of making sure that this is an anonymous user that can be given static content. Consequently, a lot of the customization is telling Varnish, “Even though there’s a cookie associated with this request you really can ignore the cookie and cache it” when the browser requests things like CSS files, JavaScript, theme images, or uploaded static files.

There are four basic places where I’ve added code: vcl_recv, vcl_hash, vcl_fetch and vcl_deliver. Below are the snipped (and very slightly-sanitized) sections from the config file I’m currently running in production (as of 10/10/2009).

vcl_recv

vcl_recv is where we configure what happens when Varnish receives a request from a browser client for some content.

This first segment clears out some cookies that are unnecessary. With the cookies set, Varnish won’t cache the associated content. So we tell Varnish to unset the cookie before continuing.

When I first set up Pressflow and Varnish, I was looking at the cache hit/miss rate and noticed a lot of the CSS and JS files in /sites and /misc were not being cached. So I thought I’d be clever and tell Varnish that it really should cache these files by un-setting the cookies. For a while, it worked great. Hit rates were up, lots of stuff was now being cached. Then I need to roll out a security update and run update.php. With this config in place, update.php would either reject the attempt (because the admin user session cookie has been unset, reverting the user to an anonymous session) or, if $update_free_access is set to TRUE, causing an endless loop back to step one. I’m not quite sure what in /sites and /misc is the root of the problem (both seem to cause it), but I’ve disabled this until we can identify and work around it.

Because of the update.php problem above, and because update.php uses Garland instead of the site theme, we can tell Varnish to cache the theme, module, and uploaded files here.

I noticed that we were also seeing a lot of misses on much of the core JS and CSS (like jquery.js), so we told Varnish to cache them.

We also run Moodle vhosts on this server. This was my first attempt at convincing Moodle to cache its images and theme files. It failed miserably. Moodle currently requires the session cookies be set on the files in question. Un-setting any of them forces the user to re-authenticate on every pageload. Hence the commenting.

Lastly, we don’t want server-status or cron cached, so tell Varnish to ignore it and pass them straight to the backend without further processing.

vcl_hash

vcl_hash is where (I believe) Varnish looks at the hash for content it has cached to make sure it’s still good.

This seems to help improve hit rates and non-intrusive, so I left it in.

vcl_fetch

vcl_fetch is where Varnish makes a request to the backend (Apache) for content it can’t for various reasons deliver from cache.

vcl_deliver

vcl_deliver is where Varnish delivers the requested content back to the browser client - either from cache or from a backend request.

Backend config

Lastly, we were seeing too many 503 errors under load, so we increased the timeouts to 600 seconds. So far we haven’t seen any ill effects from the log timeouts.

/etc/conf.d/varnishd

We have made a couple of changes to the default varnishd startup options:

Couple of notes here on this:

1. connect_timeout=600 - we were seeing random 503 errors when the system was under load, even though there were Apache workers available. We extended the timeouts to 600 seconds, figuring it was better to have an individual user occasionally get an element that loads slowly or times out than many users across the site seeing an uninformative “503 guru meditation”. So far we’ve not seen any poor side effects.
2. malloc,2G - we’re running the site with malloc storage (instead of the default file-based) at 2 gigabytes. The sites being served from behind this Varnish instance are quite a bit bigger than that, but because Varnish is running on the same box as Apache, we decided to throttle Varnish to leave resources for Apache/PHP. It’s likely this is suboptimal, but it works.

Tweaking and tuning Varnish, or “Is this thing on?”

So I’ve installed Varnish and Pressflow. The site’s up and running. How do I tell if it’s doing any good?

With Pressflow and Varnish installed with the default configurations we saw an immediate drop in load and better performance, but I wanted to optimize it to cache as much as possible. Fortunately, Varnish comes with an excellent set of tools to see what it’s doing:

varnishtop

This command shows the most often-made requests to the backend:

It’s excellent for spotting often-requested items that are currently not being cached. The “-b” flag filters for requests made to the backend. “-i TxURL” filters for the request URL that triggered the request to the backend. Its output looks something like this:

Top of the list, most often-requested URL from the backend. A prime candidate for caching.

varnishhist

This command hows a histogram for the past 1000 requests, whether they were cache hits (denoted by a ‘|’) or misses (denoted by a ‘#’), and how long the requests took to process (further to the right, longer time). It’s good for a high-level view of how the server is doing under load.

varnishlog

This command displays all varnish traffic for a specific client. It’s helpful for seeing exactly what a particular page or request is doing. Set it to your workstation IP, load the page, see everything Varnish does with your connection including hit/miss/pass status. Varnishlog is really useful, but it puts out an overwhelmingly-large amount of data that isn’t easily filtered. The “-o” option groups all of the entries for a specific request together (without it all entries from all requests are displayed fifo) and it accepts a tag (”ReqStart” in this example) and regex (the IP address in this case) to filter for only requests associated with that tag & regex. It’s the only way I’ve found to filter down the firehose of log entries into something useful.

varnishstat

This command provides an overview of the stats for the current Varnish instance. It shows hit/miss/pass rates and ratios, lots of other gory internal details.

Watch that RAM, or “vmstat, oh how I love thee!”

Varnish can eat RAM like there’s no tomorrow. Be careful and be sure to configure its max memory to be something less than your available RAM. I forgot when I first set things up. The system worked great for a while, and then took a nosedive as the Varnish cache ate up all the available RAM and pushed the system into a swap death spiral.

It’s OK to not cache everything

This is a concept I struggled with at first - “oh no! It’s not caching xyz! I must fix that!” Remember that even if you can’t cache all the static content on your site, you’re still doing a lot of good offloading the most commonly-accessed content onto Varnish. Every connection you can serve from Varnish frees up an Apache thread to do something else.

The Resolution

I am delighted to report that the site is currently serving more than 30,000 hits per day without any trouble. We’ve seen traffic exceed 150 simultaneous clients without pushing the system into swap, nor is it seeing significant iowait. As best we can tell, at peak traffic times it’s entirely processor-bound with all four cores running at 95% or higher servicing apache threads.

OK, now what?

Moving forward, how do we do better? There were some great discussions at DrupalCamp PDX this weekend about how we can do more with Varnish and Drupal/Pressflow. Some interesting ideas that came up:

• Edge Side Includes (ESI), and making Drupal aware of them. Specifically interesting in relation to pushing the internal Drupal panels and block caches into an ESI store.
• A Varnish module within Drupal that hooks into the internal Drupal cache functions and can talk to the Varnish server on its management port to tell it to purge or extend the lifetime of content.
• Preconfigured Amazon EC2 images to quickly scale sites … oh wait, there’s Project Mercury!
• Intelligent ways to front multiple Drupal webnodes with one (or a clustered pair) of Varnish nodes.

I’m up for taking a whack at it. Any Drupal core wizards out there interested?

Photo credits:
[1] [2] [3]

Don’t miss out on your chance to get paid to work on open source software for the summer!

Google is going to pay students $4,500 to work on some of the coolest open source projects in the world. The OSU OSL has been accepted as a mentoring organization again for 2009. Interested in working with the OSL this summer? Check out our ideas page and submit a proposal!

The application deadline is April 3rd at noon Pacific time. Coding starts on May 23rd.

Are you going to the Open Source Bridge conference in Portland in June?  Hopefully I’ll see you there.  I’ve submitted a proposal to talk about what we’re doing here at the OSL to support the Oregon Virtual School District.

Hmm. Distinctly suboptimal. Perhaps this is a good time to play with Yahoo Pipes? I’d looked at it before, but had never gotten around to actually building a pipe. This seems like a nice simple thing to try.

1. Log into pipes.yahoo.com and create a new pipe.
2. Grab the RSS feed one wishes to work with. In my case, I grabbed the feed for all analyzed vulnerabilities: http://nvd.nist.gov/download/nvd-rss-analyzed.xml
3. Drag “Fetch Feed” from the “Sources” section of the left sidebar into the workspace and enter the feed URL.
4. Drag “Filter” from the “Operators” section into the workspace and connect the two objects.
5. As soon as the feed and the filter are connected, the rules item list is populated with all the fields available in the feed. Select the ones you want to filter on - item.description and item.title in my case - and add the filter criteria.
6. Connect the filter object to the pipe output object and watch the results show up in the debugger. If they look good, save the pipe and click “run pipe” at the top of the screen.
7. When the pipe is run, it spits out a nice preview of the contents along with buttons to grab the output in various formats including JSON and RSS.

Oh, my. That was easy. Now I have a feed that provides me with a listing of all analyzed vulnerabilities right at the top of my Google Reader subscriptions. I guess that means I need to take off the training wheels and do something a bit more challenging, eh?

What: TrustTheVote! Intro in Portland, Oregon
When: Feb 18, 2009, 6:00 - 7:30 p.m.,
Where: CubeSpace, 622 SE Grand Ave, Portland

Come see what’s new in the world of open digital voting!

So far I love the phone, but as a heavy Google Apps user, I was unhappy with the options for calendar, contacts and mail. The mobile Google interface is very good, but it bothered me that I couldn’t sync my Google calendars to the iPhone’s built-in calendar without plugging into iTunes and syncing via iCal subscriptions. Plus, the mobile Google Calendar event creation is - unlike just about everything else in the Google mobile suite - completely unusable. Well, at least I get SMS reminders and access to my Gmail accounts. I could live with that.

Then yesterday Google released Calendar and contact syncing for iPhone and Windows Mobile devices. O frabjous day!

After heeding their warnings that I should back up my contacts (the initial sync wipes the local contacts from the phone), I dove right in. The process was well-documented (with one exception I’ll get to below) and I was up and running syncing to my Google Apps account in remarkably short order. Contacts and calendar events show up remarkably quickly on the phone. The sync process is mostly transparent. I did run into a few small issues:

1. I didn’t clean up my Gmail contacts list before syncing. A bit of a mistake, since I didn’t bother keeping it accurate and as well-ordered as my local contacts. I didn’t keep many phone numbers in there, also, so I had to re-enter a few and there were quite a few duplicates that could be merged - one entry for a person with just a jabber address, another with just an e-mail, etc. If you set the mobile sync up, I highly recommend going through and cleaning up your Gmail contacts first.
2. Alerts - items on the Gcal with a pop-up alert set cause the sync’ed item to set a local alert on the phone. Excellent, now I can switch away from SMS alerts. My only quibble is that I normally have a Gcal tab up in my browser while I work, so I get alerted both on the screen and by my phone. Mildly annoying, but I guess I could just close the calendar tab when not using it.
3. Last, and most importantly, it’s not immediately obvious how to set up syncing subscribed calendars. Thanks to the excellent pointers by Jeff Forman, I was able to get my subscribed calendars syncing properly. Basically, once you’ve set up the sync, go to m.google.com with the phone browser and select the “Sync” section. If you’re a Google Apps user like me, there’s the additional step of tapping on the button at the bottom of the page that says “Google Apps user? Tap to configure your domain” and entering your domain name. Then you can access the sync settings for your domain account and select the calendars you want. If you don’t go to the main m.google.com site first and try going straight to m.google.com/sync you’ll do what I did and spend a frustrating few minutes failing to log in with your domain account. It’s a bit arcane and convoluted, but not difficult once you figure it out.

All in all, I’m delighted. I can now schedule events on any of my calendars, have local reminders, and see all my calendars in their full color-coded glory on my phone. Thanks Google!