Serveur dédié : installer et configurer Varnish 4

Cette semaine, j’ai décidé de mettre mon installation de Varnish à jour.

La version 3.0.5 date de décembre 2013 et il est temps de mettre le serveur à jour pour bénéficier des dernières nouveautés et corrections de bugs. Nous passons donc de Varnish 3 à Varnish 4.

Cela ne se fait pas sans peine car chez Varnish, ils renomment certaines directives d’une version à l’autre… ce qui fait planter le serveur Varnish puisqu’il ne reconnait plus les directives. Résultat : le fichier de configuration de la version précédente plantera obligatoirement sous la dernière version !

Ce tutoriel en 3 étapes nous donnera l’occasion de mettre à jour Varnish et de scinder notre fichier de configuration en plusieurs modules de manière à en simplifier l’édition et la maintenance futures.

Etape 1 : mise à jour des dépôts Varnish

Pour mettre à jour Varnish, il suffit de pointer apt vers les derniers dépôts à jour. On édite donc /etc/apt/sources.list :

nano /etc/apt/sources.list

et on y met à jour nos dépôts:

# varnish
deb http://repo.varnish-cache.org/debian/ wheezy varnish-4.0

On rafraîchit la liste des paquets et on lance la mise à jour :

apt-get update && apt-get upgrade

Varnish est maintenant mis à jour mais loin d’être fonctionnel étant donné que le format du fichier de configuration a changé.

Etape 2 : le nouveau fichier de configuration de Varnish 4 pour WordPress

Certaines directives ont changé de nom et, malgré avoir lu le guide de migration officiel, j’ai modifié mon fichier de configuration en corrigeant les erreurs une à une. Cela prend du temps mais au final, le fichier est plus clair qu’avant.

Voici donc mon nouveau fichier default.vcl optimisé pour WordPress :

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
	.host = "127.0.0.1";
	.port = "8080";
	.connect_timeout = 600s;
	.first_byte_timeout = 600s;
	.between_bytes_timeout = 600s;
	.max_connections = 800;
}

backend example {
.host = "127.0.0.1";
.port = "8088";
}

import std;
include "xforward.vcl";
include "bigfiles.vcl";
include "mobile_cache.vcl";
include "mobile_pass.vcl";
include "static.vcl";

sub vcl_recv {
    # Happens before we check if we have this in cache already.
    # Typically you clean up the request here, removing cookies you don't need,
    # rewriting the request, etc.
    
    if (req.http.host ~ "example.com") {
        set req.backend_hint = example;
    } elseif (req.http.host ~ "skyminds.net") {
        set req.backend_hint = default;
    }
                 
        # ---------------------- WORDPRESS SPECIFIC CONFIG -------------------- #
	# Did not cache the RSS feed
	if (req.url ~ "/feed") {
	return (pass);
	}
	# Blitz hack
	if (req.url ~ "/mu-.*") {
	return (pass);
	}
	# Did not cache the admin and login pages
	if (req.url ~ "/wp-(login|admin)") {
	return (pass);
	}
	
	# First remove the Google Analytics added parameters, useless for our backend
	if(req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=") {
		set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
		set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
		set req.url = regsub(req.url, "\?&", "?");
		set req.url = regsub(req.url, "\?$", "");
	}
	# Remove the "has_js" cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
	# Remove any Google Analytics based cookies
	set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
	# Remove the Quant Capital cookies (added by some plugin, all __qca)
	set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
	# Remove the wp-settings-1 cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
	# Remove the wp-settings-time-1 cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
	# Remove the wp test cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");
	# remove cookies for comments cookie to make caching better.
	set req.http.cookie = regsub(req.http.cookie, "dcd9527364a17bb2ae97db0ead3110ed=[^;]+(; )?", "");
	
	# remove ?ver=xxxxx strings from urls so css and js files are cached.
	set req.url = regsub(req.url, "\?ver=.*$", "");
	# Remove "replytocom" from requests to make caching better.
	set req.url = regsub(req.url, "\?replytocom=.*$", "");
	# Strip hash, server doesn't need it.
	set req.url = regsub(req.url, "\#.*$", "");
	# Strip trailing ?
	set req.url = regsub(req.url, "\?$", "");

	# Are there cookies left with only spaces or that are empty?
	if (req.http.cookie ~ "^ *$") {
	unset req.http.cookie;
	}	
	# Drop any cookies sent to WordPress.
	if (!(req.url ~ "wp-(login|admin)")) {
                       unset req.http.cookie;
        }
        # Cache the following files extensions
	if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
	unset req.http.cookie;
	}
	
	# Normalize Accept-Encoding header and compression
	# https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
	if (req.http.Accept-Encoding) {
	# Do no compress compressed files...
	if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
	unset req.http.Accept-Encoding;
	} elsif (req.http.Accept-Encoding ~ "gzip") {
	set req.http.Accept-Encoding = "gzip";
	} elsif (req.http.Accept-Encoding ~ "deflate") {
	set req.http.Accept-Encoding = "deflate";
	} else {
	unset req.http.Accept-Encoding;
	}
	}
	
	# Check the cookies for wordpress-specific items
	if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
	return (pass);
	}
	if (!req.http.cookie) {
	unset req.http.cookie;
	}
	# ---------------------- /WORDPRESS SPECIFIC CONFIG -------------------- #
	
	# No cache for big video files
	if (req.url ~ "\.(avi|mp4)") {
		return (pass);
	}   
	
	# Do not cache HTTP authentication and HTTP Cookie
	if (req.http.Authorization || req.http.Cookie) {
	# Not cacheable by default
	return (pass);
	}
	# Cache all others requests
	return (hash);                     
}

sub vcl_backend_response {
    # Happens after we have read the response headers from the backend.
    # 
    # Here you clean the response headers, removing silly Set-Cookie headers
    # and other mistakes your backend does.
    
    # Drop any cookies WordPress tries to send back to the client.
    if (!(bereq.url ~ "wp-(login|admin)")) {
                       unset beresp.http.set-cookie;
            }
            
}

sub vcl_deliver {
    # Happens when we have all the pieces we need, and are about to send the
    # response to the client.
    # 
    # You can do accounting or modifying the final object here.
    
	if (obj.hits > 0) {
	set resp.http.X-Cache = "cached";
	} else {
	set resp.http.x-Cache = "uncached";
	}

	# Remove some headers
	unset resp.http.X-Powered-By;
	unset resp.http.X-Varnish;
	unset resp.http.Via;
	unset resp.http.Age;
	unset resp.http.Link;
	unset resp.http.Server;
	return (deliver);
}

sub vcl_pipe {
	# Note that only the first request to the backend will have
	# X-Forwarded-For set. If you use X-Forwarded-For and want to
	# have it set for all requests, make sure to have:
	# set bereq.http.connection = "close";
	# here. It is not set by default as it might break some broken web
	# applications, like IIS with NTLM authentication.
	#set bereq.http.Connection = "Close";
	return (pipe);
}

# The data on which the hashing will take place
sub vcl_hash {
	hash_data(req.url);
	if (req.http.host) {
		hash_data(req.http.host);
	} else {
		hash_data(server.ip);
	}
	# hash cookies for requests that have them
	if (req.http.Cookie) {
		hash_data(req.http.Cookie);
	}
}
sub vcl_hit {
	return (deliver);
}
sub vcl_miss {
	return (fetch);
}
sub vcl_init {
	return (ok);
}
sub vcl_fini {
	return (ok);
}

Etape 3 : fichiers de configuration supplémentaires

Avec cette nouvelle version, je me suis servi des fichiers de configuration de DreamHost disponibles sur Github.

Le fait de séparer certaines actions est plus simple à gérer : les objets statiques seront gérés par le fichier static.vcl, les gros fichiers par bigfiles.vcl et toute la configuration pour WordPress sera dans le fichier de configuration principal. C’est plus simple à maintenir.

Il m’a fallu modifier certains de ces fichiers donc je les redonne ici.

1. xforward.vcl permet d’ajouter l’entête X-Forwarded-For et normalise l’IP du client

On édite :

nano /etc/varnish/xforward.vcl

avec :

# xforward.vcl -- X-Forwarded-For HTTP Headers
#
# This should generally be loaded first to make sure that the headers
# get set appropriately for all requests.  Note that when using this
# you MUST NOT fall through to the VCL default handler for vcl_recv
# since that will run the code again, resulting in the client.ip
# being added twice.
sub vcl_recv {
	if (req.restarts == 0) {
		if (req.http.X-Forwarded-For) {
			set req.http.X-Forwarded-For =
				req.http.X-Forwarded-For + ", " + client.ip;
		} else {
			set req.http.X-Forwarded-For = client.ip;
		}
	}
}

2. bigfiles.vcl s’occupe de transmettre les gros fichiers sans passer par le cache

On édite :

nano /etc/varnish/bigfiles.vcl

avec :

# bigfiles.vcl -- Bypass Cache for Large Files
#
# Copyright (C) 2013 DreamHost (New Dream Network, LLC)
#
# You must have "import std;" in your main vcl:
# import std;

sub vcl_backend_response {
# Bypass cache for files > 10 MB
if (std.integer(beresp.http.Content-Length, 0) > 10485760) {
	set beresp.uncacheable = true;
        set beresp.ttl = 120s;
        return (deliver);
}
}

3. mobile_cache.vcl crée un cache à part pour les clients mobiles

On édite :

nano /etc/varnish/mobile_cache.vcl

avec :

# mobile_cache.vcl -- Separate cache for mobile clients
#
# Copyright (C) 2013 DreamHost (New Dream Network, LLC)
#
# If the User-Agent looks like a mobile device, then we add the string
# "mobile" to the hash_data.  This results in mobile devices having
# a separate cache from non-mobile devices.
#
# Note that if the backend does anything more sophisticated than having
# a "desktop" and a "mobile" version of pages (for example serving one
# page to iPhones and another to Androids), this will not be desirable.
# Also if the backend disagrees with this logic as far as what is a
# "mobile" User-Agent, then we may save the wrong version of pages in
# the cache.
sub vcl_hash {
	# General User-Agent list (anything that remotely looks like a mobile device)
	if (req.http.User-Agent ~ "(?i)ipod|android|blackberry|phone|mobile|kindle|silk|fennec|tablet|webos|palm|windows ce|nokia|philips|samsung|sanyo|sony|panasonic|ericsson|alcatel|series60|series40|opera mini|opera mobi|au-mic|audiovox|avantgo|blazer|danger|docomo|epoc|ericy|i-mode|ipaq|midp-|mot-|netfront|nitro|pocket|portalmmm|rover|sie-|symbian|cldc-|j2me|up\.browser|up\.link|vodafone|wap1\.|wap2\.") {
		hash_data("mobile");
	}
}

4. mobile_pass laisse passer les requêtes initiées par des mobiles ou tablettes

On édite :

nano /etc/varnish/mobile_pass.vcl

avec :

# mobile_pass.vcl -- Mobile pass-through support for Varnish
#
# Copyright (C) 2013 DreamHost (New Dream Network, LLC)
#
# This simply bypasses the cache for anything that looks like a mobile
# (or tablet) device.
# Also passes through some requests that are specifically for the WordPress
# Jetpack mobile plugin.
sub vcl_recv {
	# Rules specifically for the Jetpack Mobile module
	if (req.url ~ "\?(.*&)?(ak_action|app-download)=") {
		return(pass);
	}
	if (req.http.Cookie ~ "(^|;\s*)akm_mobile=") {
		return(pass);
	}

	# General User-Agent blacklist (anything that remotely looks like a mobile device)
	if (req.http.User-Agent ~ "(?i)ipod|android|blackberry|phone|mobile|kindle|silk|fennec|tablet|webos|palm|windows ce|nokia|philips|samsung|sanyo|sony|panasonic|ericsson|alcatel|series60|series40|opera mini|opera mobi|au-mic|audiovox|avantgo|blazer|danger|docomo|epoc|ericy|i-mode|ipaq|midp-|mot-|netfront|nitro|pocket|portalmmm|rover|sie-|symbian|cldc-|j2me|up\.browser|up\.link|vodafone|wap1\.|wap2\.") {
		return(pass);
	}
}

5. static.vcl s’occupe de la mise en cache des objets statiques

On édite :

nano /etc/varnish/static.vcl

avec :

# static.vcl -- Static File Caching for Varnish
#
sub vcl_recv {
	if (req.method ~ "^(GET|HEAD)$" && req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)(\?.*)?$") {
		if (req.url ~ "nocache") {
			return(pass);
		}
		set req.url = regsub(req.url, "\?.*$", "");
		unset req.http.Cookie;
		return(hash);
	}
	
		# Added security, the "w00tw00t" attacks are pretty annoying so lets block it before it reaches our webserver
	if (req.url ~ "^/w00tw00t"){
	     return( synth(403, "not permitted !"));	
	}
}

sub vcl_backend_response {
	if (bereq.method ~ "^(GET|HEAD)$" && bereq.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") {
		unset beresp.http.set-cookie;
		set beresp.ttl = 24h;
		set beresp.grace = 5m;
	}
	
	###	
	if (beresp.http.content-type ~ "text") {
              set beresp.do_gzip = true;
        }

        # set minimum timeouts to auto-discard stored objects
        set beresp.grace = 5m;
	# no cache for error pages
	if ( beresp.status == 404 || beresp.status >= 500 ) {
		set beresp.ttl = 0s;
		return(deliver);
	}
        # Set 2min cache if unset for static files
	if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") {
	        set beresp.ttl = 120s;
	        return (deliver);
	}
	# If WordPress cookies found then page is not cacheable
	if (bereq.http.Cookie ~"(wp-postpass|wordpress_logged_in|comment_author_)") {
		set beresp.ttl = 0s;
		return(deliver);
	}
	# don't cache search results
	if( bereq.url ~ "\?s=" ){
		 return(deliver);
	}

	return (deliver);
}

A titre de référence, je me suis également inspiré de :

• https://github.com/dreamhost/varnish-vcl-collection/tree/master/lib
• https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
• https://github.com/nicolargo/varnish-nginx-wordpress/blob/master/varnish/varnish4-wordpress

Conclusion

Voilà, vous venez de mettre Varnish à jour et l'organisation de la configuration en plusieurs fichiers rend la lecture plus lisible.

Articles en rapport:

• Serveur dédié : retirer Varnish, devenu inutile avec HTTPS
• Serveur dédié : installer APC comme système de cache et configurer Varnish comme reverse-proxy pour Apache pour améliorer les performances
• WordPress : résoudre le problème de la table wp_options à qui manquent une colonne Unique et une Primary Key
• Serveur dédié : remplacer gzip par pigz pour profiter de la compression multi-core
• BASH : supprimer les fichiers de session PHP obsolètes
• Serveur dédié : mise à jour vers PHP7.1 sous Debian
• Créer une clé SSH pour ouvrir une session distante sans mot de passe
• PHP : résoudre l’erreur “PHP Fatal error: Uncaught Error: Class DOMDocument”