Serveur dédié : installer et configurer Varnish 4

Cette semaine, j’ai décidé de mettre mon installation de Varnish à jour.

La version 3.0.5 date de décembre 2013 et il est temps de mettre le serveur à jour pour bénéficier des dernières nouveautés et corrections de bugs. Nous passons donc de Varnish 3 à Varnish 4.

Cela ne se fait pas sans peine car chez Varnish, ils renomment certaines directives d’une version à l’autre… ce qui fait planter le serveur Varnish puisqu’il ne reconnait plus les directives. Résultat : le fichier de configuration de la version précédente plantera obligatoirement sous la dernière version !

Ce tutoriel en 3 étapes nous donnera l’occasion de mettre à jour Varnish et de scinder notre fichier de configuration en plusieurs modules de manière à en simplifier l’édition et la maintenance futures.

Etape 1 : mise à jour des dépôts Varnish

Pour mettre à jour Varnish, il suffit de pointer apt vers les derniers dépôts à jour. On édite donc /etc/apt/sources.list :

nano /etc/apt/sources.list

et on y met à jour nos dépôts:

# varnish
deb http://repo.varnish-cache.org/debian/ wheezy varnish-4.0

On rafraîchit la liste des paquets et on lance la mise à jour :

apt-get update && apt-get upgrade

Varnish est maintenant mis à jour mais loin d’être fonctionnel étant donné que le format du fichier de configuration a changé.

Etape 2 : le nouveau fichier de configuration de Varnish 4 pour WordPress

Certaines directives ont changé de nom et, malgré avoir lu le guide de migration officiel, j’ai modifié mon fichier de configuration en corrigeant les erreurs une à une. Cela prend du temps mais au final, le fichier est plus clair qu’avant.

Voici donc mon nouveau fichier default.vcl optimisé pour WordPress :

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;
# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
.max_connections = 800;
}
backend example {
.host = "127.0.0.1";
.port = "8088";
}
import std;
include "xforward.vcl";
include "bigfiles.vcl";
include "mobile_cache.vcl";
include "mobile_pass.vcl";
include "static.vcl";
sub vcl_recv {
# Happens before we check if we have this in cache already.
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
if (req.http.host ~ "example.com") {
set req.backend_hint = example;
} elseif (req.http.host ~ "skyminds.net") {
set req.backend_hint = default;
}
# ---------------------- WORDPRESS SPECIFIC CONFIG -------------------- #
# Did not cache the RSS feed
if (req.url ~ "/feed") {
return (pass);
}
# Blitz hack
if (req.url ~ "/mu-.*") {
return (pass);
}
# Did not cache the admin and login pages
if (req.url ~ "/wp-(login|admin)") {
return (pass);
}
# First remove the Google Analytics added parameters, useless for our backend
if(req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=") {
set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
set req.url = regsub(req.url, "\?&", "?");
set req.url = regsub(req.url, "\?$", "");
}
# Remove the "has_js" cookie
set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
# Remove any Google Analytics based cookies
set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
# Remove the Quant Capital cookies (added by some plugin, all __qca)
set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
# Remove the wp-settings-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
# Remove the wp-settings-time-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
# Remove the wp test cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");
# remove cookies for comments cookie to make caching better.
set req.http.cookie = regsub(req.http.cookie, "dcd9527364a17bb2ae97db0ead3110ed=[^;]+(; )?", "");
# remove ?ver=xxxxx strings from urls so css and js files are cached.
set req.url = regsub(req.url, "\?ver=.*$", "");
# Remove "replytocom" from requests to make caching better.
set req.url = regsub(req.url, "\?replytocom=.*$", "");
# Strip hash, server doesn't need it.
set req.url = regsub(req.url, "\#.*$", "");
# Strip trailing ?
set req.url = regsub(req.url, "\?$", "");
# Are there cookies left with only spaces or that are empty?
if (req.http.cookie ~ "^ *$") {
unset req.http.cookie;
}	
# Drop any cookies sent to WordPress.
if (!(req.url ~ "wp-(login|admin)")) {
unset req.http.cookie;
}
# Cache the following files extensions
if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
unset req.http.cookie;
}
# Normalize Accept-Encoding header and compression
# https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
if (req.http.Accept-Encoding) {
# Do no compress compressed files...
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
unset req.http.Accept-Encoding;
}
}
# Check the cookies for wordpress-specific items
if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
return (pass);
}
if (!req.http.cookie) {
unset req.http.cookie;
}
# ---------------------- /WORDPRESS SPECIFIC CONFIG -------------------- #
# No cache for big video files
if (req.url ~ "\.(avi|mp4)") {
return (pass);
}   
# Do not cache HTTP authentication and HTTP Cookie
if (req.http.Authorization || req.http.Cookie) {
# Not cacheable by default
return (pass);
}
# Cache all others requests
return (hash);                     
}
sub vcl_backend_response {
# Happens after we have read the response headers from the backend.
# 
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
# Drop any cookies WordPress tries to send back to the client.
if (!(bereq.url ~ "wp-(login|admin)")) {
unset beresp.http.set-cookie;
}
}
sub vcl_deliver {
# Happens when we have all the pieces we need, and are about to send the
# response to the client.
# 
# You can do accounting or modifying the final object here.
if (obj.hits > 0) {
set resp.http.X-Cache = "cached";
} else {
set resp.http.x-Cache = "uncached";
}
# Remove some headers
unset resp.http.X-Powered-By;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Age;
unset resp.http.Link;
unset resp.http.Server;
return (deliver);
}
sub vcl_pipe {
# Note that only the first request to the backend will have
# X-Forwarded-For set. If you use X-Forwarded-For and want to
# have it set for all requests, make sure to have:
# set bereq.http.connection = "close";
# here. It is not set by default as it might break some broken web
# applications, like IIS with NTLM authentication.
#set bereq.http.Connection = "Close";
return (pipe);
}
# The data on which the hashing will take place
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
# hash cookies for requests that have them
if (req.http.Cookie) {
hash_data(req.http.Cookie);
}
}
sub vcl_hit {
return (deliver);
}
sub vcl_miss {
return (fetch);
}
sub vcl_init {
return (ok);
}
sub vcl_fini {
return (ok);
}

Etape 3 : fichiers de configuration supplémentaires

Avec cette nouvelle version, je me suis servi des fichiers de configuration de DreamHost disponibles sur Github.

Le fait de séparer certaines actions est plus simple à gérer : les objets statiques seront gérés par le fichier static.vcl, les gros fichiers par bigfiles.vcl et toute la configuration pour WordPress sera dans le fichier de configuration principal. C’est plus simple à maintenir.

Il m’a fallu modifier certains de ces fichiers donc je les redonne ici.

1. xforward.vcl permet d’ajouter l’entête X-Forwarded-For et normalise l’IP du client

On édite :

nano /etc/varnish/xforward.vcl

avec :

# xforward.vcl -- X-Forwarded-For HTTP Headers
#
# This should generally be loaded first to make sure that the headers
# get set appropriately for all requests.  Note that when using this
# you MUST NOT fall through to the VCL default handler for vcl_recv
# since that will run the code again, resulting in the client.ip
# being added twice.
sub vcl_recv {
if (req.restarts == 0) {
if (req.http.X-Forwarded-For) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
}

2. bigfiles.vcl s’occupe de transmettre les gros fichiers sans passer par le cache

On édite :

nano /etc/varnish/bigfiles.vcl

avec :

# bigfiles.vcl -- Bypass Cache for Large Files
#
# Copyright (C) 2013 DreamHost (New Dream Network, LLC)
#
# You must have "import std;" in your main vcl:
# import std;
sub vcl_backend_response {
# Bypass cache for files > 10 MB
if (std.integer(beresp.http.Content-Length, 0) > 10485760) {
set beresp.uncacheable = true;
set beresp.ttl = 120s;
return (deliver);
}
}

3. mobile_cache.vcl crée un cache à part pour les clients mobiles

On édite :

nano /etc/varnish/mobile_cache.vcl

avec :

# mobile_cache.vcl -- Separate cache for mobile clients
#
# Copyright (C) 2013 DreamHost (New Dream Network, LLC)
#
# If the User-Agent looks like a mobile device, then we add the string
# "mobile" to the hash_data.  This results in mobile devices having
# a separate cache from non-mobile devices.
#
# Note that if the backend does anything more sophisticated than having
# a "desktop" and a "mobile" version of pages (for example serving one
# page to iPhones and another to Androids), this will not be desirable.
# Also if the backend disagrees with this logic as far as what is a
# "mobile" User-Agent, then we may save the wrong version of pages in
# the cache.
sub vcl_hash {
# General User-Agent list (anything that remotely looks like a mobile device)
if (req.http.User-Agent ~ "(?i)ipod|android|blackberry|phone|mobile|kindle|silk|fennec|tablet|webos|palm|windows ce|nokia|philips|samsung|sanyo|sony|panasonic|ericsson|alcatel|series60|series40|opera mini|opera mobi|au-mic|audiovox|avantgo|blazer|danger|docomo|epoc|ericy|i-mode|ipaq|midp-|mot-|netfront|nitro|pocket|portalmmm|rover|sie-|symbian|cldc-|j2me|up\.browser|up\.link|vodafone|wap1\.|wap2\.") {
hash_data("mobile");
}
}

4. mobile_pass laisse passer les requêtes initiées par des mobiles ou tablettes

On édite :

nano /etc/varnish/mobile_pass.vcl

avec :

# mobile_pass.vcl -- Mobile pass-through support for Varnish
#
# Copyright (C) 2013 DreamHost (New Dream Network, LLC)
#
# This simply bypasses the cache for anything that looks like a mobile
# (or tablet) device.
# Also passes through some requests that are specifically for the WordPress
# JetPack mobile plugin.
sub vcl_recv {
# Rules specifically for the JetPack Mobile module
if (req.url ~ "\?(.*&)?(ak_action|app-download)=") {
return(pass);
}
if (req.http.Cookie ~ "(^|;\s*)akm_mobile=") {
return(pass);
}
# General User-Agent blacklist (anything that remotely looks like a mobile device)
if (req.http.User-Agent ~ "(?i)ipod|android|blackberry|phone|mobile|kindle|silk|fennec|tablet|webos|palm|windows ce|nokia|philips|samsung|sanyo|sony|panasonic|ericsson|alcatel|series60|series40|opera mini|opera mobi|au-mic|audiovox|avantgo|blazer|danger|docomo|epoc|ericy|i-mode|ipaq|midp-|mot-|netfront|nitro|pocket|portalmmm|rover|sie-|symbian|cldc-|j2me|up\.browser|up\.link|vodafone|wap1\.|wap2\.") {
return(pass);
}
}

5. static.vcl s’occupe de la mise en cache des objets statiques

On édite :

nano /etc/varnish/static.vcl

avec :

# static.vcl -- Static File Caching for Varnish
#
sub vcl_recv {
if (req.method ~ "^(GET|HEAD)$" && req.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)(\?.*)?$") {
if (req.url ~ "nocache") {
return(pass);
}
set req.url = regsub(req.url, "\?.*$", "");
unset req.http.Cookie;
return(hash);
}
# Added security, the "w00tw00t" attacks are pretty annoying so lets block it before it reaches our webserver
if (req.url ~ "^/w00tw00t"){
return( synth(403, "not permitted !"));	
}
}
sub vcl_backend_response {
if (bereq.method ~ "^(GET|HEAD)$" && bereq.url ~ "\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$") {
unset beresp.http.set-cookie;
set beresp.ttl = 24h;
set beresp.grace = 5m;
}
###	
if (beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
# set minimum timeouts to auto-discard stored objects
set beresp.grace = 5m;
# no cache for error pages
if ( beresp.status == 404 || beresp.status >= 500 ) {
set beresp.ttl = 0s;
return(deliver);
}
# Set 2min cache if unset for static files
if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") {
set beresp.ttl = 120s;
return (deliver);
}
# If WordPress cookies found then page is not cacheable
if (bereq.http.Cookie ~"(wp-postpass|wordpress_logged_in|comment_author_)") {
set beresp.ttl = 0s;
return(deliver);
}
# don't cache search results
if( bereq.url ~ "\?s=" ){
return(deliver);
}
return (deliver);
}

A titre de référence, je me suis également inspiré de :

• https://github.com/dreamhost/varnish-vcl-collection/tree/master/lib
• https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/master/default.vcl
• https://github.com/nicolargo/varnish-nginx-wordpress/blob/master/varnish/varnish4-wordpress

Conclusion

Voilà, vous venez de mettre Varnish à jour et l’organisation de la configuration en plusieurs fichiers rend la lecture plus lisible.

Tags : bash, cache, reverse-proxy, root, serveur dédié, terminal, Varnish