How can I pre-compress files with mod_deflate in Apache 2.x?

Question

I am serving all content through apache with Content-Encoding: zip but that compresses on the fly. A good amount of my content is static files on the disk. I want to gzip the files beforehand rather than compressing them every time they are requested.

This is something that, I believe, mod_gzip did in Apache 1.x automatically, but just having the file with .gz next to it. That's no longer the case with mod_deflate.

Answer 1

This functionality was misplaced in mod_gzip anyway. In Apache 2.x, you do that with content negotiation. Specifically, you need to enable MultiViews with the Options directive and you need to specify your encoding types with the AddEncoding directive.

Answer 2

To answer my own question with the really simple line I was missing in my confiuration:

Options FollowSymLinks MultiViews

I was missing the MultiViews option. It's there in the Ubuntu default web server configuration, so don't be like me and drop it off.

Also I wrote a quick Rake task to compress all the files.

namespace :static do
    desc "Gzip compress the static content so Apache doesn't need to do it on-the-fly."
    task :compress do
        puts "Gzipping js, html and css files."
        Dir.glob("#{RAILS_ROOT}/public/**/*.{js,html,css}") do |file|
            system "gzip -c -9 #{file} > #{file}.gz"
        end
    end
end

Answer 3

I am afraid MultiViews will not work as expected: the doc says Multiviews works "if the server receives a request for /some/dir/foo, if /some/dir has MultiViews enabled, and /some/dir/foo does not exist...", in other words: if you have a file foo.js and foo.js.gz in the same directory, just activating MultiViews will not cause the .gz file to be sent even if the AcceptEncoding gzip header is transmitted by the browser (you can verify this behavior by temporarily disabling mod_deflate and monitoring the response with e.g. HTTPFox).

I am not sure if there is a way around this with MultiViews (maybe you can rename the original file and then add a special AddEncoding directive), but I believe you can construct a mod_rewrite rule to handle this.

Answer 4

I have an Apache 2 built from source, and I found I had to modify the following in my httpd.conf file:

Add MultiViews to Options:

Options Indexes FollowSymLinks MultiViews

Uncomment AddEncoding:

AddEncoding x-compress .Z
AddEncoding x-gzip .gz .tgz

Comment AddType:

#AddType application/x-compress .Z
#AddType application/x-gzip .gz .tgz

Answer 5

It is possible to serve pre-compressed files using mod_negotiation although it is a bit finicky. The primary difficulty is that only requests for files which do not exist are negotiated. So if foo.js and foo.js.gz both exist, responses for /foo.js will always be uncompressed (although responses for /foo would work correctly).

The easiest solution I've found (from François Marier) is to rename uncompressed files with a double file extension, so foo.js is deployed as foo.js.js so requests for /foo.js negotiate between foo.js.js (no encoding) and foo.js.gz (gzip encoding).

I combine that trick with the following configuration:

Options +MultiViews
RemoveType .gz
AddEncoding gzip .gz

# Send .tar.gz without Content-Encoding: gzip
<FilesMatch ".+\.tar\.gz$">
    RemoveEncoding .gz
    # Note:  Can use application/x-gzip for backwards-compatibility
    AddType application/gzip .gz
</FilesMatch>

I wrote a post which discusses the reasoning for this configuration and some alternatives in detail.

Answer 6

mod_gzip compressed content on the fly as well. You can pre-compress the files by actually logging into your server, and doing it from shell.

cd /var/www/.../data/
for file in *; do
    gzip -c $file > $file.gz;
done;

Answer 7

You can use mod_cache to proxy local content in memory or on disk. I don't know if this will work as expected with mod_deflate.

Answer 8

I have a lot of big .json files. Most readers are in this situation. The preview answers didn't talk about the returned "Content-type".

I you want the following request return a pre-compressed file with "Content-Type: application/json" transparently, use Multiview with ForceType

http://www.domain.com/(...)/bigfile.json
-> Content-Encoding:gzip, Content-Type: Content-Encoding:gzip

1) files must be rename: "file.ext.ext"

2) Multiview works great with ForceType

In the file system:

// Note there is no bigfile.json
(...)/bigfile.json.gz
(...)/bigfile.json.json

In your apache config:

<Directory (...)>
    AddEncoding gzip .gz
    Options +Multiviews
    <Files *.json.gz>
        ForceType application/json
    </Files>
</Directory>

Short and simple :)