Can I “pre-generate” all possible static-html pages of my “dynamic” website?

Question

Some websites like for example http://www.idealo.co.uk seem to serve only static html, although their content is dynamic.

For example if I navigate through a certain category, I get a link to a static html page:

http://www.idealo.co.uk/cat/5666/electric-guitars.html

Now if I apply a custom filter, again I get a link to something that seems to be static html:

http://www.idealo.co.uk/cat/5666F456496-735760-1502100/electric-guitars.html

How is this achieved? Are there any frameworks out there that help to "pre-generate" all possible dynamic pages, in such way that whenever a new input is given, the page already exists (i.e. the static html is already available)?

Background: we run a small search engine for real estate offers. Offers are updated by our scrapper once a day (the content is static through the day). The content is searchable on a Ruby-on-Rails website. As the traffic increases, performance is becoming an issue. I'm wondering if there is any framework / tool that could batch-generate all our searches so that we could serve static html.

Answer 1

Their site isn't dynamic. They're using URL rewriting (e.g. mod_rewrite) to translate the input URLs into a request that can be satisfied by a script.

For example:

/cat/5666/electric-guitars.html

Might be rewritten to:

/cat.php?id=5666

A quick trick to test this is to go to /cat/5666/foo.html

The use of .html in this case is probably to hide what kind of scripting is used on their site, as a weak security-through-obscurity measure.

In response to your problem - no, there's no (easy) way to generate all possible results into static HTML files. You're looking at potentially billions of permutations. If you're having performance issues, look into performance profiling, caching, query optimisation, etc.

Answer 2

What you're describing is, in a sense, caching. With caching your application will generate pages (and even parts of pages) only when their content has changed. Rails has a lot of cache functionality built in, which you can tune to fit your needs. Start by reading the Rails Guide on caching which describes the Rails' capabilities as well as common add-ons. Google around for "Rails 3 caching"—there's tons of information out there. Finally, you can add software to your server stack that does additional caching, such as Squid and Varnish. With the right tools (and research) you can get 95% of the benefit of a static site without the effort of turning your site into a quasi-static Frankenapp by hand.

Answer 3

I finally found this blog post, which points to a few tools that do what I was looking for. I'm adding it here just for future reference:

Hyde

"Hyde is a static website generator powered by Python & Django. Hyde supports all the Django template tags & filters and even has a few of its own. The built-in web server + auto-generator provide instant refresh and unlimited flexibility..."

Jekyll

"Jekyll is a simple, blog-aware, static site generator. It takes a template directory containing raw text files in various formats, runs it through Markdown (or Textile) and Liquid converters, and spits out a complete, ready-to-publish static website suitable for serving with your favorite web server..."

blatter

"Blatter is a tiny tool for creating and publishing static web sites built from dynamic templates..."