Static Resources and Cache-Busting on Google App Engine Python

Hello and happy new year!

In a couple of previous posts I discussed using Google Closure JavaScript Compiler in your build process to minimise the size of your JavaScript resources. Arguably a bigger win for the end user experience is to ensure the caching of all of your website static resources (JavaScript, stylesheets and images) is properly optimised. In this post I want to share how I think I have achieved this Google App Engine (Python).

Ideally, we would like the end user's browser to use its cache for every static resource, on every request, until that resource changes because we have deployed a new application version. Adding the (relatively new) default_expiration and expiration directives to your app.yaml file will achieve the first part of this requirement:

- url: /images
  static_dir: images
  expiration: "99d"

Behind the scenes, these directives add additional headers to the HTTP response, letting the browser know that they can safely serve these files from cache on subsequent requests.

But how do we tell the end user browser when our static resources have changed?


The solution is called cache-busting, and there are a number of similiar approaches. These involve changing the request URI for the static resources. This could mean appending a version identifier to the individual JS, CSS and image filenames. When the browser sees the request for a nominally different file it has no reason to pull the cached copy, so it requests the file again, with the specified caching headers in place again to prevent it being requested again:

First Request
URI: yourapp.blogspot.com\images\banner_v1.jpg (requested)

Second Request
URI: yourapp.appspot.com\images\banner_v1.jpg (not requested - cached)

Third Request
URI: yourapp.appspot.com\images\banner_v2.jpg (updated file - re-requested)

Fourth Request
URI: yourapp.appspot.com\images\banner_v2.jpg (not requested - cached)


This technique works, but how you consistently achieve it without introducing errors, confusing source control with trivial changes or simply absorbing too much development time is worth a discussion. Making manual changes to resource file names, the app.yaml static resource directives and the HTML for every application deployment is not very practical.

My approach to removing the development pain associated with cache busting works as follows: Instead of mangling the actual filename of the resource, I change the URI path. This means filenames themselves do not need to change. I also setup static file directives in the app.yaml such that the intermediate path variance (due to changing version) is ignored:

- url: /images/.*/(.*)
  static_files: static/img/\1
  upload: static/img/(.*)

Using a directives such as this, the position or naming of the actual static resource in the source code never changes. For example, the following URI's all serve the same static file:

yourapp.appspot.com/images/v1/banner.jpg » /static/img/banner.jpg
yourapp.appspot.com/images/v2/banner.jpg » /static/img/banner.jpg

Within the HTML (or any other resource specifying a URI) the source code contains a known token:

img src="/images/xx-replace-with-version-id-zz/banner.jpg"

My build script, written in Apache Ant, replaces the known token in the prepared build files with the version identifier. The original source files (whose known token works fine in development) are not updated:

<copy overwrite="true" todir="${path.build.src}">
    <fileset dir="${path.assets.src}">
      <include name="**/*.html" />
      <include name="**/*.css" />
    </fileset>
 
    <filterset begintoken="xx-" endtoken="-zz">
      <filter token="replace-with-version-id" value="${generated.id.timestamp}" />
    </filterset>
</copy>

The build script goes on to deploy the application either to its online test or production versions and then uses the same version identifier to tag newly deployed source code in my source control.

I know some developers prefer using a dynamically generated version identifier (using os.environ['CURRENT_VERSION_ID']) within their HTML or other URI carrying source. Since I already use a build script for JS compilation and Python linting, my approach of replacing a known token makes more sense to me - It removes complexity from my code and allows a full correspondence with the version identifier between the deployed version and the source control records.

I'd be interested in hearing if someone has an idea for improving this process or thinks another approach has distinct advantages I have not considered. If you have questions, leave them here as well.