Project structure for Google App Engine

Question

I started an application in Google App Engine right when it came out, to play with the technology and work on a pet project that I had been thinking about for a long time but never gotten around to starting. The result is BowlSK. However, as it has grown, and features have been added, it has gotten really difficult to keep things organized - mainly due to the fact that this is my first python project, and I didn't know anything about it until I started working.

What I have:

• Main Level contains:
  • all .py files (didn't know how to make packages work)
  • all .html templates for main level pages
• Subdirectories:
  • separate folders for css, images, js, etc.
  • folders that hold .html templates for subdirecty-type urls

Example:
http://www.bowlsk.com/ maps to HomePage (default package), template at "index.html"
http://www.bowlsk.com/games/view-series.html?series=7130 maps to ViewSeriesPage (again, default package), template at "games/view-series.html"

It's nasty. How do I restructure? I had 2 ideas:

• Main Folder containing: appdef, indexes, main.py?

  • Subfolder for code. Does this have to be my first package?
  • Subfolder for templates. Folder heirarchy would match package heirarchy
  • Individual subfolders for css, images, js, etc.

• Main Folder containing appdef, indexes, main.py?

  • Subfolder for code + templates. This way I have the handler class right next to the template, because in this stage, I'm adding lots of features, so modifications to one mean modifications to the other. Again, do I have to have this folder name be the first package name for my classes? I'd like the folder to be "src", but I don't want my classes to be "src.WhateverPage"

Is there a best practice? With Django 1.0 on the horizon, is there something I can do now to improve my ability to integrate with it when it becomes the official GAE templating engine? I would simply start trying these things, and seeing which seems better, but pyDev's refactoring support doesn't seem to handle package moves very well, so it will likely be a non-trivial task to get all of this working again.

Answer 1

First, I would suggest you have a look at "Rapid Development with Python, Django, and Google App Engine"

GvR describes a general/standard project layout on page 10 of his slide presentation.

Here I'll post a slightly modified version of the layout/structure from that page. I pretty much follow this pattern myself. You also mentioned you had trouble with packages. Just make sure each of your sub folders has an __init__.py file. It's ok if its empty.

Boilerplate files

• These hardly vary between projects
• app.yaml: direct all non-static requests to main.py
• main.py: initialize app and send it all requests

Project lay-out

• static/*: static files; served directly by App Engine
• myapp/*.py: app-specific python code
  • views.py, models.py, tests.py, __init__.py, and more
• templates/*.html: templates (or myapp/templates/*.html)

Here are some code examples that may help as well:

main.py

import wsgiref.handlers

from google.appengine.ext import webapp
from myapp.views import *

application = webapp.WSGIApplication([
  ('/', IndexHandler),
  ('/foo', FooHandler)
], debug=True)

def main():
  wsgiref.handlers.CGIHandler().run(application)

myapp/views.py

import os
import datetime
import logging
import time

from google.appengine.api import urlfetch
from google.appengine.ext.webapp import template
from google.appengine.api import users
from google.appengine.ext import webapp
from models import *

class IndexHandler(webapp.RequestHandler):
  def get(self):
    date = "foo"
    # Do some processing        
    template_values = {'data': data }
    path = os.path.join(os.path.dirname(__file__) + '/../templates/', 'main.html')
    self.response.out.write(template.render(path, template_values))

class FooHandler(webapp.RequestHandler):
  def get(self):
    #logging.debug("start of handler")

myapp/models.py

from google.appengine.ext import db

class SampleModel(db.Model):

I think this layout works great for new and relatively small to medium projects. For larger projects I would suggest breaking up the views and models to have their own sub-folders with something like:

Project lay-out

• static/: static files; served directly by App Engine
  • js/*.js
  • images/*.gif|png|jpg
  • css/*.css
• myapp/: app structure
  • models/*.py
  • views/*.py
  • tests/*.py
  • templates/*.html: templates

Answer 2

My usual layout looks something like this:

• app.yaml
• index.yaml
• request.py - contains the basic WSGI app
• lib
  • __init__.py - common functionality, including a request handler base class
• controllers - contains all the handlers. request.yaml imports these.
• templates
  • all the django templates, used by the controllers
• model
  • all the datastore model classes
• static
  • static files (css, images, etc). Mapped to /static by app.yaml

I can provide examples of what my app.yaml, request.py, lib/init.py, and sample controllers look like, if this isn't clear.

Answer 3

I implemented a google app engine boilerplate today and checked it on github. This is along the lines described by Nick Johnson above (who used to work for Google).

Follow this link gae-boilerplate

Answer 4

I think the first option is considered the best practice. And make the code folder your first package. The Rietveld project developed by Guido van Rossum is a very good model to learn from. Have a look at it: http://code.google.com/p/rietveld

With regard to Django 1.0, I suggest you start using the Django trunk code instead of the GAE built in django port. Again, have a look at how it's done in Rietveld.

Answer 5

I like webpy so I've adopted it as templating framework on Google App Engine.
My package folders are typically organized like this:

app.yaml
application.py
index.yaml
/app
   /config
   /controllers
   /db
   /lib
   /models
   /static
        /docs
        /images
        /javascripts
        /stylesheets
   test/
   utility/
   views/

Here is an example.

Answer 6

I am not entirely up to date on the latest best practices, et cetera when it comes to code layout, but when I did my first GAE application, I used something along your second option, where the code and templates are next to eachother.

There was two reasons for this - one, it kept the code and template nearby, and secondly, I had the directory structure layout mimic that of the website - making it (for me) a bit easier too remember where everything was.