Measuring Task Queue Costs in Google App Engine

Question

I am measuring the cost of requests to GAE by inspecting the x-appengine-estimated-cpm-us-dollars header. This works great and in combination with x-appengine-resource-usage and x-traceurl I can even get more detailed information.

However, a large part of my application run in the context of task queues. Thus, a huge part of the instance hour costs are consumed by queues. Each time code is executed outside of a request its costs are not included in the x-appengine-estimated-cpm-us-dollars header.

I am looking for a way to measure the full costs consumed by each request. I.e. costs generated by the request itself and the cost of the tasks that have been added by this request.

Answer 1

It is an overkill. There is a tool you can download google app engine log and convert them to sqlite. http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py

With this tool, cpm usd for both task request and normal request would be all downloaded together. You can store daily log into separate sqlite file and do as much analysis as you want.

In terms of relate the cost of task back to original request. The log data downloaded with this tool includes the full output of logging module.

1. So you can simply logging an generate id in the original request
2. pass the id to task.
3. logging the received id again in the task request.
4. find normal and task request pair via id.

for example:

# in org request
a_id = genereate_a_random_id() 
logging.info(a_id) # the id will be included 

taskqueue.add(url='/path_to_task', params={'id': a_id})


# in task request
a_id = self.request.get('id')
logging.info(a_id)

EDIT1

I think there is another possible way to estimate the cost of normal request + task request. The trick is change the async task to sync (assume the cost would be the same). I didn't try it but it is much easier to try.

# in org request, add a variable to identify debug
debug = self.request.get('DEBUG')

if debug:
    self.redirect('/path_to_task')
else:
    taskqueue.add(url='/path_to_task')

Thus, while testing the normal request with DEBUG parameter. It will firstly process the normal request then return x-appengine-estimated-cpm-us-dollars for normal request. Later it will redirect your test client to the relative task request (task request could also be access and trigger via url client as normal request) and return x-appengine-estimated-cpm-us-dollars for task request. You can simply add them together to get the total cost.