Invalid characters (&) in "where" request to Eve

Question

I found this problem in my development system and have reproduced it in the Eve demo found here

This the code I run.

    import requests
    import json
    import string
    import random
    def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
        return ''.join(random.choice(chars) for _ in range(size))

    name = "max and me"
    data = {"lastname":id_generator(), "firstname":name}

    print data
    res = requests.post("http://127.0.0.1:5000/people", data = data)
    print res.text

    data = 'where={"firstname":"%s"}' % (name)
    res = requests.get("http://127.0.0.1:5000/people", params = data)
    print res.text

first I make a POST including the variable , and the I do a GET with a where on the same variable . First I set the variable to "max and me" and all runs fine Then I run the with set to "max & me" and eve throws up all over me:

    127.0.0.1 - - [23/Apr/2014 20:22:07] "GET /people?where=%7B%22firstname%22:%22max%20&%20me%22%7D HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/eve/endpoints.py", line 53, in collections_endpoint
    response = get(resource, lookup)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/eve/methods/common.py", line 226, in rate_limited
    return f(*args, **kwargs)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/eve/auth.py", line 45, in decorated
    return f(*args, **kwargs)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/eve/methods/common.py", line 429, in decorated
    r = f(*args, **kwargs)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/eve/methods/get.py", line 104, in get
    cursor = app.data.find(resource, req, lookup)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/eve/io/mongo/mongo.py", line 145, in find
    spec = parse(req.where)
  File "/Users/hingem/Documents/python/venv/lib/python2.7/site-packages/eve/io/mongo/parser.py", line 26, in parse
    v.visit(ast.parse(expression))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 1
    {"firstname":"max
                     ^
SyntaxError: EOL while scanning string literal

It's probably me not handling the <&> symbol probably - but I feel that I have tried all sorts of encoding...And I get stuck....every time

Answer 1

Try replacing & with its escaped version:%26. I just tried this GET to a Eve-powered API

?where={"name": "You %26 me"}

and it worked just fine.

Answer 2

I think you cannot have ampersand in unencoded way in the URL - it means next part of query in URL. Usually you have to encode ampersand as %26.

I think this is partially bug of eve, because if you put unencoded ampersand into query part of URL, Flask (which is the base of Eve) splits it into request args and Eve then tries to make pythonic ast of the "where" part (which is the way it handles query formats in Python syntax), which ends just after the max and in consequence it raises this error, because its obviously not correct python expression. In this code https://github.com/nicolaiarocci/eve/blob/develop/eve/io/mongo/mongo.py#L151 there should be catched probably SyntaxError too. Maybe you can try to fill in bug on eve's github and suggest that eve should return 400 bad request.

I hope this post explains the core of the problem.