How to send a pygame image over websockets?

StackOverflow https://stackoverflow.com/questions/18409464

Question

I'm currently trying to code something that will let websites view my webcam. I'm roughly following the tutorial linked on this website, except using Python and pygame instead of Processing.

At the moment, my code is grabbing a pygame image (which was originally a SimpleCV image), attempting to convert it into jpg format, and send it over websockets to the client where it will display it inside an img tag. However, I can't seem to figure out how to convert a pygame image into jpg and get it to display properly on the web browser.

This is my code for the server, which uses Flask and gevent:

#!/usr/bin/env python

import base64
import cStringIO
import time

from geventwebsocket.handler import WebSocketHandler
from gevent.pywsgi import WSGIServer
from flask import Flask, request, render_template


import pygame
pygame.init()

import SimpleCV as scv

app = Flask(__name__)
cam = scv.Camera(0)

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/camera')
def camera():

    if request.environ.get('wsgi.websocket'):
        ws = request.environ['wsgi.websocket']

        while True:            
            image = cam.getImage().flipHorizontal().getPGSurface()
            data = cStringIO.StringIO()
            pygame.image.save(image, data)
            ws.send(base64.b64encode(data.getvalue()))
            time.sleep(0.5)

if __name__ == '__main__':
    http_server = WSGIServer(('',5000), app, handler_class=WebSocketHandler)
    http_server.serve_forever()

This is my HTML file:

<!DOCTYPE HTML>
<html>
<head>
<title>Flask/Gevent WebSocket Test</title>
    <script src="http://code.jquery.com/jquery-1.10.1.min.js"></script>
    <script type="text/javascript" charset="utf-8">
        $(document).ready(function(){
            if ("WebSocket" in window) {
                cam = new WebSocket("ws://" + document.domain + ":5000/camera");
                cam.onmessage = function (msg) {
                    $("#cam").attr('src', 'data:image/jpg;base64,' + msg.data);
                };
                cam.onerror = function(e) {
                    console.log(e);
                }
            } else {
                alert("WebSocket not supported");
            }
        });
    </script>
</head>
<body>
    <img id="cam" src="" width="640" height="480" />
</body>
</html>

These are the specific lines that I think I'm having trouble with:

while True:
    image = cam.getImage().flipHorizontal().getPGSurface()
    data = cStringIO.StringIO()
    pygame.image.save(image, data)
    ws.send(base64.b64encode(data.getvalue()))
    time.sleep(0.5)

Currently, if I try and run my code, going to localhost:5000 will display an invalid jpg image. It also becomes really laggy if I try running it on Firefox, but that may be an unrelated issue that I can debug later.

I've checked and made sure that the pygame image is a valid one, since I'm converting it from another library, and also checked that I was using websockets correctly by sending text data back and forth.

I've also tried calling pygame.image.to_string to try and convert the pygame surface into RGB format, but that also doesn't work.

What am I doing wrong?

Was it helpful?

Solution

Using the underlying PIL image, we can write to a file-like object, read back and base-64 encode it:

from geventwebsocket.handler import WebSocketHandler
from gevent.pywsgi import WSGIServer
from flask import Flask, request
from time import sleep
from cStringIO import StringIO

import pygame
pygame.init()

import SimpleCV as scv

app = Flask(__name__)
cam = scv.Camera(0)


@app.route('/camera')
def camera():

    if request.environ.get('wsgi.websocket'):
        ws = request.environ['wsgi.websocket']

        while True:
            fp = StringIO()
            image = cam.getImage().flipHorizontal().getPIL()
            image.save(fp, 'JPEG')
            ws.send(fp.getvalue().encode("base64"))
            #fp.close() << benchmark and memory tests needed
            sleep(0.5)


if __name__ == '__main__':
    http_server = WSGIServer(('',5000), app, handler_class=WebSocketHandler)
    http_server.serve_forever()

OTHER TIPS

I'm fighting with the same issue and the problem is a double codification. In the Python file, you have to remove the line "ws.send(base64.b64encode(data.getvalue()))" and send the image without encoded. Then in the js file, your script will make the codification and that all.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top