Augmented reality with Optical character recognition

Question

I have implemented Augmented reality program using Qualcomm's vuforia library. Now I want to add Optical character recognition feature to my program so that i can translate the text from one language to another in real time. I am planning to use Tesseract OCR library. But my question is How do i Integrate Tesseract with QCAR? can some body suggest me proper way to do it?

Answer 1

What you need is an access to the camera frames, so you can send them to Tesseract. The Vuforia SDK offers a way to access the frames using the QCAR::UpdateCallback interface (documentation here).

What you need to do is create a class that implements this protocol, register it to the Vuforia SDK using the QCAR::registerCallback() (see here), and from there you'll get notified each time the Vuforia SDK has processed a frame.

This callback will be provided a QCAR::State object, from which you can get access to the camera frame (see the doc for QCAR::State::getFrame() here), and send it to the Tesseract SDK.

But be aware of the fact that the Vuforia SDK works with frames in a rather low resolution (on many phones I tested, it returns frames in the 360x240 to 720x480 range, and more often the former than the latter), which may not be accurate enough for Tesseract to detect text.

Answer 2

As complimentary information to @mbrenon 's answer: Tesseract only does text recognition and doesn't support ROI text extraction, so you will need to add that to your system after capturing your image.

You can read these academic papers which report on the additional steps for using Tesseract on mobile phones and provide some evaluation performances:

TranslatAR: Petter, M.; Fragoso, V.; Turk, M.; Baur, Charles, "Automatic text detection for mobile augmented reality translation," Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on , vol., no., pp.48,55, 6-13 Nov. 2011

Mobile Camera Based Detection and Translation