In jellybean the user needs to download the offline speech recognition package.
This article sais:
Previously, when you pressed the voice icon and spoke a command or query, Android had to digitize your voice, upload it to the cloud, process the waveform, turn it into text, and send the text back down to your phone. Now the phones are powerful enough that this can be built into the device, with no extra network I/O needed. As you can imagine this results in much faster voice recognition than previous versions.
The app user will have to do this:
- Go to “Language and Input” in the Setting
- Tap on "Download offline speech recognition" under the "Voice Search"
- Choose the language pack you want your Android device to recognize
- Download the pack and enjoy the offline voice typing
Another helper link:
Google have restricted certain Jelly Bean devices from using the offline recognition due to hardware constraints.