Yes, it's quite slow. OpenGL ES 2.0 doesn't provide a faster way on Android. You could use PBOs on OpenGL ES 3.0 but that may not be an option for you, since very few devices support that at the moment and it requires Android 4.3.
EDIT: I do know that some devices have serious performance issues uploading RGB textures, maybe it happens in the opposite direction too. Try using RGBA format instead.