You cannot integrate custom codec to whole multimedia layer by just installing apk. To do so you have to recompile your own modified firmware. Which is complicated and not applicable for production (you cannot force everybody to install your Android OS version).
But you can develop your own multimedia player which will contain custom codec support (like MXPlayer app or any other). And then in manifest of your app you can mark support of this custom format (.steve), so Android will knew this app can handle this format.