Kalman Filter is often thought of as a linear filter where you have all model matrices but the idea of filter and its first applications come from non-linear models. In that case you use functions instead of matrices.
If the functions for prediction and update are highly non-linear you can use statistical methods to estimate your parameters on-line. The first look what you can take is unscented kalman filter which recovers mean and covariance from deterministic sampling technique - unscented transformation. I think in your case this could be the best to start with.
There are other variants of Kalman Filter. You can start from wikipedia but if you google "adaptive kalman filter" you can see the variety of the subject.
If you want to get deeper into the subject but not necessary start with all maths I recommend very good book: Kalman Filter for Beginners to start with by Phil Kim . There are also other possibility as sensor fusion, but it is another broad subject.