After some time, fabionuno answered my question here
In the code of TouchImageView.cs
SetOnTouchListener(new TouchImageViewListener(this));
should be changed to
base.SetOnTouchListener(new TouchImageViewListener(this));
Anyway, for large images TouchImageView proved to be very slow (at least in my case). For anyone, who would have similar problem, I recommend using WebView (or your own class inheriting WebView) implementing GestureListener (as stated in the Xamarin tutorial)