I would guess the JITter is trying to find common sub-expressions, etc. I think it is worth attempting to convert it to a Dictionary(Of <lookupType>, Func(Of <ParameterTypes>, <ReturnType>))
as that would also mean the JITter only compiles the cases actually used. Of course, if your individual delegates end up being based upon large closures, that probably explains part of your original performance issue.
Alternatively, and to partially confirm what the JITter is doing, don't change the switch
code, but in each case call a delegate that evaluates the result for that case. This will force the JITter to not inline the call and thus not attempt many optimisations. In fact, you can probably avoid the delegate call and just supply each case in a subroutine with the System.Runtime.CompilerServices.MethodImpl(MethodImplOptions.NoInlining)
attribute.