Great question!
In general this is certainly possible. Consider this module:
{-# LANGUAGE MultiParamTypeClasses, FunctionalDependencies, UndecidableInstances #-}
module M where
class C a b | a -> b where
f :: a -> b
instance C a b => C [a] [b]
where f = map f
It compiles by itself just fine. However, if you import this module and define
g x = x + f [x]
you'll get
Context reduction stack overflow; size = 201
Use -fcontext-stack=N to increase stack size to N
C [b] b
In the second argument of `(+)', namely `f [x]'
In the expression: x + f [x]
In an equation for `g': g x = x + f [x]
Regarding the mtl instances, I don't see how something like this is possible, but I also don't have a proof that it's not.