I'd suggest to solve it with separating the layers. The library has it's own specializations and the user can overwrite them if needed. If this is acceptable, the following is the library code:
namespace impl
{
template <class A, class B, class Enable=void>
struct Widget;
}
template <class A, class B, class Enable=void>
struct Widget : impl::Widget< A, B > {};
struct Foo
{
};
namespace impl
{
template <class B>
struct Widget<Foo, B>
{
};
}
and the user code stays as it is.