The problem you are experiencing is known to the C++ committee. This problem is being tracked by LWG issue 2106. This issue is currently in Open status, meaning the resolution has not yet been decided upon.
Update
LWG issue 2106 was accepted in time for C++17. The PR is at the bottom of the issue (link corrected in this update).