There is a bug in Mavericks. If you create an NSButton in XIB and give it a layer and set its shadow it works fine, but if you create one programmatically it clips its shadow.
I suspect the AppKit team did some crazy hacks when they made buttons do fast layer compositing in Mavericks (they won't redraw their backgrounds unless necessary now, for instance), because they tried to make it all happen magically, which is always always a bad idea.
Note that if you make an NSTextField the shadow code works as you'd expect. It's just NSButtons (so far) that I've found have this issue.
Please file a radar.