To have an effective 'mask', you should use a layer with touch enabled as foreground. In that layer, you can accept touches outside the menu, and do nothing with them, and pass those inside the mask. However, this will cause issues : if the touchBegan inside the mask, and the user drags its finger outside, the menu will still continue scrolling. You can either subclass CCAdvancedMenu to implement your own touch processing requirements, or create your own version of it. I ended up doing the latter because my game designer wanted many other features from it and subclassing became a hindrance rather than a help. BTW, dont rely on zOrder for the touch processing, they are not related.
Also, the 'touch mask' approach will probably come back and haunt you later on (for example if you add other objects alongside your scrolling menu that also need touch, the touches will be masked out), it would be best if all your objects that are concerned with implementing touch related requirements encapsulate their own implementation.