I dont think that is possible, even by trying to manipulate the transforms Microsoft is using for the LibraryStackItems. i have tried it myself and nothing was working. (Manipulating the RotateTransform, adding another RotateTransform to negate the initial RotateTransform etc.). Event if you get the initial Layout with zero Rotation, some weired logic will mess up your changes. And thats just the RotateTransform. There is a TranslateTransform too.. (and a ScaleTransform and a ScewTransform)
I think we just have to accept, thats how the control works and that its not appropriate to manipulate the rotation.
So here is my advise, based on many projects with the SurfaceSDK on Microsoft PixelSense itself and the Surface-Touch-Pack:
Use a single ScatterView for both of your requirements: The stack and the cards removed by the user. A stack is easily coded with a ScatterView and the Rotation can be set with the Orientation-property. And as a nice side-effect, it is really easy to handle the transitations between the stack and the opened cards.