Simply select the framework you want to delete in your list of frameworks in your Xcode project (which you see I've done on the left side of my Test project) and you should see this dialog when you click the "delete" key:
Just click the remove reference button.
Do NOT click the "Move To Trash" button, or else you'll get the nasty surprise of losing frameworks out of your installed copy of Xcode.