Memory use that grows and never drops back down is usually a sign that you are creating things and never discarding them when you are done.
I would guess that you are using storyboards, and simply invoking a new segue every time you want to display a new screen, but never popping view controllers/dismissing modals/doing unwind segues.
When you invoke a segue, iOS creates a new instance of the target view controller. Thus you can end up with an ever-increasing number of view controllers in memory, when what you meant to do was to dismiss a view controller and return to the previous view controller.