Simply put, it's very likely not really your fault. The iOS low memory killer (a rather aggressive kernel thread by the name of Jetsam) uses "priority bands" to maintain candidate processes to kill on low memory, and virtually everyone is a candidate - with background processes being likelier than foreground ones. . (Image taken from Priority bands in Jetsam, from http://newosxbook.com/articles/MemoryPressure.html)
What you can do is try to gain some more breathing room by responding to low memory events (i.e. implement didReceiveLowMemoryWarning) and unload any Images/UIImageViews, etc you aren't strictly requiring - even if you do a little, it helps you survive. Also, you didn't point out which UIBackgroundMode you are using - that might help (it defines app profile in kernel, thereby taming Jetsam a little).