linux application crash due to Out of memory

Question

I developed an application for an embedded Linux system that does alot of UDP packet transferring. The application does all the memory allocation in the beginning of the application then reuses the allocated memory to avoid fragmentation.

I ran a massif pro-filer on the desktop, for both the stack and the heap and the value doesn't grow big.

However, running it on the embedded system for a very long time the application exits with the following error messages. Is this leaking heap or over usage of stack?

2013 Jun 18 16:30:52+00:00 sys-nam kernel [warning] sh invoked oom-killer: gfp_mask=0x4d0, order=0, oom_adj=0
2013 Jun 18 16:30:52+00:00 sys-nam kernel [warning] Call Trace:
2013 Jun 18 16:30:52+00:00 sys-nam kernel [warning] [<ffffffff8010f448>] dump_stack+0x8/0x34
2013 Jun 18 16:30:52+00:00 sys-nam kernel [warning] [<ffffffff802ff9e0>] T.659+0xd8/0x270
2013 Jun 18 16:30:52+00:00 sys-nam kernel [warning] [<ffffffff802ffcb8>] __out_of_memory+0x140/0x1f0
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff802ffddc>] out_of_memory+0x74/0xe8
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff80304588>] __alloc_pages_nodemask+0x718/0x730
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff8031ab18>] __pte_alloc+0x48/0x200
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff8031be9c>] handle_mm_fault+0x11cc/0x1698
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff8031c4e4>] __get_user_pages+0x17c/0x3f8
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff8034111c>] get_arg_page+0x3c/0xd0
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff8034131c>] copy_strings+0x16c/0x328
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff803414f0>] copy_strings_kernel+0x18/0x30
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff80387294>] compat_do_execve+0x214/0x3e8
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff80260188>] sys32_execve+0x38/0x68
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff80103ac4>] handle_sysn32+0x44/0x84
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] 
2013 Jun 18 16:30:53+00:00 sys-nam kernel [info] Mem-Info:
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] DMA32 per-cpu:
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] CPU    0: hi:  186, btch:  31 usd: 167
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] CPU    1: hi:  186, btch:  31 usd: 144
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] Normal per-cpu:
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] CPU    0: hi:   18, btch:   3 usd:  15
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] CPU    1: hi:   18, btch:   3 usd:   5
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] active_anon:6581 inactive_anon:6576 isolated_anon:0
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] active_file:3 inactive_file:30 isolated_file:0
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] unevictable:213762 dirty:1 writeback:0 unstable:0
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] free:969 slab_reclaimable:3240 slab_unreclaimable:4064
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] mapped:1590 shmem:31 pagetables:212 bounce:0
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] DMA32 free:3536kB min:3240kB low:4048kB high:4860kB active_anon:26324kB inactive_anon:26304kB active_file:100kB inactive_file:0kB unevictable:623504kB isolated(anon):0kB isolated(file):0kB present:729420kB mlocked:0kB dirty:0kB writeback:0kB mapped:1828kB shmem:124kB slab_reclaimable:5352kB slab_unreclaimable:13676kB kernel_stack:1152kB pagetables:848kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:239 all_unreclaimable? no
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] lowmem_reserve[]: 0 77 77
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] Normal free:420kB min:352kB low:440kB high:528kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:40kB unevictable:231544kB isolated(anon):0kB isolated(file):0kB present:79360kB mlocked:0kB dirty:4kB writeback:0kB mapped:4532kB shmem:0kB slab_reclaimable:7608kB slab_unreclaimable:2580kB kernel_stack:784kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] lowmem_reserve[]: 0 0 0
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] DMA32: 330*4kB 20*8kB 4*16kB 3*32kB 1*64kB 3*128kB 2*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4648kB
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] Normal: 70*4kB 16*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 408kB
2013 Jun 18 16:30:54+00:00 sys-nam kernel [warning] 213845 total pagecache pages
2013 Jun 18 16:30:54+00:00 sys-nam kernel [info] 261888 pages RAM
2013 Jun 18 16:30:54+00:00 sys-nam kernel [info] 25027 pages reserved
2013 Jun 18 16:30:54+00:00 sys-nam kernel [info] 9701 pages shared
2013 Jun 18 16:30:54+00:00 sys-nam kernel [info] 233184 pages non-shared
2013 Jun 18 16:30:54+00:00 sys-nam kernel [err] Out of memory: kill process 7747 (sh) score 7766 or a child
2013 Jun 18 16:30:54+00:00 sys-nam kernel [err] Killed process 7752 (wglADBPServer)

Answer 1

2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff802ffddc>] out_of_memory+0x74/0xe8
2013 Jun 18 16:30:53+00:00 sys-nam kernel [warning] [<ffffffff80304588>] __alloc_pages_nodemask+0x718/0x730

Most likely you are running out of heap space due to calls to new, malloc, allocate, or the like, as the error occurs during calls to the __alloc_pages_nodemask routine.

Look in your code for all instances of code that allocates heap memory and ensure that the memory is getting released when you are finished using it.