Debugging page allocation failure on Coldfire uCLinux

Question

I'm sometimes getting this crash output below on my Coldfire uCLinux system. How do I work out what's causing the problem?

Apr  4 10:44:33 (none) user.debug syslog: starting NTP
sh: page allocation failure. order:8, mode:0xd0
Stack from 41da5dcc:
        4005b0f2 400553b6 40207431 406131f8 00000008 000000d0 00000008 00000000
        000000a2 000a2000 000a2000 0000000c 40544a14 00000000 405434fc 00000077
        41da5eac 00000000 00000010 00000000 41da5008 41da5000 00000000 00000100
        00000000 41da5000 00000000 000200d0 4024eecc 00000080 00000000 00000000
        4005de52 000000d0 00000008 4024eec8 00000000 00000001 00004d09 00079100
        00000004 00003f20 00013424 41cd7000 41da5fcc 41da5f2a 00015790 00000000
Call Trace with CONFIG_FRAME_POINTER disabled:

 [4005b0f2]  [400553b6]  [40207431]  [4005de52]  [40067d64]
 [40093892]  [4004b15e]  [400390d8]  [40020e70]  [400677d8]
 [40020e70]  [401f0c92]  [40068468]  [4006aa4e]  [40020ea0]
 [4002386c]
Mem-Info:
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Active_anon:0 active_file:0 inactive_anon:0
 inactive_file:4484 dirty:0 writeback:0 unstable:0
 free:8806 slab:565 mapped:0 pagetables:0 bounce:0
DMA free:35216kB min:1016kB low:1268kB high:1524kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:17936kB present:65024kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 4*32kB 6*64kB 3*128kB 46*256kB 44*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 35216kB
4484 total pagecache pages
0 pages RAM
0 pages reserved
0 pages shared
0 pages non-shared
Allocation of length 663552 from process 476 (sh) failed
DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Active_anon:0 active_file:0 inactive_anon:0
 inactive_file:4484 dirty:0 writeback:0 unstable:0
 free:8804 slab:567 mapped:0 pagetables:0 bounce:0
DMA free:35216kB min:1016kB low:1268kB high:1524kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:17936kB present:65024kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 4*32kB 6*64kB 3*128kB 46*256kB 44*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 35216kB
4484 total pagecache pages
Unable to allocate RAM for process text/data, errno 12
sh: page allocation failure. order:8, mode:0xd0
Stack from 41ea6dcc:
        4005b0f2 400553b6 40207431 40645848 00000008 000000d0 00000008 00000000
        000000a2 000a2000 000a2000 0000000c 40544a6c 00000000 405434fc 00000077
        41ea6eac 00000000 00000010 00000000 41ea6008 41ea6000 00000000 00000100
        00000000 41ea6000 00000000 000200d0 4024eecc 00000080 00000000 00000000
        4005de52 000000d0 00000008 4024eec8 00000000 00000001 00004d09 00079100
        00000004 00003f20 00013424 410ae600 41ea6fcc 41ea6f2a 00015790 00000000
Call Trace with CONFIG_FRAME_POINTER disabled:

 [4005b0f2]  [400553b6]  [40207431]  [4005de52]  [40067d64]
 [40093892]  [4004b15e]  [400390d8]  [40020e70]  [400677d8]
 [40020e70]  [401f0c92]  [40068468]  [4006aa4e]  [40020ea0]
 [400239c2]  [4002386c]
Mem-Info:

Answer 1

Your system has run out of 1 MB free pages. With the power of two allocator, you need a free page of size 1 MB to allocate 663552 byes. This is caused by memory fragmentation. Normally, an MMU would reorganize the free space so that it appears contiguous for new allocations.

You can only take care of the problem through prevention. If the 663552 bytes are the sh binary, you will have to prevent it from being continously re-loaded into memory. This might be done by putting it into an XIP file system.

It might be a heap allocation done by the shell. In this case, you will have to change whatever processing is causing such a large malloc.

At the system level, you will also have to see which programs are large or cause large mallocs and change their behavior so that they don't cause more fragmentation.