Linux(Ubuntu) load average higher than total-true-utilization?

Question

I have a dell pd2950(2x4core) server running Ubuntu server 12.04LTS. And there's a VLC encoder instance running. Recently I updated the script(VLM) for VLC to increase quality and this means I'm increasing the CPU utilization too. So I started to tune the script to avoid exceeding maximum utilization. I use top to monitor the CPU utilization. I found that the load average is higher than 100%(I have 8-cores totally so 8.00 is 100%) but there's still 20-35% is idle, like:

top - 21:41:19 up 2 days, 17:15,  1 user,  load average: 9.20, 9.65, 8.80
Tasks: 148 total,   1 running, 147 sleeping,   0 stopped,   0 zombie
Cpu(s): 32.8%us,  0.7%sy, 29.7%ni, 36.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1982680k total,  1735672k used,   247008k free,   126284k buffers
Swap:        0k total,        0k used,        0k free,   774228k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 9715 wilson    RT   0 2572m 649m  13m S  499 33.5  13914:44 vlc
11663 wilson    20   0 17344 1328  964 R    2  0.1   0:02.00 top
    1 root      20   0 24332 2264 1332 S    0  0.1   0:01.06 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.09 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:27.05 ksoftirqd/0
    4 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/0:0
    5 root       0 -20     0    0    0 S    0  0.0   0:00.00 kworker/0:0H

To confirm my CPU(s) don't have Hyper-Thread, I tried:

wilson@server:/$ nproc
8

And to reduce the sampling deviation cause by refresh time, I also tried:

wilson@server:/$ top -d 0.1

I looked at the number %id for a long time, it haven't been lower than 14.

I also tried:

wilson@server:/$ uptime
21:57:20 up 2 days, 17:31,  1 user,  load average: 9.03, 9.12, 9.35

The 1m load average often reach 14-15. So I'm wondering what's wrong with my system? Has anyone ever have this problem?

More information:

I'm using VLC with x264 codec to encode a live HTTP stream(application/octet-stream). It use ffmpeg(libavc) to decode and output as Apple HLS(.ts segment). I found this problem after I added arguments for x264:

level=41,ref=5,b-adapt=2,direct=auto,me=umh,subq=8,rc-lookahead=60,analyse=all

This almost equal to preset=slower. And as you can see, my VLC in running in real-time. The parameter is:

wilson@server:/$ chrt -p -f 99 vlc-wrapper

Answer 1

There does not appear to be anything wrong with your system. What is wrong seems to be your understanding of CPU accounting. In particular, load average has nearly nothing at all to do with CPU usage. Load average is based on the number of processes that are ready to run (not waiting on I/O, network, keyboard input, etc...), if there is an available CPU for them to be scheduled on. While it's true that, given an 8 core system, if all 8 cores are 100% busy with a single CPU-bound thread each, your load average should be around 8.00, it is entirely possible to have a load average of 200.0 with near-0% CPU utilization. All that would indicate is you have 200 processes that are ready to run, but as soon as they get scheduled, they do almost nothing before they go back to waiting for input of some sort.

Your top output shows that vlc seems to be using roughly the equivalent of 5 of your cores, but it doesn't indicate whether you have 5 cores at 100% each, or if all 8 cores are at 62.5% each. All of the other processes listed by top also contribute to your load average, as well as CPU usage. In particular, top running with a short delay like your example of 0.1 seconds, will probably increase your load average by almost 1 itself, even though, overall, it's not using a lot of CPU time.

Answer 2

Read this:

Understanding load average vs. cpu usage

If the load average is at 7, with 4 hyper-threaded processors, shouldn't that means that the CPU is working to about 7/8 capacity?

No it just means that you have 7 running processes in the job queue on average.

But I think that we can't use load average as a reference number to determine system is overload or not. So that I wonder if there's a kernel-level cpu utitlization statistical tools or not?(why kernel level because reduce performance loss)