High memory utilization due to autoflushing log

Question

Is it possible to have high memory utilization if autoflush_log is set to true?

I have a custom production environment(staging) setup like this

require Rails.root.join("config/environments/production")

Calamus::Application.configure do
  config.action_mailer.default_url_options = {:protocol => 'https', :host => xx.xx.xx.xx }
end

This was logging the sql statements. so I set the log level to info

config.log_level = :info

But adding the above line resulted in unicorn processes taking too much memory and the machine was running very high on memory

When I added this line

config.autoflush_log = false

The memory utilization became normal. Can anyone see the connection? Why would autoflushing cause high memory utilization?

Answer 1

Generally speaking, writing to files is an expensive operation as any IO operation. This leads to other questions such as, how much am I logging to cause system slowdown? What file system am I logging to, is it shared, etc?

Have you looked at the logs? Are there exceptions being raised? How big does your log grow? And at what rate?

First, I would look at what's being logged, and the rate at which this is happening. Second, if logging is normal but mem utilization is high, I would bring logging down to further investigate.

Fist. Only log anything above or equal to a warning

config.log_level = :warn

without setting autoflush_log.

When the system is high, I would look at what "top" tells me as well as /var/log/syslog

An alternative utility to "top" is "htop". It gives you more info I find (easier on the eyes) as well as being highly configurable. http://hisham.hm/htop/