The answer from @MichelFeldheim was the genesis, but we improved upon it to handle multiple applications writing to multiple log files.
Central Logging Server
In the central logging server, install syslog-ng and configure it thus:
sudo apt-get install syslog-ng
add the following to /etc/syslog-ng/syslog-ng.conf:
destination d_php { file("$PROGRAM" owner(www-data) group(www-data) perm(0644)); };
filter f_php { program("^\/var\/log\/"); };
log { source(s_all); filter(f_php); destination(d_php); flags(final); };
source s_all {
# ....
# .... LET THE PREVIOUS CONTENT STAY - add the following line
tcp(port(5140) keep_alive(yes));
};
restart syslog service:
sudo service syslog-ng restart
On FE Servers
On each of the FE Servers, install syslog-ng and configure it thus:
sudo apt-get install syslog-ng
add the following to /etc/syslog-ng/syslog-ng.conf on each of the FE servers:
destination php { tcp("log.example.com" port(5140)); };
log { source(s_all); filter(f_php); destination(php); };
filter f_php { facility(user); };
restart syslog servers:
sudo service syslog-ng restart
Application Code Changes:
Now, the application code can be changed thus. Suppose each of the application have code like this writing to a separate file and you want the same structure to be reflected in the central log server:
// PREVIOUS CODE: using PEAR Log
include '/usr/share/php/Log.php';
$log = Log::singleton('file', '/var/log/nginx/xxx.log', '', array(), PEAR_LOG_INFO);
// PREVIOUS CODE: Using error_log
ini_set('error_log' , '/var/log/nginx/xxx.log');
The new code should look like:
// NEW CODE: using PEAR Log
include '/usr/share/php/Log.php';
$log = Log::singleton('syslog', LOG_USER, '/var/log/nginx/xxx.log', array(), PEAR_LOG_INFO);
// NEW CODE: Using error_log
ini_set(‘error_log’, ‘syslog’);
openlog('/var/log/nginx/xxx.log', LOG_NDELAY, LOG_USER);
If your FE servers and the Logging servers are all within the same EC2 security group, then there is no need to open the ports, since within the groups, all ports can be accessed freely, so long as a service is listening to it.
This approach allows your each of your apps, modules to decide whether they want central logging or not.