Huge error_log filled with zeros at Joomla site in public_html directory causing server hang

Question

I am not sure if this is related to PHP code/Joomla plugin/component or server issue so to go and post my question at serverfault.

I have a dedicated server running CentOS release 6.4 (Final) and CPanel 11.38.1 (build 15) with [Apache/2.2.24 (Unix) PHP/5.2.17] and recently we had brute force attacks which were causing system hangs with messages like "HANG: chkservd on server". After setting up a DenyHosts and changing ssh default port the server stabilized for some days until it started to suddenly hang with no specific errors in logs nor any email notifications.

The server runs just one website but with some traffic (about 2000 visitors per day) and a large Joomla database. The site was up and running flawlessly since March 2013, until two weeks ago.

After investigating, I have noticed that the error_log under the public_html (default Joomla site directory) size was over 200MB but with no specific error messages but just lines with date/time and zeros "0"!

[24-Jul-2013 12:09:18] 0
[24-Jul-2013 12:09:18] 0
[24-Jul-2013 12:09:18] 0
[24-Jul-2013 12:09:19] 0
[24-Jul-2013 12:09:19] 0
[24-Jul-2013 12:09:19] 0
[24-Jul-2013 12:09:19] 0
[24-Jul-2013 12:09:20] 0
[24-Jul-2013 12:09:26] 0
[24-Jul-2013 12:09:26] 0
[24-Jul-2013 12:09:30] 0

Almost every second, thousand lines like those above! Sometimes the site crashes with "jos-Error: Application Instantiation Error" and some other times with an error:

PHP Fatal error: Call to a member function get() on a non-object in ...

which is due to database issues when Joomla tries to load params from the database and it fails.

I have setup a backup of the site on a local windows server machine with apache and php installed [Apache/2.2.19 (Win32) PHP/5.3.6] and these zero "0" errors also spawn on every site call with some more details in \Apache2\logs\error.log like:

[Wed Jul 24 14:42:32 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/
[Wed Jul 24 14:42:32 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/
[Wed Jul 24 14:42:32 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/
[Wed Jul 24 14:42:33 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/
[Wed Jul 24 14:42:33 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/
[Wed Jul 24 14:42:33 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/
[Wed Jul 24 14:42:56 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/mousiki
[Wed Jul 24 14:42:56 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/mousiki
[Wed Jul 24 14:42:56 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/mousiki
[Wed Jul 24 14:42:57 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/mousiki
[Wed Jul 24 14:42:57 2013] [error] [client 192.168.1.66] 0, referer: http://virtdev.ose.domain.com/mousiki

thus I assume it is not server related, but Joomla site related from a plugin/component.

Also, we have scanned the site with online site scanners and plain file search with antivirus tools for bot injections with no positive results.

How we can locate what is causing these zeros as errors?

Answer 1

Finally I found what is/was causing these zero errors.

First of all, to locate what was spawning these errors I did a grep to all sites php files for the php function error_log() (http://php.net/manual/en/function.error-log.php), which writes directly the string argument message without any other information like filename, line numbers etc. At the results I saw there was an error_log(0); at a Joomla plugin that it was enabled. After opening the plugin php file, I figured out that it was my addition in order to debug something and I've had totally forgotten there. So, after deleting the line there was no more zero character errors. I don't even remember what I wanted to achieve by that call, it doesn't make any sense at all. Maybe I wanted to use error_reporting(0) to disable all errors and mistyped.

And to make it clear, these errors were not related and responsible at all for the server crash/hang. I fixed the crashes by creating a new database and restoring the backup file, so, the old database was damaged.

Now, after 24 hours, there was no crashes/hangs and everything works perfect again.

EDIT:

After some further investigation, I have realized that this error_log(0) was not my addition and it was already there when I installed the plugin! The plugin/component is Joooid, an Android client for Joomla (http://www.joooid.com). The code is located at the plg_joooidcontent inside joooidcontent.php:

...
/**
 * @since   1.6
 */
public function onContentPrepare($context, &$row, &$params, $page=0)
{

    error_log(0);
    //echo "<pre>"; 
    $row ->text = $this->executeTokens($row->text);
    //print_r($row);
    //die();

}
...

Even now it is there with the latest release. I will send them a link to this question so to inform them about it.