Resolve a DRG-11119 error in emagent.trc on Oracle 11.2

Question

I was able to fix the issue, and decided to answer my own question for others that might run into this problem and hit the same wall I did. I still don't know what caused it, but it is fixed.

Further research pointed me to the following link, which contained enough hints to point me in the right direction. http://docs.oracle.com/cd/B14099_19/manage.1012/b16242/emctl.htm

The section 2.7.6 "Reevaluating Metric Collections" had information on the files that the Enterprise Manager Metrics are stored in. To avoid dead links, I will copy some excerpts of that article here:

1. Go to $ORACLE_HOME/sysman/admin/metadata directory, where $ORACLE_HOME is the Oracle Home of the Management Agent.

2. Locate the XML file for the target type. For example, if you are interested in the host metric 'Filesystem Space Available(%)' metric, look for the host.xml file.

I actually grep'd for textIndexStats in this directory and found it in a file called database.xmlp. I found a lot of information inside the following line:

  <Metric NAME="textIndexStats" TYPE="RAW" IS_METRIC_LONG_RUNNING="TRUE" >

The most useful piece of information came from SQL embedded as CDATA, which included the lines:

cursor idx_cur IS
select owner,job_name,comments
from dba_scheduler_jobs where job_name like 'EM_IDX_STAT_JOB%' and
upper(owner) = 'DBSNMP';

idx_rec idx_cur%ROWTYPE;
BEGIN
 OPEN idx_cur;
 FETCH idx_cur into idx_rec;
 guid := :1;
 IF idx_cur%FOUND THEN
   dbms_lob.createtemporary(statData,false);
   dbms_lob.createtemporary(sizeData,false);
   dbms_lob.createtemporary(objectsData,false);
   idx_name := substr(idx_rec.comments,1,instr(idx_rec.comments,'|')-1);

This makes it obvious that the non-existent index name was being parsed out of a comments column in dba_scheduler_jobs for the DBSNMP user, with a job name like 'EM_IDX_STAT_JOB%'.

Running the same query used in the cursor above showed me a number of records in the scheduler table. Apparently they aren't true scheduler entries but are used to queue this script, which inserts data into sysman.mgmt_text_index_stats. A number of CTXCAT and missing indexes were in the scheduler table. Apparently the rows in the scheduler table are only removed on success, and an incorrect entry will hang around for years.

To fix this issue, I ran the following as user DBSNMP:

BEGIN
   for idx_rec in (
    select owner,job_name,comments
    from dba_scheduler_jobs
    where job_name like 'EM_IDX_STAT_JOB%' and upper(owner) = 'DBSNMP')
  LOOP
  DBMS_SCHEDULER.DROP_JOB( idx_rec.job_name );
  END LOOP;
END;
/

This has eliminated the issue of the SPAM'd trace log file. It would be good if CTXCAT indexes could not be added, or that they were handled gracefully when in there. I hope this helps the next DBA down the road, because I spent way too much time on it.