regex matching unix log monitoring in SCOM 2012 SP1 tests ok but doesn't match on Unix server

Question

What I am trying to do is use the regular expression to perform the suppression of specific events (such as event codes) in SCOM unix log monitoring.

The expression is:

((?i:warning)(?!(.*1222)|(.*1001)))

ie Search the log for "warning" but don't match if events 1222 or 1001 exist on the line.

I use the built in test function in SCOM when creating the rule and the tests with sample logs come back as expected but when I inject test lines into the unix log, no alerts get generated.

I suspect it could be the syntax not being accepted on the system (its running RedHat 6 ) I have tested this with regex tools and looks to work as expected.

When I try and test it on the server i get:

[root@bld02 ~]# grep ((?i:warning)(?!(.*1222)|(.*1001))) /var/log/messages
-bash: !: event not found

[root@bld02 ~]# tail /var/log/messages
Nov 13 15:07:26 bld02 root: SCOM Test Warning Event ID 1001 Round 18
Nov 13 15:07:29 bld02 root: SCOM Test Warning Event ID 1000 Round 18
Nov 13 15:07:35 bld02 root: SCOM Test Warning Event ID 1002 Round 18

So I am expecting 2 matches.

What should the regex be? I have tested many variations like: .*((?i:warning)(?!(.*1222)|(.*1001))).* , used the -Rn option, used quotes ect but nothing matches when I manually run it... what am I missing?

Answer 1

The "Test Log File Expression" utility for SCOM runs on the Windows computer, and uses a different RegEx parser (.NET) than is used in the actual log file parsing. The log files are parsed on the UNIX/Linux computer using standard POSIX RegEx. Aparantly POSIX RegEx doesn't support negative lookaheads (?!) which is why when I run the command manually it returns no results.

Answer 2

grep doesn't support look aheads unless you enable perl comparability via the -P flag:

Try this:

grep -P '((?i:warning)(?!(.*1222)|(.*1001)))' /var/log/messages

I also added quotes around the regex - I'm pretty sure you need those too.

Answer 3

You can split the positive and negative matching (-v):

cat /var/log/messages | grep -i warning | grep -v '1222|1001' 

Answer 4

You are all over complicating the solution

1. create your log file reader as normal ie point at /bogus/file.log, and capture your key words ie warning|error|fail etc
2. export the management pack
3. edit it to enable filtering

ie find the WriteActions tag for the rule and insert the filter there ie

</DataSources>
<ConditionDetection ID="Filter" TypeID="System!System.ExpressionFilter">
<Expression>
  <RegExExpression>
     <ValueExpression>
        <XPathQuery Type="String">//row</XPathQuery>
     </ValueExpression>
     <Operator>DoesNotMatchRegularExpression</Operator>
     <Pattern>exampleexpression</Pattern>
 </RegExExpression>
</Expression>
</ConditionDetection>
<WriteActions>

import as is if you want and edit in the console after as its much easier, don't remove the last expression as it may remove the filter. Remember to increment the version number of the management pack

4. import the management pack
5. add a filter using an AND group ie //row contains warning and //row does not match regular expression (1222|1001)

you will now have the items filtered out as required, but still have everything else.

Be warned do not have multiple log file readers reading the same log file. it doesn't work well.