Ignore certain characters in DB2 SELECT statement

Question

When building a text search into a website I'm passing it through a PHP loop:

$queryList=explode(' ', $queryString);
foreach ($queryList as $queryParm) {
    foreach ($fields as $key => $query) {
        $whereStr.=($key == 0) ? " AND (ucase($query) LIKE ?" : " OR ucase($query) LIKE ?";
        $db->parms[]="%$queryParm%";
        if ($key+1 == count($fields)) $whereStr.=")";
    }
}

This will separate the query's words and make each one a separate parameter. I'm building a regex to only allow alphanumeric plus spaces, but now the problem is I need the SQL to match this regex, for example if there's an item named "FakeCo's Unicorn Repellant #123" then searching "fakecos unicorn repellant 123" should yield the result, ignoring the fact that the "'" and the "#" exist. In fact due to the structure of the list "123 fakeco" should show the result too.

Is there a way to apply a regex to a DB2 statement?

This is being ran against DB2 for i (version 7.1)

Edit: attempted solution (insert this before PHP mentioned above), some characters caused issues & had to be removed, poor performance when it did run

$sqlNeedsEscape=array("?", "'");
$ignoreChars="'\"?~!@#$%^&*()-=+[]{}|<>,./\\";
$ignoreChars=str_split($ignoreChars);
$whereBefore='';
$whereAfter='';
foreach ($ignoreChars as &$ic) {
    if (in_array($ic, $sqlNeedsEscape)) $ic="'$ic";
    error_log($ic);
    $whereBefore.="replace(";
    $whereAfter.=",'$ic','')";
}

Then change the $whereStr.= line to:

$whereStr.=($key == 0) ? " AND (ucase(trim($whereBefore $query $whereAfter)) LIKE ?" : " OR $whereBefore ucase(trim($query)) $whereAfter LIKE ?";

Effectively it was replace(replace(replace(query, '!', ''), '@', ''), '#', '') for each field, only wrap several more replaces around that. I quit chasing this idea when I saw the impact it had on performance. I will speak with our RPG devs about the UDF solution suggested by Buck Calabro. It sounds like the only "real" solution possible.

Answer 1

Since you're dealing with IBM i version 7.1, the best solution IMO would be to get the DB2 Omnifind Text Search Server (5733-OMF) product loaded...

In short Omnifind provides "high-speed linguistic text searches"; so a search for mouse would find not only anything containing mouse but anything containing mice as well.

http://pic.dhe.ibm.com/infocenter/iseries/v7r1m0/index.jsp?topic=%2Frzash%2Frzashkickoff.htm http://www-304.ibm.com/partnerworld/wps/servlet/ContentHandler/whitepaper/i/omnifind/search

It is a no-charge licensed product. Note that it is also available for 6.1.

In addition to providing the CONTAINS() function there's a SCORE() function that allows you to rank search results.

Prior to 6.1, IBM offered a for charge product known as (DB2 Extender) Text Search Engine (5770-DE1) that had much of the same functionality but that was not as nice to use.

HTH,

Charles

Answer 2

The DB2 for i LIKE predicate does not support regex expressions.

Answer 3

As @JamesA notes, as of 7.1 there is no support for a regex in the WHERE clause. That doesn't mean that someone hasn't written a User Defined Function to implement one. Dennis Lovelady has shared his idea on the Midrange dot com mailing list at http://archive.midrange.com/midrange-l/201209/msg00810.html It is written in RPG, so you'd need to get one of the IBM i programmers to load it up for you. Because this is a UDF, the database optimiser can't use an index, so queries done this way aren't as performant as one might expect.

Answer 4

You could use Mysql's RLIKE function: http://dev.mysql.com/doc/refman/5.0/en/pattern-matching.html