Oracle - Optimize query, large database table, CLOB field

Question

So I've been wracking my brain on this and admittedly am not very good with Oracle. We have a table that holds about 60 million records with values stored in it for buildings. Have added appropriate indexes where I thought was fit, but still poor performance. Here is the query as that should help:

  SELECT count(*)
    FROM viewBuildings
   INNER JOIN tblValues
           ON viewBuildings.bldg_id = tblValues.bldg_id
   WHERE bldg_deleted = 0
     AND (bldg_summary = 1
         OR (bldg_root = 0 AND bldg_def = 0)
         OR bldg_parent = 1)
     AND field_id IN (207)
     AND UPPER(dbms_lob.substr(v_value, 2000, 1)) = UPPER('2320')

So the above is just one example of a query that can be constructed. It's looking in tblValues on the v_value CLOB field for a match of '2320'. It uppercases as it can search both numeric and text values. tblValues has 60 million records. It is indexed by the building id and also the field id.

I may need to supply more info, but as far as stats go, the number that jumped out to me was "consistent gets". Consistent gets = 74069. Is that a large number?

Any advice would be great, primarily in dealing with a CLOB field on a large database table. Can't use context type indexing as I need exact matches and the data being looked up can be numeric or string.

EDIT (more information): tblBuildings is part of viewBuildings (a view), has 80,000 records tblValues has the values of each building, has 68,000,000 records tblValues has about 550 fields per building (field_id)

Desired result: Query to return results in < 5 seconds. Is this unreasonable? Sometimes it will indefinitely run, other times maybe 80 seconds.

Explain Plan results

Plan hash value: 1480138519
-----------------------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name                             | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------------------------------|
|   0 | SELECT STATEMENT                    |                                  |     1 |   192 |    32   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE                     |                                  |     1 |   192 |            |          |
|   2 |   NESTED LOOPS                      |                                  |     1 |   192 |    15   (0)| 00:00:01 |
|   3 |    NESTED LOOPS                     |                                  |     1 |   183 |    12   (0)| 00:00:01 |
|*  4 |     FILTER                          |                                  |       |       |            |          |
|   5 |      NESTED LOOPS OUTER             |                                  |     1 |    64 |    10   (0)| 00:00:01 |
|*  6 |       TABLE ACCESS BY INDEX ROWID   | TBLBUILDINGS                     |     1 |    60 |     9   (0)| 00:00:01 |
|*  7 |        INDEX RANGE SCAN             | SAA_4                            |    17 |       |     3   (0)| 00:00:01 |
|   8 |         NESTED LOOPS                |                                  |     1 |    21 |     3   (0)| 00:00:01 |
|   9 |          TABLE ACCESS BY INDEX ROWID| TBLBUILDINGSTATUSES              |     1 |    15 |     2   (0)| 00:00:01 |
|* 10 |           INDEX RANGE SCAN          | IDX_BUILDINGSTATUS_EXCLUDEQUERY  |     1 |       |     1   (0)| 00:00:01 |
|* 11 |          INDEX RANGE SCAN           | IDX_BUILDING_STATUS_ASID_DELETED |     1 |     6 |     1   (0)| 00:00:01 |
|  12 |       TABLE ACCESS BY INDEX ROWID   | TBLBUILDINGSTATUSES              |     1 |     4 |     1   (0)| 00:00:01 |
|* 13 |        INDEX UNIQUE SCAN            | PK_TBLBUILDINGSTATUS             |     1 |       |     0   (0)| 00:00:01 |
|* 14 |     TABLE ACCESS BY INDEX ROWID     | TBLVALUES                        |     1 |   119 |     2   (0)| 00:00:01 |
|* 15 |      INDEX UNIQUE SCAN              | PK_SAA_6                         |     1 |       |     1   (0)| 00:00:01 |
|  16 |    INLIST ITERATOR                  |                                  |       |       |            |          |
|* 17 |     INDEX RANGE SCAN                | SAA_7                            |     1 |     9 |     3   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):

   4 - filter("TBLBUILDINGSTATUSES"."BUILDING_STATUS_HIDE_REPORTS" IS NULL OR
              "TBLBUILDINGSTATUSES"."BUILDING_STATUS_HIDE_REPORTS"=0)
   6 - filter("TBLBUILDINGS"."BLDG_SUMMARY"=1 OR "TBLBUILDINGS"."BLDG_SUB_BUILDING_PARENT"=1 OR
              "TBLBUILDINGS"."BLDG_BUILDING_DEF"=0 AND "TBLBUILDINGS"."BLDG_ROOT"=0)
   7 - access("TBLBUILDINGS"."BLDG_DELETED"=0)
       filter( NOT EXISTS (SELECT 0 FROM "TBLBUILDINGSTATUSES" "TBLBUILDINGSTATUSES","TBLBUILDINGS" "TBLBUILDINGS" WHERE
              "TBLBUILDINGS"."BLDG_ID"=:B1 AND "TBLBUILDINGSTATUSES"."BUILDING_STATUS_ID"="TBLBUILDINGS"."BUILDING_STATUS_ID" AND
              "TBLBUILDINGSTATUSES"."BUILDING_STATUS_EXCLUDE_QUERY"=1))
  10 - access("TBLBUILDINGSTATUSES"."BUILDING_STATUS_EXCLUDE_QUERY"=1)
  11 - access("TBLBUILDINGS"."BLDG_ID"=:B1 AND "TBLBUILDINGSTATUSES"."BUILDING_STATUS_ID"="TBLBUILDINGS"."BUILDING_STATUS_ID")
       filter("TBLBUILDINGSTATUSES"."BUILDING_STATUS_ID"="TBLBUILDINGS"."BUILDING_STATUS_ID")
  13 - access("TBLBUILDINGSTATUSES"."BUILDING_STATUS_ID"(+)="TBLBUILDINGS"."BUILDING_STATUS_ID")
  14 - filter(UPPER("DBMS_LOB"."SUBSTR"("TBLVALUES"."V_VALUE",2000,1))=U'2320')
  15 - access("TBLVALUES"."FE_ID"=207 AND "TBLBUILDINGS"."BLDG_ID"="TBLVALUES"."BLDG_ID")
  17 - access("TBLINSPECTORBUILDINGMAP"."IN_ID"=1 AND ("TBLINSPECTORBUILDINGMAP"."IAM_BUILDING_ACCESS_LEVEL"=0 OR
              "TBLINSPECTORBUILDINGMAP"."IAM_BUILDING_ACCESS_LEVEL"=1) AND "TBLBUILDINGS"."BLDG_ID"="TBLINSPECTORBUILDINGMAP"."BLDG_ID")

 44 rows selected

Plan hash value: 2137789089

---------------------------------------------------------------------------------------------
| Id  | Operation                         | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                  |         |  8168 | 16336 |    29   (0)| 00:00:01 |
|   1 |  COLLECTION ITERATOR PICKLER FETCH| DISPLAY |  8168 | 16336 |    29   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------

Okay, I gathered statistics as you suggested and then here is the plan_table_output. Looks like IDX_CURVAL_FE_ID is the problem here? That is an index on the values table for the field id.

SQL_ID  d4aq8nsr1p6uw, child number 0
-------------------------------------
SELECT  /*+ gather_plan_statistics */ count(*)     FROM 
viewAssetsForUser1    INNER JOIN tblCurrentValues            ON 
viewAssetsForUser1.as_id = tblCurrentValues.as_id    WHERE as_deleted = 
:"SYS_B_0"      AND (as_summary = :"SYS_B_1"          OR (as_root = 
:"SYS_B_2" AND as_asset_def = :"SYS_B_3")          OR 
as_sub_asset_parent = :"SYS_B_4")      AND fe_id IN (:"SYS_B_5")      
AND UPPER(dbms_lob.substr(cv_value, :"SYS_B_6", :"SYS_B_7")) = 
UPPER(:"SYS_B_8")

Plan hash value: 4033422776

-----------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                             | Name                      | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  |  OMem |  1Mem | Used-Mem |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                           |      1 |        |      1 |00:08:43.19 |   56589 |  56084 |       |       |          |
|   1 |  SORT AGGREGATE                       |                           |      1 |      1 |      1 |00:08:43.19 |   56589 |  56084 |       |       |          |
|*  2 |   FILTER                              |                           |      1 |        |      0 |00:08:43.19 |   56589 |  56084 |       |       |          |
|   3 |    NESTED LOOPS                       |                           |      1 |        |      0 |00:08:43.19 |   56589 |  56084 |       |       |          |
|   4 |     NESTED LOOPS                      |                           |      1 |    115 |      0 |00:08:43.19 |   56589 |  56084 |       |       |          |
|*  5 |      FILTER                           |                           |      1 |        |      0 |00:08:43.19 |   56589 |  56084 |       |       |          |
|*  6 |       HASH JOIN RIGHT OUTER           |                           |      1 |     82 |      0 |00:08:43.19 |   56589 |  56084 |  1348K|  1348K|  742K (0)|
|   7 |        TABLE ACCESS FULL              | TBLASSETSTATUSES          |      1 |      4 |      4 |00:00:00.01 |       3 |      0 |       |       |          |
|   8 |        NESTED LOOPS                   |                           |      1 |        |      0 |00:08:43.19 |   56586 |  56084 |       |       |          |
|   9 |         NESTED LOOPS                  |                           |      1 |    163 |      0 |00:08:43.19 |   56586 |  56084 |       |       |          |
|* 10 |          TABLE ACCESS BY INDEX ROWID  | TBLCURRENTVALUES          |      1 |    163 |      0 |00:08:43.19 |   56586 |  56084 |       |       |          |
|* 11 |           INDEX RANGE SCAN            | IDX_CURVAL_FE_ID          |      1 |  16283 |  61357 |00:00:05.98 |     132 |    132 |       |       |          |
|* 12 |          INDEX RANGE SCAN             | SAA_1                     |      0 |      1 |      0 |00:00:00.01 |       0 |      0 |       |       |          |
|* 13 |         TABLE ACCESS BY INDEX ROWID   | TBLASSETS                 |      0 |      1 |      0 |00:00:00.01 |       0 |      0 |       |       |          |
|* 14 |      INDEX UNIQUE SCAN                | PK_TBLINSPECTORBRIDGEMAP2 |      0 |      1 |      0 |00:00:00.01 |       0 |      0 |       |       |          |
|* 15 |     TABLE ACCESS BY GLOBAL INDEX ROWID| TBLINSPECTORASSETMAP      |      0 |      1 |      0 |00:00:00.01 |       0 |      0 |       |       |          |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter(:SYS_B_0=0)
   5 - filter(("TBLASSETSTATUSES"."ASSET_STATUS_HIDE_REPORTS" IS NULL OR "TBLASSETSTATUSES"."ASSET_STATUS_HIDE_REPORTS"=0))
   6 - access("TBLASSETSTATUSES"."ASSET_STATUS_ID"="TBLASSETS"."ASSET_STATUS_ID")
  10 - filter(UPPER("DBMS_LOB"."SUBSTR"("TBLCURRENTVALUES"."CV_VALUE",:SYS_B_6,:SYS_B_7))=SYS_OP_C2C(UPPER(:SYS_B_8)))
  11 - access("TBLCURRENTVALUES"."FE_ID"=:SYS_B_5)
  12 - access("TBLASSETS"."AS_DELETED"=:SYS_B_0 AND "TBLASSETS"."AS_ID"="TBLCURRENTVALUES"."AS_ID")
  13 - filter((("TBLASSETS"."AS_ROOT"=:SYS_B_2 AND "TBLASSETS"."AS_ASSET_DEF"=:SYS_B_3) OR "TBLASSETS"."AS_SUMMARY"=:SYS_B_1 OR 
              "TBLASSETS"."AS_SUB_ASSET_PARENT"=:SYS_B_4))
  14 - access("TBLASSETS"."AS_ID"="TBLINSPECTORASSETMAP"."AS_ID" AND "TBLINSPECTORASSETMAP"."IN_ID"=1)
  15 - filter(("TBLINSPECTORASSETMAP"."IAM_ASSET_ACCESS_LEVEL"=0 OR "TBLINSPECTORASSETMAP"."IAM_ASSET_ACCESS_LEVEL"=1))

Answer 1

Bad Index Cost If the stats are fresh, and the optimizer has a relatively good cardinality estimate, why would it pick a bad plan? Perhaps there is a parameter making indexes look artificially cheap. Take a look at: select * from v$parameter where name in ('optimizer_index_cost_adj', 'optimizer_index_caching'); Are they significantly different than their default values, 100 and 0?

Also, take a look at select * from sys.aux_stats$; Maybe your system statistics make full table scans look too expensive. Some versions of Oracle have bugs with workload statistics where the numbers are wrong by several orders of magnitude.

Or perhaps your table is just incredibly huge, and 16K index reads is the best access path. Look at DBA_SEGMENTS.BYTES to find the size of your table and LOB segment.

Even if the table is medium-sized, and the plan changed to a full table scan, that probably won't get the run time to within 5 seconds. But combined with your idea to partition, it might be enough.

LOB STORAGE Given your example, I assume most of the CLOBs are relatively small? Perhaps you have an unusual LOB setting wasting a lot of space, such as DISABLE STORAGE IN ROW. You may want to check your table DDL, or post all of it here. Or if you can replace the CLOB with a VARCHAR2, that would be even better.

FBI A function-based index on the CLOB may significantly speed things up. But it may be a very large index: create index TBLCURRENTVALUES_FBI on TBLCURRENTVALUES(UPPER(dbms_lob.substr(v_value, 2000, 1)));

CURSOR_SHARING The queries are changing a bit, which makes tuning difficult. Looks like this latest version has CURSOR_SHARING=FORCE, which is unusual. For an expensive query, using literals can be a good thing - the extra time spent building query plans is probably worth it. If the system parameter can't change, look into the hint /*+ cursor_sharing_exact */.

Answer 2

You can do any number of optimization but ultimately its the huge amount of data which causes the problem. When you execute the query and track it on performance graph on OEM , you will that major amount of time will be spent on IO. That is taking data in and out of memory.

So whats the solution: It will be to partition the table. Whenever data is huge , you should look to partition the table so that you deal with only relevant data. In order to partition the table you need some point to segregate the data and looking at your data it can be building id.

You can read more about it at this url : http://docs.oracle.com/cd/E11882_01/server.112/e25523/partition.htm#g471747

Partitioning comes up with many other features, like local indexes which help to optimize queries even more.

Partitioning will not be a solution if you are deal with whole of large table data all the time but then that puts a question mark over database schema.

SO yes query optimization will help but as data is large you should evaluate table partitioning as well.