SQL Server 2008 Stored Procedure Performance issue

Question

Hi I have a Stored Procedure

     ALTER PROCEDURE [dbo].[usp_EP_GetTherapeuticalALternates]
    (
      @NDCNumber CHAR(11) ,
      @patientid INT ,
      @pbmid INT
    )
AS 
    BEGIN
        TRUNCATE TABLE TempTherapeuticAlt
        INSERT  INTO TempTherapeuticAlt
                SELECT  --PR.ProductID AS MedicationID ,
                        NULL AS MedicationID ,
                        PR.ePrescribingName AS MedicationName ,
                        U.Strength AS MedicationStrength ,
                        FRM.FormName AS MedicationForm ,
                        PR.DEAClassificationID AS DEASchedule ,
                        NULL AS NDCNumber
        --INTO    #myTemp
                FROM    DatabaseTwo.dbo.Product PR
                        JOIN ( SELECT   MP.MarketedProductID
                               FROM     DatabaseTwo.dbo.Therapeutic_Concept_Tree_Specific_Product TCTSP
                                        JOIN DatabaseTwo.dbo.Marketed_Product MP ON MP.SpecificProductID = TCTSP.SpecificProductID
                                        JOIN ( SELECT   TCTSP.TherapeuticConceptTreeID
                                               FROM     DatabaseTwo.dbo.Marketed_Product MP
                                                        JOIN DatabaseTwo.dbo.Therapeutic_Concept_Tree_Specific_Product TCTSP ON MP.SpecificProductID = TCTSP.SpecificProductID
                                                        JOIN ( SELECT
                                                              PR.MarketedProductID
                                                              FROM
                                                              DatabaseTwo.dbo.Package PA
                                                              JOIN DatabaseTwo.dbo.Product PR ON PA.ProductID = PR.ProductID
                                                              WHERE
                                                              PA.NDC11 = @NDCNumber
                                                             ) PAPA ON MP.MarketedProductID = PAPA.MarketedProductID
                                             ) xxx ON TCTSP.TherapeuticConceptTreeID = xxx.TherapeuticConceptTreeID
                             ) MPI ON PR.MarketedProductID = MPI.MarketedProductID
                        JOIN ( SELECT   P.ProductID ,
                                        O.Strength ,
                                        O.Unit
                               FROM     DatabaseTwo.dbo.Product AS P
                                        INNER JOIN DatabaseTwo.dbo.Marketed_Product
                                        AS M ON P.MarketedProductID = M.MarketedProductID
                                        INNER JOIN DatabaseTwo.dbo.Specific_Product
                                        AS S ON M.SpecificProductID = S.SpecificProductID
                                        LEFT OUTER JOIN DatabaseTwo.dbo.OrderableName_Combined
                                        AS O ON S.SpecificProductID = O.SpecificProductID
                               GROUP BY P.ProductID ,
                                        O.Strength ,
                                        O.Unit
                             ) U ON PR.ProductID = U.ProductID
                        JOIN ( SELECT   PA.ProductID ,
                                        S.ScriptFormID ,
                                        F.Code AS NCPDPScriptFormCode ,
                                        S.FormName
                               FROM     DatabaseTwo.dbo.Package AS PA
                                        INNER JOIN DatabaseTwo.dbo.Script_Form
                                        AS S ON PA.NCPDPScriptFormCode = S.NCPDPScriptFormCode
                                        INNER JOIN DatabaseTwo.dbo.FormCode AS F ON S.FormName = F.FormName
                               GROUP BY PA.ProductID ,
                                        S.ScriptFormID ,
                                        F.Code ,
                                        S.FormName
                             ) FRM ON PR.ProductID = FRM.ProductID
                            WHERE   
                            ( PR.OffMarketDate IS NULL )
        OR ( PR.OffMarketDate = '' )
        OR (PR.OffMarketDate = '1899-12-30 00:00:00.000')
        OR ( PR.OffMarketDate <> '1899-12-30 00:00:00.000'
             AND DATEDIFF(dd, GETDATE(),PR.OffMarketDate) > 0
           )
                GROUP BY PR.ePrescribingName ,
                        U.Strength ,
                        FRM.FormName ,
                        PR.DEAClassificationID
               -- ORDER BY pr.ePrescribingName

        SELECT  LL.ProductID AS MedicationID ,
                temp.MedicationName ,
                temp.MedicationStrength ,
                temp.MedicationForm ,
                temp.DEASchedule ,
                temp.NDCNumber ,
                fs.[ReturnFormulary] AS FormularyStatus ,
                copay.CopaTier ,
                copay.FirstCopayTerm ,
                copay.FlatCopayAmount ,
                copay.PercentageCopay ,
                copay.PharmacyType,
                dbo.udf_EP_GetBrandGeneric(LL.ProductID) AS BrandGeneric
        FROM    TempTherapeuticAlt temp
                OUTER APPLY ( SELECT TOP 1
                                        ProductID
                              FROM      DatabaseTwo.dbo.Product
                              WHERE     ePrescribingName = temp.MedicationName
                            ) AS LL
                OUTER APPLY [dbo].[udf_EP_tbfGetFormularyStatus](@patientid,
                                                              LL.ProductID,
                                                              @pbmid) AS fs
                OUTER APPLY ( SELECT TOP 1
                                        *
                              FROM      udf_EP_CopayDetails(LL.ProductID,
                                                            @PBMID,
                                                            fs.ReturnFormulary)
                            ) copay
        --ORDER BY LL.ProductID
        TRUNCATE TABLE TempTherapeuticAlt
    END

On my dev server I have data of 63k in each table

so this procedure took about 30 seconds to return result.

On my Production server, it is timing out, or taking >1 minute. I am wondering my production server tables are full with 1400 millions of records,

can this be a reason.

if so what can be done, I have all required indexes on tables.

any help would be greatly appreciated.

thanks

Execution Plan http://www.sendspace.com/file/hk8fao

Major Leakage

OUTER APPLY [dbo].[udf_EP_tbfGetFormularyStatus](@patientid,
                                                              LL.ProductID,
                                                              @pbmid) AS fs

Answer 1

Some strategies that may help:

1. Remove the first ORDER BY statement, those are killer on complex queries shouldn't be necessary.

2. Use CTEs to break the query into smaller pieces that can be individually addressed.

3. Reduce the nesting in the first set of JOINs

4. Extract the second and third set of joins (the GROUPED ones) and insert those into a temporary indexed table before joining and grouping everything.

5. You did not include the definition for function1 or function2 -- custom functions are often a place where performance issues can hide.

Without seeing the execution plan, it's difficult to see where the particular problems may be.

Answer 2

You have a query that selects data from 4 or 5 tables , some of them multiple times. It's really hard to say how to improve without deep analysis of what you are trying to achieve and what table structure actually is.

Data size is definitely an issue; I think it's quite obvious that the more data has to be processed, the longer query will take. Some general advices... Run the query directly and check execution plan. It may reveal bottlenecks. Then check if statistics is up to date. Also, review your tables, partitioning may help a lot in some cases. In addition, you can try altering tables and create clustered index not on PK (as it's done by default unless otherwise specified), but on other column[s] so your query will benefit from certain physical order of records. Note : do it only if you are absolutely sure what you are doing.

Finally, try refactoring your query. I have a feeling that there is a better way to get desired results (sorry, without understanding of table structure and expected results I cannot tell exact solution, but multiple joins of the same tables and bunch of derived tables don't look good to me)