Finding Covariance using SQL

Question

# dt---------indx_nm1-----indx_val1-------indx_nm2------indx_val2
2009-06-08----ABQI------1001.2------------ACNACTR----------300.05
2009-06-09----ABQI------1002.12 ----------ACNACTR----------341.19
2009-06-10----ABQI------1011.4------------ACNACTR----------382.93
2009-06-11----ABQI------1015.43 ----------ACNACTR----------362.63

I have a table that looks like ^ (but with hundreds of rows that dates from 2009 to 2013). Is there a way that I could calculate the covariance : [(indx_val1 - avg(indx_val1)) * (indx_val2 - avg(indx_val2)] divided by total number of rows for each value of indx_val1 and indx_val2 (loop through the entire table) and return just a simple value for cov(ABQI, ACNACTR)

Answer 1

Since you have aggregates operating over two different groups, you will need two different queries. The main one groups by dt to get your row values per date. The other query has to perform AVG() and COUNT() aggregates across the whole rowset.

To use them both at the same time, you need to JOIN them together. But since there's no actual relation between the two queries, it is a cartesian product and we'll use a CROSS JOIN. Effectively, that joins every row of the main query with the single row retrieved by the aggregate query. You can then perform the arithmetic in the SELECT list, using values from both:

So, building on the query from your earlier question:

SELECT 
 indxs.*,
 ((indx_val2 - indx_val2_avg) * (indx_val1 - indx_val1_avg)) / total_rows AS cv
FROM (
    SELECT 
      dt,
      MAX(CASE WHEN indx_nm = 'ABQI' THEN indx_nm ELSE NULL END) AS indx_nm1,
      MAX(CASE WHEN indx_nm = 'ABQI' THEN indx_val ELSE NULL END) AS indx_val1,
      MAX(CASE WHEN indx_nm = 'ACNACTR' THEN indx_nm ELSE NULL END) AS indx_nm2,
      MAX(CASE WHEN indx_nm = 'ACNACTR' THEN indx_val ELSE NULL END) AS indx_val2
    FROM table1 a
    GROUP BY dt
  ) indxs 
  CROSS JOIN (
    /* Join against a query returning the AVG() and COUNT() across all rows */
    SELECT
      'ABQI' AS indx_nm1_aname,
      AVG(CASE WHEN indx_nm = 'ABQI' THEN indx_val ELSE NULL END) AS indx_val1_avg,
      'ACNACTR' AS indx_nm2_aname,
      AVG(CASE WHEN indx_nm = 'ACNACTR' THEN indx_val ELSE NULL END) AS indx_val2_avg,
      COUNT(*) AS total_rows
    FROM table1 b
    WHERE indx_nm IN ('ABQI','ACNACTR')
    /* And it is a cartesian product */
  ) aggs
WHERE
  indx_nm1 IS NOT NULL
  AND indx_nm2 IS NOT NULL
ORDER BY dt

Here's a demo, building on your earlier one: http://sqlfiddle.com/#!6/2ec65/14

Answer 2

Here is a Scalar-valued function to perform a covariance calculation on any two column table formatted to XML.

To Test: Compile the function then execute the Alpha Test

    CREATE Function [dbo].[Covariance](@XmlTwoValueSeries xml)
    returns float
    as
    Begin
    /*

    -- -----------
    -- ALPHA TEST
    -- -----------
    IF object_id('tempdb..#_201610101706') is not null DROP TABLE #_201610101706
    select *
    into #_201610101706
    from
    (
        select *
        from
        (
            SELECT '2016-01' Period, 1.24 col0, 2.20 col1
            union
            SELECT '2016-02' Period, 1.6 col0, 3.20 col1
            union
            SELECT '2016-03' Period, 1.0 col0, 2.77 col1
            union
            SELECT '2016-04' Period, 1.9 col0, 2.98 col1
        ) A
    ) A


    DECLARE @XmlTwoValueSeries xml  
    SET @XmlTwoValueSeries = (
    SELECT col0,col1 FROM #_201610101706
    FOR
    XML PATH('Output')
    )

    SELECT dbo.Covariance(@XmlTwoValueSeries) Covariance

    */
    declare @returnvalue numeric(20,10)

    set @returnvalue = 
    (
        SELECT  SUM((x - xAvg) *(y - yAvg)) / MAX(n) AS [COVAR(x,y)]
        from 
        (
            SELECT  1E * x x,
                    AVG(1E * x) OVER (PARTITION BY (SELECT NULL)) xAvg,
                    1E * y y,
                    AVG(1E * y) OVER (PARTITION BY (SELECT NULL)) yAvg,
                    COUNT(*) OVER (PARTITION BY (SELECT NULL)) n
            FROM    
            (
                SELECT 
                    e.c.value('(col0/text())[1]', 'float' ) x,
                    e.c.value('(col1/text())[1]', 'FLOAT' ) y
                FROM @XmlTwoValueSeries.nodes('Output') e(c)            
            ) A
        ) A
    )
    return @returnvalue
    end



    GO