Minimize variance “intuitively”

Question

We have a problem the arises in asset management. I think (and hope) it raises interesting enough questions for this forum to consider it. We made pretty extensive searches of the literature and find things that talk around this question, but nothing directly dealing with the issue.

Background

We have time series data for assets, from which we calculate a correlation matrix. For 5 assets using Mathematica it might look something like this:

m = Correlation[data]

{{1.0, 0.635562, 0.698852, 0.404792, -0.32746}, {0.635562, 1.0, 0.410075, 0.314375, -0.0636438}, {0.698852, 0.410075, 1.0, 0.374416, -0.260137}, {0.404792, 0.314375, 0.374416, 1.0, 0.293135}, {-0.32746, -0.0636438, -0.260137, 0.293135, 1.0}}

m //TableForm

1.000000, 0.635562, 0.698852, 0.404792, -0.32746

0.635562, 1.000000, 0.410075, 0.314375, -0.0636438

0.698852, 0.410075, 1.000000, 0.374416, -0.260137

0.404792, 0.314375, 0.374416, 1.000000, 0.293135

-0.32746, -0.0636438, -0.260137, 0.293135, 1.000000

In asset management one wants diversification. If two or more assets in a portfolio correlate too highly it concentrates rather than diversifying the risk.

What we want

We want an approach or method to construct a portfolio of assets that minimizes the portfolio’s “concentration” risk while always holding some position in all instruments

I’ll illustrate concentration risk with a couple of examples below, but first...

Why this is an interesting problem or question?

A couple of things make this an interesting and challenging question:

While related to “efficient frontier” we have no assumptions about future performance for the individual instruments to use.

Minimizing variance gives an answer, but not an intuitively satisfying or even useful one.

Principal Components Analysis seems a natural way to look at this, but also doesn’t appear to give us what we need.

We’ve looked at using entropy maximization, but while one of our guys familiar with discrete entropy thought it seemed promising, when we tried thinking about this in terms of continuous entropy it proved a dead end. The following code gives an idea of what we looked at (not certain it actually runs):

(* This function computes the distribution of returns, given a \
distribution of price movements.*)

returnDistribution[P_, n_] := Module[ {xv, wv},
  wv = Array[w, n];
  xv = Array[x, n];
  varsInPlane = 
   Append[Delete[xv, n], (r - Total[Delete[wv*xv, n]])/wv[[n]]];
  Integrate[PDF[P, varsInPlane], 
   Map[Function[{#, -Infinity, Infinity}], Delete[xv, n]]
   ]
  ]

dist = MultinormalDistribution[
   ConstantArray[0, 3], {{1, 0.9, 0}, {0.9, 1, 0}, {0, 0, 1}}];
rd = returnDistribution[dist, 3]

(* The problem we want to solve is this *)
wv = Array[w, n];
Maximize[{rd*Log[rd, 2], 
  Total[wv] == 1 && And @@ Thread[wv >= 0]}, wv]

Attilio Meucci has a paper "Managing Diversification" and some MatLab code that looked promising,

http://www.symmys.com/AttilioMeucci/Research/PublFinance/PublFinance.html

but after looking one of my colleagues commented on it as:

He seems to be doing roughly the same thing I wanted to do (before discarding the idea), except he is cheating a bit. Instead of actually computing the continuous entropy, he is just treating each principal component as a discrete bet. But basically, his paper is a refined version of the calculations I did with multiple correlated coinflips. This forces you to get entropy from other variables, rather than from extra digits of the current variable.

To do what he is doing to a non-normal distribution might be trickier, though I imagine that one could use generative models for situations of the form "95% standard normal distribution, 5% something really bad".

Examples of concentration risk

A couple of simple examples to illustrate concentration risk

It's easiest to understand what we want to achieve if we look at a portfolio of 3 assets in a thought experiment. Assume 2 of the instruments have correlations of 1 (100%) and the third at 0, it's correlation matrix would look like this:

1, 1, 0

1, 1, 0

0, 0, 1

From our perspective in this case, it would make sense to put 25% in each of the 2 correlated stocks and 50% in the uncorrelated one.

25%, 25%, 50%

This offsets the risk of concentrating in correlated instruments, while recognizing that the 100% correlated assets are in fact still different instruments whose correlation in the future may change.

One might make the case that as the two assets that have 100% correlation move the same, then a wide range of possible allocations could equally serve our purposes e.g:

50%, 0%, 50%

0%, 50%, 50%

10%, 40%, 50%

... or any such variations on the theme.

But, as we don’t know how their future correlation will evolve, we think the best and most intuitive solution remains at:

25%, 25%, 50%

Another example

In a portfolio of 5 assets with 4 having 100% correlation and 1 having 0% correlation the correlation matrix would look like the following:

1, 1, 1, 1, 0

1, 1, 1, 1, 0

1, 1, 1, 1, 0

1, 1, 1, 1, 0

0, 0, 0, 0, 1

and the portfolio allocation we want would have the following proportions:

12.5%, 12.5%, 12.5%, 12.5%, 50%

Of course the real world presents us with greater complication.

Things we’ve tried

Minimizing variance (promising but doesn’t work)

Someone suggested minimizing variance to do this, but as one can see it doesn’t produce an intuitive solution:

Some Mathematica code illustrating this follows:

For 3 assets:

m3 = {{1, 1, 0}, {1, 1, 0 }, { 0, 0 , 1}};

Array[x, Length@m3];

Minimize[{p.m3.p, Tr@p == 1 && And @@ Thread[p >= 0]}, p]

{1/2, {x[1] -> 1/4, x[2] -> 1/4, x[3] -> 1/2}}

This looks good. It gives us:

25%, 25%, 50%

but...

For 5 assets:

m5 = {{1, 1, 1, 1, 0}, {1 , 1, 1, 1, 0 }, {1 , 1, 1, 1, 0 }, {1 , 1,1, 1, 0 }, { 0, 0 , 0, 0, 1}};

p = Array[x, Length@m5];

Minimize[{p.m5.p, Tr@p == 1 && And @@ Thread[p >= 0]}, p]

{1/2, {x[1] -> 1/16, x[2] -> 1/4, x[3] -> 1/8, x[4] -> 1/16, x[5] ->1/2}}

Not so good as it gives us:

6.25%, 25%, 12.50%, 6.25%, 50%

So, minimizing variance doesn’t work even for this simple (if artificial) case let alone something more realistic.

A promising solution

One contributor to our discussion suggested a promising approach - at least for cases that do not have any negative correlations. Perhaps it would lead someone to suggest a more complete solution.

Again with Mathematica code:

m = {{1, 1, 0}, {1, 1, 0}, {0, 0, 1}};

Tr /@ PseudoInverse[m]/Tr[ Tr /@ PseudoInverse[m]]

{1/4, 1/4, 1/2}  

Exactly what we would want. Note: For those not familiar with Mathematica code the functions: “Tr” finds the trace of a matrix and “/@” maps a function to a list or matrix. The rest probably makes sense.

Another example for four assets:

m = {{1, 1, 1, 0}, {1, 1, 1, 0}, {1, 1, 1, 0}, {0, 0, 0, 1}};

Tr /@ PseudoInverse[m]/Tr[ Tr /@ PseudoInverse[m]]

{1/6, 1/6, 1/6, 1/2}

Again, exactly what we want.

This works better than minimizing variance, but in a more real world example (the first one described in the post) we get something that doesn’t work:

m = {{1.0, 0.635562, 0.698852, 0.404792, -0.32746}, {0.635562, 1.0, 0.410075, 0.314375, -0.0636438}, {0.698852, 0.410075, 1.0, 0.374416, -0.260137}, {0.404792, 0.314375, 0.374416, 1.0, 0.293135}, {-0.32746, -0.0636438, -0.260137, 0.293135, 1.0}}

Tr /@ PseudoInverse[m]/Tr[ Tr /@ PseudoInverse[m]]

{0.267806, 0.0898877, 0.22403, -0.0541658, 0.472441}

In this case we have a negative allocation (-0.0541658) for the 4th asset, something that doesn’t make sense for what we want to achieve.

Conclusion

So, we need a kind of MinimizeVarianceIntuitively function. I hope all of this describes what we want to achieve clearly enough. Any suggestions or ideas for attacking this problem in completely different ways or for extending any of the things we’ve tried already much appreciated.

People who look at this casually alway suggest that CAPM (Capital Asset Pricing Model) does this. It does not. Consider that the correlations described above relate asset to asset not each asset to the "market", so it has no "beta" with which to work. Also, the described problem makes no assumptions of either return or use of a risk free rate or a normal distribution of prices. I can see how CAPM might seem like a way to do this, but I don't see how it applies.

Many thanks,

Arun Garapata

Answer 1

I'll propose a simple and slightly flawed approach. For each asset sum its covariances. Then take its percentage inversely to the ratio of that sum with the total sum of covariances.

In code, using your original example, we'd do as follows.

cormat = {{1.0, 0.635562, 0.698852, 
0.404792, -0.32746}, {0.635562, 1.0, 0.410075, 
0.314375, -0.0636438}, {0.698852, 0.410075, 1.0, 
0.374416, -0.260137}, {0.404792, 0.314375, 0.374416, 1.0, 
0.293135}, {-0.32746, -0.0636438, -0.260137, 0.293135, 1.0}};

n = Length[cormat];
xvars = Array[x, n];
corrs = Map[Total, cormat];
cortot = Total[Total[cormat]];

unnormalized = 
  xvars /. First[Solve[xvars == cortot/corrs, xvars]]

Out[26]= {4.129760099114916, 4.337254104111004, 4.479986290069387, \ 4.173066277624756, 15.51647047130198}

normalized = unnormalized/Total[unnormalized]

Out[27]= {0.1265379371734397, 0.1328956583819156, 0.1372690447157368, \ 0.1278648603757522, 0.4754324993531559}

I suspect these values comprise a reasonable outcome for this example.

There are two flaws. One is that this does not account for relative strength of assets. I don't think that's an issue because I suspect you already weighted for that, i.e. one dollar of asset 1 is worth one dollar of asset 2 is worth...

--edit--

This last cannot be quite right. You'd need to account both for price of assets, and estimated returns of same. Not sure how you want to handle this in the code i showed. Maybe weight covariances slightly so that "better" assets (in terms of returns vs. price) give better covariance numbers?

--end edit--

A bigger issue is that this will not play nice when there are large negative correlations. How to handle that depends on what exatly is the behavior you want. For example, if a pair of assets are negatively correlated, do you find them preferable to zero correlation? If no, then an easy fix would be to turn negatives into some fraction of positive correlation (or make them zero, if negative correlation is considered no better or worse than uncorrelated assets).

If, as I suspect, a pair with negative correlation makes them together relatively more desirable to have positions (makes you more recession-proof), then we have to adjust the code above so we do not blow up the values when correlation totals hit or exceed -1. A reasonable heuristic, I think, would be to leave the negatives in a row alone if their sum is in the range (-0.5,0), else normalize them to sum to -0.5.

Obviously all this is heuristic, and parameters can be tuned e.g. could use inverse-power relation instead of just inverse. Could handle negatives in some way other than what I propose. Could add a reading of tea leaves...

Note I am out of my field here so any misuse of terminology is unintentional. Maybe also out of my depth.

Daniel Lichtblau Wolfram Research

Answer 2

In your extreme test examples you there are an infinite number of solutions; Mma picks out one of them. If you want it to pick out a solution that makes the (unidentified) weights roughly equal to each other, then you should tell it that you care about that aspect: For example you can add a penalty for deviating from equality. Try this:

lambda = .00001;
NMinimize[{p.m5.p - 
   lambda*PDF[DirichletDistribution[Table[2, {5}]], Most[p]], 
  Tr@p == 1 && And @@ Thread[p >= 0]}, p]

I've used the Dirichlet distribution with a mean of 1/5. The tuning parameters include both lambda and the parameter "2" in the Dirichlet distribution (which controls the concentration around the mean). You will have to horse around with these tuning parameters depending on the actual problem. For your less extreme example, you might try

lambda = .0001;
NMinimize[{p.m.p - 
   lambda*PDF[DirichletDistribution[Table[2, {5}]], Most[p]], 
  Tr@p == 1 && And @@ Thread[p >= 0]}, p]

--Mark