Filling missing data after outer join

Question

I have two time series which are at the same sampling rate. I would like to perform an outer join and then fill in any missing data (post outer join, there can be points in time where data exists in one series but not the other even though they are the same sampling rate) with the most recent previous value.

How can I perform this operating using Deedle?

Edit:

Based on this, I suppose you can re-sample before the join like so:

// Get the most recent value, sampled at 2 hour intervals
someSeries|> Series.sampleTimeInto
  (TimeSpan(2, 0, 0)) Direction.Backward Series.lastValue

After doing this you can safely Join. Perhaps there is another way?

Answer 1

You should be able to perform the outer join on the original series (it is better to turn them into frames, because then you'll get nice multi-column frame) and then fill the missing values Frame.fillMissing.

// Note that s1[2] is undefined and s2[3] is undefined
let s1 = series [ 1=>1.0; 3=>3.0; 5=>5.0 ]
let s2 = series [ 1=>1.1; 2=>2.2; 5=>5.5 ]

// Build frames to make joining easier
let f1, f2 = frame [ "S1" => s1 ], frame [ "S2" => s2 ]

// Perform outer join and then fill the missing data
let f = f1.Join(f2, JoinKind.Outer)
let res = f |> Frame.fillMissing Direction.Forward

The final result and the intermediate frame with missing values look like this:

val it : Frame<int,string> =
     S1        S2        
1 -> 1         1.1       
2 -> <missing> 2.2       
3 -> 3         <missing> 
5 -> 5         5.5       

> 
val it : Frame<int,string> =
     S1 S2  
1 -> 1  1.1 
2 -> 1  2.2 
3 -> 3  2.2 
5 -> 5  5.5 

Note that the result can still contain missing values - if the first value is missing, the fillMissing function has no previous value to propagate and so the series may start with some missing values.