Excel UDF calculation should return 'original' value

Question

I've been struggling with a VBA problem for a while now and I'll try to explain it as thoroughly as possible.

I have created a VSTO plugin with my own RTD implementation that I am calling from my Excel sheets. To avoid having to use the full-fledged RTD syntax in the cells, I have created a UDF that hides that API from the sheet. The RTD server I created can be enabled and disabled through a button in a custom Ribbon component.

The behavior I want to achieve is as follows:

• If the server is disabled and a reference to my function is entered in a cell, I want the cell to display Disabled
• If the server is disabled, but the function had been entered in a cell when it was enabled (and the cell thus displays a value), I want the cell to keep displaying that value
• If the server is enabled, I want the cell to display Loading

Sounds easy enough. Here is an example of the - non functional - code:

Public Function RetrieveData(id as Long)
  Dim result as String

  // This returns either 'Disabled' or 'Loading'
  result = Application.Worksheet.Function.RTD("SERVERNAME", "", id)
  RetrieveData = result

  If(result = "Disabled") Then

    // Obviously, this recurses (and fails), so that's not an option
    If(Not IsEmpty(Application.Caller.Value2)) Then

      // So does this
      RetrieveData = Application.Caller.Value2

    End If

  End If
End Function

The function will be called in thousands of cells, so storing the 'original' values in another data structure would be a major overhead and I would like to avoid it. Also, the RTD server does not know the values, since it also does not keep a history of it, more or less for the same reason.

I was thinking that there might be some way to exit the function which would force it to not change the displayed value, but so far I have been unable to find anything like that.

Any ideas on how to solve this are greatly appreciated!

Thanks, Che

EDIT:
Due to popular demand, some additional info on why I want to do all this: As I said, the function will be called in thousands of cells and the RTD server needs to retrieve quite a bit of information. This can be quite hard on both network and CPU. To allow the user to decide for himself whether he wants this load on his machine, he or she can disable the updates from the server. In that case, he or she should still be able to calculate the sheets with the values currently in the fields, yet no updates are pushed into them. Once new data is required, the server can be enabled and the fields will be updated.

Again, since we are talking about quite a bit of data here, I would rather not store it somewhere in the sheet. Plus, the data should be usable even if the workbook is closed and loaded again.

Answer 1

Different tack=new answer.

A few things I've discovered the hard way, that you might find useful:

1. In a UDF, returning the RTD call like this

' excel equivalent: =RTD("GeodesiX.RTD",,"status","Tokyo")
result = excel.WorksheetFunction.rtd( _
    "GeodesiX.RTD", _
    Nothing, _
    "geocode", _
    request, _
    location)

behaves as if you'd inserted the commented function in the cell, and NOT the value returned by the RTD. In other words, "result" is an object of type "RTD-function-call" and not the RTD's answer. Conversely, doing this:

' excel equivalent: =RTD("GeodesiX.RTD",,"status","Tokyo")
result = excel.WorksheetFunction.rtd( _
    "GeodesiX.RTD", _
    Nothing, _
    "geocode", _
    request, _
    location).ToDouble ' or ToString or whetever

returns the actual value, equivalent to typing "3.1418" in the cell. This is an important difference; in the first case the cell continues to participate in RTD feeding, in the second case it just gets a constant value. This might be a solution for you.

2. MS VSTO makes it look as though writing an Office Addin is a piece of cake... until you actually try to build an industrial, distributable solution. Getting all the privileges and authorities right for a Setup is a nightmare, and it gets exponentially worse if you have the bright idea of supporting more than one version of Excel. I've been using Addin Express for some years. It hides all this MS nastiness and let's me focus on coding my addin. Their support is first-rate too, worth a look. (No, I am not affiliated or anything like that).

3. Be aware that Excel can and will call Connect / RefreshData / RTD at any time, even when you're in the middle of something - there's some subtle multi-tasking going on behind the scenes. You'll need to decorate your code with the appropriate Synclock blocks to protect your data structures.

4. When you receive data (presumably asynchronously on a separate thread) you absolutely MUST callback Excel on the thread on which you were intially called (by Excel). If you don't, it'll work fine for a while and then you'll start getting mysterious, unsolvable crashes and worse, orphan Excels in the background. Here's an example of the relevant code to do this:

    Imports System.Threading
    ...
    Private _Context As SynchronizationContext = Nothing
    ...
    Sub New
      _Context = SynchronizationContext.Current
      If _Context Is Nothing Then
         _Context = New SynchronizationContext ' try valiantly to continue    
      End If
    ...
    Private Delegate Sub CallBackDelegate(ByVal GeodesicCompleted)

    Private Sub GeodesicComplete(ByVal query As Query) _
        Handles geodesic.Completed ' Called by asynchronous thread

        Dim cbd As New CallBackDelegate(AddressOf GeodesicCompleted)

        _Context.Post(Function() cbd.DynamicInvoke(query), Nothing)
    End Sub
    Private Sub GeodesicCompleted(ByVal query As Query)

        SyncLock query

            If query.Status = "OK" Then

                Select Case query.Type

                    Case Geodesics.Query.QueryType.Directions
                        GeodesicCompletedTravel(query)

                    Case Geodesics.Query.QueryType.Geocode
                        GeodesicCompletedGeocode(query)

                End Select
            End If

            ' If it's not resolved, it stays "queued", 
            ' so as never to enter the queue again in this session
            query.Queued = Not query.Resolved

        End SyncLock

        For Each topic As AddinExpress.RTD.ADXRTDTopic In query.Topics
            AddinExpress.RTD.ADXRTDServerModule.CurrentInstance.UpdateTopic(topic)
        Next

    End Sub

5. I've done something apparently akin to what you're asking in this addin. There, I asynchronously fetch geocode data from Google and serve it up with an RTD shadowed by a UDF. As the call to GoogleMaps is very expensive, I tried 101 ways and several month's of evenings to keep the value in the cell, like what you're attempting, without success. I haven't timed anything, but my gut feeling is that a call to Excel like "Application.Caller.Value" is an order of magnitude slower than a dictionary lookup.

In the end I created a cache component which saves and re-loads values already obtained from a very-hidden spreadsheet which I create on the fly in Workbook OnSave. The data is stored in a Dictionary(of string, myQuery), where each myQuery holds all the relevant info.

It works well, fulfils the requirement for working offline and even for 20'000+ formulas it appears instantaneous.

HTH.

---

Edit: Out of curiosity, I tested my hunch that calling Excel is much more expensive than doing a dictionary lookup. It turns out that not only was the hunch correct, but frighteningly so.

Public Sub TimeTest()
    Dim sw As New Stopwatch
    Dim row As Integer
    Dim val As Object
    Dim sheet As Microsoft.Office.Interop.Excel.Worksheet
    Dim dict As New Dictionary(Of Integer, Integer)

    Const iterations As Integer = 100000
    Const elements As Integer = 10000

    For i = 1 To elements + 1
        dict.Add(i, i)
    Next
    sheet = _ExcelWorkbook.ActiveSheet

    sw.Reset()
    sw.Start()
    For i As Integer = 1 To iterations
        row = 1 + Rnd() * elements
    Next
    sw.Stop()
    Debug.WriteLine("Empty loop     " & (sw.ElapsedMilliseconds * 1000) / iterations & " uS")

    sw.Reset()
    sw.Start()
    For i As Integer = 1 To iterations
        row = 1 + Rnd() * elements
        val = sheet.Cells(row, 1).value
    Next
    sw.Stop()
    Debug.WriteLine("Get cell value " & (sw.ElapsedMilliseconds * 1000) / iterations & " uS")

    sw.Reset()
    sw.Start()
    For i As Integer = 1 To iterations
        row = 1 + Rnd() * elements
        val = dict(row)
    Next
    sw.Stop()
    Debug.WriteLine("Get dict value " & (sw.ElapsedMilliseconds * 1000) / iterations & " uS")

End Sub

Results:

Empty loop     0.07 uS
Get cell value 899.77 uS
Get dict value 0.15 uS

Looking up a value in a 10'000 element Dictionary(Of Integer, Integer) is over 11'000 times faster than fetching a cell value from Excel.

Q.E.D.

Answer 2

Maybe... Try making your UDF wrapper function non-volatile, that way it won't get called unless one of its arguments changes.

This might be a problem when you enable the server, you'll have to trick Excel into calling your UDF again, it depends on what you're trying to do.

Perhaps explain the complete function you're trying to implement?

Answer 3

You could try Application.Caller.Text
This has the drawback of returning the formatted value from the rendering layer as text, but seems to avoid the circular reference problem.
Note: I have not tested this hack under all possible circumstances ...