Global Keyboard hook for (Excel) automation addin (not VSTO)

Question

Apologies for the length of this post, but I think the background is necessary to convey what I’m trying to achieve.

I’ve been tasked with updating an old Excel UDF addin (it was previously using the api code from Steve Dalton’s book with JNI and just about every other technology imaginable). The functions were very unsympathetic to Excel’s calculation model – these functions took several ranges and wrote data back to them (in another thread) while at the same time allowing users to edit this data (which was then saved and uploaded to a remote server). All this made it abysmally slow to load, but gave users the necessary flexibility to change the data as needed.

I’ve migrated it to C# (it takes the data from a remote Java server via WSDL), but I found that most functions were being called 50+ times and it ran far, far slower than the original addin (it’s an automation addin using Extensibility.IDTExtensibility2 – so no VSTO tricks available here).

In a fit of desperation, I decided to try rewrite the swathes of UDFs to be array functions and not accept input (Excel would complain about overwriting an array) – obviously this is now several orders of magnitude faster, but missing the key requirement than users be able to modify the output data.

Realising Excel didn’t provide any before-edit-is-committed validation callback events (I’d played around with Worksheet.OnEntry and added a VBComponent but it isn’t triggered before the error about overwriting an array or list data validation).

I assumed it would be simple enough to implement a global keyboard hook so started writing some windows forms to intercept edits (just a TextBox for single form entries and a ComboBox for cells with list data validation) and also got it copying data from the clipboard into the selected range.

Currently all of this is driven from bespoke context-menu entries (or double click), which will not be accepted by the users – I must, as a minimum, be able to intercept F2, Ctl+V and direct typing on an active cell. But I have no idea how to register a global keyboard hook within an automation addin.

So my question is; Is it possible to intercept every edit attempt and provide my own handling? Or failing that, how can I register a global keyboard hook to intercept F2, Ctl+V and direct typing on an active cell?

I’ve tried the hook found here Using global keyboard hook (WH_KEYBOARD_LL) in WPF / C# but cannot get the App.xaml + App.xaml.cs working in this context (this is my first encounter with C# and windows programming in general), so may be someone just needs to enlighten me wrt the App.xaml + App.xaml.cs configuration ().

Please note; this is not an VSTO addin, it’s implemented using Extensibility.IDTExtensibility2.

UPDATE EDIT: @TimWilliams and @CharlesWilliams enquired why my previous version , that did read-write on it's functions' arguments, had so many repeat invocations.

All the functions have a mandatory id key parameter, most also take a date or date range, the following is what occurred (in quite a large workbook ~30 sheets with graphs):

1. When the workbook is first loaded the function calls all fire with stale (previously saved) values but these are disregarded as the first line of C# in each function is a (quick) test against the backing model to see whether the model has been loaded, no function arguments are inspected/unmarshalled.

2. The user chooses to load a model via webservices or previously serialized file.

3. Visual updates are disabled and a setup sheet is populated with some data, in order; some key dates (beginning and end of dates - various date ranges on different sheets are calculated with EMONTH +/- 12), some other static data (author/editor name etc), and finally the mandatory key id (that identifies the model data)

4. Now each function's method in the automation addin has the id key so if data is found it's returned otherwise default values from function parameters are used. (Note: The model maintains a copy of the objects' fields requested by functions so it knows what has been output. For further invocations, if data is present in the cache layer it is updated (hand rolled clone of either original model data or previous cache value) with the incoming function parameters - the diff of cache layer and model data is later uploaded to a webservice) - Not well explained "cache" is actually a wrapper around same data structure as the model data

5. If data is returned (indicating first load), it is put onto a blocking queue that is served by a worker thread that writes the data back to the ranges specified.

The abysmal performance seems to stem from very long chains of dependant functions (confusing Excel?), and the fact that the the functions seem to be invoked before their dependencies, e.g.

Given DATE_RANGE is a chain A1-An, of An=EMONTH(An-1,12) where A1 is a constant LAST_DATE from the setup page that has already been populated

Then fn(MODEL_ID, DATE_RANGE) is invoked once the named cell MODEL_ID is populated, but DATE_RANGE has incorrect values, and fn is called repeatedly as each EMONTH completes, and the function methods try to convert ranges to dates (if invalid dates then returning early). Meanwhile the worker thread starts throwing application busy exceptions (and so re-queuing the ranges writes and sleeping for arbitrary period of 250ms). Eventually the contention subsides, but you'll have a chance to make and starting drinking a coffee first (probably even grind the beans too).

Having written this horrid code, I considered just writing dates out to the setup sheet and then waiting for calculation to stop before updating the MODEL_ID - this would go some way to reducing the number of functions calls. However intercepting just the edits, holding those updates in the model and marking the corresponding range as dirty seemed much cleaner.

I think the available options are, either;

• In the edit intercept version, try the vb hook OnKey with a call for every possible ASCII function to callback into a parameterized C# command (the VB code can at least be generated in a loop)
• Try the edit intercept version as a VSTO addin (this should give me the key bindings)
• Use ExcelDNA - it's looking tempting for the (previous) read-write range parameter version (which may prove of adequate performance (which would probably indicate a logical bug in my Excel handling code).

(Apologies again for the length and lack of clarity)

Answer 1

You should be able to solve your calculation chain problem since it sounds like a discrete set of sequential steps.
If using C++/XLL you would make the function arguments type P which would ensure they were calculated by Excel before being passed to the UDF. I think Ex cel DNA/addin Express should have the same effect if the parameters are defined as anything other than Object.

Excel calculates cells in a LIFO sequence which is set by the previous final calculation sequence and any cells that have been entered/changed: so the last formula changed gets calculated first.
So you should enter the formulae in your DATE_RANGE chain in reverse sequence (last one in the chain first)
Presumably you are already switching to Manual calculation mode at the start of this process. So it might be as simple as writing out the setup sheet and dates, then forcing a calculation (Application.calculate) then updating the MODEL_ID, then forcing another calculation.

And of course using Excel DNA the overhead per function call would be much lower anyway.
see http://fastexcel.wordpress.com/2011/07/07/excel-udf-technology-choices-snakes-ladders-with-vba-vb6-net-c-com-xll-interop/

Answer 2

I can't help you with the global keyboard hook, but you really should look at Excel DNA or Addin Express to dramatically improve the performance of your C# UDFs (they interface .NET with the XLL C API which is MUCH faster than c# automation).
Both Excel DNA and Addin Express also have threads in their support forums discussing how to rewrite data back from UDFs to other ranges. IIRC Excel DNA discusses the separate thread approach and Addin Express discusses using command-equivalent type UDFs to trigger a hidden XLM function

And personally I think its going to be very difficult to make your global keyboard hook approach work unobtrusively and efficiently also in all circumstances (multiple workbooks open, VBA, DDE etc etc).