It seems that the latest IFilters in Adobe 8 onwards only support the stream interface used by the Search indexer in Windows. This explains why I was getting 'no interface supported' using the latest versions.
V6.0 of the IFilter becomes flaky when not used in a single threaded application as can be seen here
The first solution I came across was to use Foxit PDF IFilter which works like a charm and if I could squeeze $600 out of my client would certainly buy. I had no problems running using this IFilter.
The second and free solution was to use the Adbobe IFilter 5.0. Nearly 10 years old but implements the file interface and is stable.
Another solution might be to switch to 64 bit. V9.0 of the 64 bit filter from Adobe is working and is stable.