As best as I can tell, when an app displays the UIDocumentInteractionController
and the "open in" menu, the apps seem to appear in a "the most-recently-used shows first" order.
I ran an app that generates PDF files and then chose to export the PDF from the app. The list of apps appeared. I chose the 2nd app. I repeated the process. This time, the apps appeared in a different order. The one I chose before was now the first.
There doesn't seem to be much of a specific order for the rest of the listed apps. It could be based on the order the apps were installed.
In other words, there is nothing your app can do. iOS displays the apps in whatever order it sees fit. If the user wants iBooks to be first, then they should choose iBooks. It will be the default the next time.
BTW - Why are users complaining? iBooks is no more special for opening PDF files than any other app.