This is a known bug in UsbManager, for several years already: Issue 20545: Unable to cleanly close USB accessory FileInputStream
A fix developed but haven't merged into Android trunk yet: Issue 61390: Potential source code fixes for UsbAccessory not unbinding cleanly
Also you may want to look at this discussion: Proper way to close a USB accessory connection to workaround ideas in some use cases.
I'm working with a customized Android ROM (CyanogenMod) so I'll probably try to apply the fix from 61390. I did a lot of testing trying to workaround it myself, but I was not able to get a stable solution. I was considering to use libusb port instead: https://github.com/OpenNI/OpenNI2/tree/master/ThirdParty/PSCommon/XnLib/ThirdParty/libusb-1.0.9-Android (from https://stackoverflow.com/a/16234205/1028256, you can upvote it there), it is told to work even on not rooted phones (I haven't tried myself), and it supports async mode, so one can avoid blocking thread in Java. But you will have to use JNI, still it's much better than any non-stable solution IMHO.