Modify libusb to accept file descriptor

Question

I modified libusb1.0 open function as follows:

static int op_open2(struct libusb_device_handle *handle, int fd) {
    struct linux_device_handle_priv *hpriv = _device_handle_priv(handle);

    hpriv->fd = fd;

    return usbi_add_pollfd(HANDLE_CTX(handle), hpriv->fd, POLLOUT);
}

where fd was obtained via android.hardware.usb.UsbDeviceConnection.html#getFileDescriptor()

final UsbDeviceConnection connection = manager.openDevice(device); 
return connection.getFileDescriptor();

Unfortunatelly I keep getting an error when I call

static int op_claim_interface(struct libusb_device_handle *handle, int iface)
{
    int fd = _device_handle_priv(handle)->fd;

    int r = ioctl(fd, IOCTL_USBFS_CLAIMINTF, &iface);

    if (r) {
        if (errno == ENOENT)
            return LIBUSB_ERROR_NOT_FOUND;
        else if (errno == EBUSY)
            return LIBUSB_ERROR_BUSY;
        else if (errno == ENODEV)
            return LIBUSB_ERROR_NO_DEVICE;

        usbi_err(HANDLE_CTX(handle),
            "claim interface failed, error %d errno %d", r, errno);
        return LIBUSB_ERROR_OTHER;
    }
    return 0;
}

claim interface failed, error -1 errno 9

which is translated to "Bad file number". The file descriptor I get from Java is a positive integer!

The only other small detail is that my native code runs as a separate binary that is spawned with a Java ProcessBuilder. But they share the same uid, so I presume the USB permissions I have from Java should still apply for libusb.

I don't need to be platform independent, so any hacks would do the job :)

Any thoughts would be much appreciated!

Additional information! The output I get from lsof is (shortened to stress the most interesting part of it)

my.activity 13374     u0_a62  exe       ???                ???       ???        ??? /system/bin/app_process
my.activity 13374     u0_a62    0       ???                ???       ???        ??? /dev/null
my.activity 13374     u0_a62    1       ???                ???       ???        ??? /dev/null
my.activity 13374     u0_a62    2       ???                ???       ???        ??? /dev/null
my.activity 13374     u0_a62    3       ???                ???       ???        ??? /dev/log/main
my.activity 13374     u0_a62    4       ???                ???       ???        ??? /dev/log/radio
my.activity 13374     u0_a62    5       ???                ???       ???        ??? /dev/log/events
my.activity 13374     u0_a62    6       ???                ???       ???        ??? /dev/log/system
my.activity 13374     u0_a62    7       ???                ???       ???        ??? /system/framework/core.jar
my.activity 13374     u0_a62    8       ???                ???       ???        ??? /system/framework/core-junit.jar
my.activity 13374     u0_a62    9       ???                ???       ???        ??? /dev/__properties__ (deleted)
...
my.activity 13374     u0_a62   44       ???                ???       ???        ??? /dev/bus/usb/002/002
...    
my.activity 13374     u0_a62   51       ???                ???       ???        ??? pipe:[51015]
my.activity 13374     u0_a62   53       ???                ???       ???        ??? pipe:[51016]
...

my_exe   13546     u0_a62  exe       ???                ???       ???        ??? /data/data/my.activity/files/my_exe
my_exe   13546     u0_a62    0       ???                ???       ???        ??? pipe:[51015]
my_exe   13546     u0_a62    1       ???                ???       ???        ??? pipe:[51016]
my_exe   13546     u0_a62    2       ???                ???       ???        ??? pipe:[51016]
my_exe   13546     u0_a62    3       ???                ???       ???        ??? /dev/log/main
my_exe   13546     u0_a62    4       ???                ???       ???        ??? /dev/log/radio
my_exe   13546     u0_a62    5       ???                ???       ???        ??? /dev/log/events
my_exe   13546     u0_a62    6       ???                ???       ???        ??? /dev/log/system
my_exe   13546     u0_a62    9       ???                ???       ???        ??? /dev/__properties__ (deleted)
my_exe   13546     u0_a62  mem       ???              b3:09         0     302530 /data/data/my.activity/files/my_exe
my_exe   13546     u0_a62  mem       ???              b3:09     36864     302530 /data/data/my.activity/files/my_exe
my_exe   13546     u0_a62  mem       ???              b3:09     40960     302530 /data/data/my.activity/files/my_exe
my_exe   13546     u0_a62  mem       ???              b3:03         0        200 /system/bin/linker
my_exe   13546     u0_a62  mem       ???              b3:03     57344        200 /system/bin/linker
my_exe   13546     u0_a62  mem       ???              b3:03     61440        200 /system/bin/linker

Which makes me think that the file descriptor 44 that I pass to my_exe is actually not inherited!

Answer 1

It turned out that the file descriptor was not inherited by the process. If one uses JNI everything should be fine, but if you want to interface with a third party application, you would need to transfer the file descriptor to the third party application.

My solution was to use UNIX sockets to transfer the fd via JNI and it worked!

P.S.

I released the project under the GNU license - https://github.com/martinmarinov/rtl_tcp_andro-

Answer 2

File handles are private to a process. Passing them to a different process as-is is guaranteed to not work on any flavor of *nix.

Android's Binder/Parcel IPC interface can marshal file descriptors.

Or you can have the handle inherited - if it's already open when the process is spawned, the handle will be valid in the spawned process as well.