Turns out this is pretty easy:
Simply invoke the tool with its full path, e.g. "C:\cygwin\bin\file.exe".
I was misled by so many googled suggestions that suggest that I need to invoke the shell in order to use the file command that I didn't even try this direct approach first.
The only (small) challenge is to pass any paths to the tool in POSIX format. REALbasic only gives me the DOS-style path format (e.g. from FolderItem.AbsolutePath), and the cygwin tools usually complain about this with a warning. I found to solutions to this:
Either disable the warning in the cygwin installation. That's done with setting the environment variable option "nodosfilewarning".
The other way is to provide a proper POSIX path. The cygwin1.dll provides a function for this (''cygwin_conv_path''), but I could not get this working in RB. Instead, I wrote my own, hoping it covers all cases:
dim objPath as String = f.AbsolutePath
if objPath.Mid(2,2) = ":\" then
objPath = "/cygdrive/"+objPath.Mid(1,1).Lowercase+objPath.Mid(3).ReplaceAll("\","/")
end if