In the open method of your driver: use a global flag to say whether the device is in use. Unset when closed (in the release method). Pseudo-code:
am_open=False
open() method:
if (!am_open)
am_open=True
...proceed...
else
return -EAGAIN; // try again later
-or-
return -EBUSY;
release() method:
am_open=False
Having said that, drivers (usually) should be a 100% reentrant :)