Finally, I found the answer. I am sure StackOverflow came in somewhere in the mix. I cannot quote the post or question now but I have all the links that enabled Silverlight run inside a browser but under trusted privileges. Now Silverlight can both run inside the browser and access the USB card reader hardware on the client PC. Here are the links
https://github.com/cureos/nativedll/blob/master/NativeDllInvoker/NativeDllHelper.cs
http://www.mistergoodcat.com/post/Silverlight-5-Tidbits-Trusted-applications
http://msdn.microsoft.com/en-us/library/gg192793(v=vs.96).aspx
http://technet.microsoft.com/en-us/library/cc730989(WS.10).aspx
At least maf748's answer got me thinking in that direction. Thanks to the SO community!