You should put that files in assets.
On a discussion about Assets-only Pub Packages Bob Nystrom gave some explaination about browser package and pub build
:
It has some gross special-case code to handle the JS files in the browser package. That's been in there since before the idea of an "asset" directory existed, and it got Grandfathered in.