I suggest to take a look at dead-simple brunch, this is official skeleton which is maximally minimal.
https://github.com/brunch/dead-simple
Maybe still take it and add your own framework, lang plugins and stuff.
Or, you can re-create your own, by following the same path: add bower.json
, brunch-config
and package.json
. That's it, I guess.