It's possible you're looking for jsctypes, which can do dlloads and make FFI wrappers for C functions exported from those DLLs.
It's also possible you're looking for something more like full-custom extension modules. You could write a builtin that dlloads a shared object, and grabs a symbol out of it that tells you what spidermonkey functions are contained therein and registers that bunch of functions. I don't recall any pre-canned way to do that -- SpiderMonkey was often extended in non-trivial ways by embedders, but it wasn't really a project priority to bake such things into the shell out of the box. Could always ask on #jsapi channel of irc.mozilla.org to double check!