Here is how sidebar works. If you render it to a div, it will use that divs height and width and will generate several internal divs that will be used for sidebar purposes. In this case, here is what I think you should do.
Create sidebar as you normally would
After sidebar is created and rendered (or if it is re-rendered), apply custom scorlll bar methods to the following div
$(w2ui['sidebar'].box).find('.w2ui-sidebar-div'); // assuming your sidebar name is 'sidebar'
Sidebar has onRender event that can be used to apply custom scrollbar.
I hope it helps.