The sortable binding and jQuery UI sortable really needs a parent element to grab onto for the stuff that you are sorting.
You would want to at least wrap your whole "item" in an element like:
<div class="container" data-bind="sortable: groupOrItems">
<div>
<!--ko if: type=='Group'-->
<div>
<div class="title" data-bind="text: name()"></div>
<!--here I would actually have a nested sortable-->
</div>
<!--/ko-->
<!--ko if: type=='Item'-->
<div class="item" data-bind="text: name()"></div>
<!--/ko-->
</div>
</div>
Updated: http://jsfiddle.net/rniemeyer/FSYYb/