What you want to do is group together the ul[@class='book-section'].
There are different ways to do that, one would be to use for-each-group in the parent template of li[@class='book-section'] so you could change that template to:
<xsl:template match="div[@class='toc']//li[@class='book-section']">
<xsl:element name="li">
<xsl:apply-templates select="xhtml:a[@class='ref-chap']"/>
<xsl:for-each-group select="ul[@class='book-section']" group-by="@class">
<ol>
<xsl:apply-templates select="current-group()/node()"/>
</ol>
</xsl:for-each-group>
</xsl:element>
</xsl:template>
Another way would be to only match the first of the ul[@class='book-section'] but then apply this to the children of the following, too, like this:
<xsl:template match="div[@class='toc']//ul[@class='book-section'][1]">
<xsl:element name="ol">
<xsl:apply-templates select="* | following-sibling::ul[@class='book-section']/*"/>
</xsl:element>
</xsl:template>