This is a classic problem. As far as I am concerned, flattening is usually solved with XSLT grouping. Of course, it's not easy to achieve with XSLT 1, when you have to use recursion-based algorithms to implement it. But let's consider a simpler solution (XSLT 2):
<xsl:stylesheet version="2.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:f="http://stackoverflow.com/xslt-functions"
xmlns:doc="http://docbook.org/ns/docbook"
xmlns="http://docbook.org/ns/docbook"
exclude-result-prefixes="#all">
<xsl:variable name="block-elements" select="QName('http://docbook.org/ns/docbook', 'table'),
QName('http://docbook.org/ns/docbook', 'figure')"/>
<xsl:template match="doc:para">
<xsl:for-each-group select="node()" group-adjacent="f:is-block-level-element(.)">
<xsl:choose>
<!-- If current node is a block element process the node as is -->
<xsl:when test="current-grouping-key()">
<xsl:apply-templates select="current-group()"/>
</xsl:when>
<!-- Otherwise, wrap it in para element -->
<xsl:otherwise>
<para>
<xsl:apply-templates select="current-group()"/>
</para>
</xsl:otherwise>
</xsl:choose>
</xsl:for-each-group>
</xsl:template>
<xsl:function name="f:is-block-level-element" as="xs:boolean">
<xsl:param name="node" as="node()"/>
<xsl:sequence select="$node instance of element() and
node-name($node) = $block-elements"/>
</xsl:function>
<xsl:template match="attribute() | node()">
<xsl:copy>
<xsl:apply-templates select="attribute() | node()"/>
</xsl:copy>
</xsl:template>
</xsl:stylesheet>
The destination content will be:
<section xmlns="http://docbook.org/ns/docbook">
<para>some text.....
<literallayout>
</literallayout>
more text....
</para>
<table>
...
</table>
<para>
even more text
</para>
<table>...</table>
<para>
<literallayout>text also look here</literallayout>
more text
<link/>
</para>
</section>