I finally figured this out. The Renderer code was a bit of a red herring because, as Stephan (stwissel) commented, Renderer.encodeChildren does nothing. Looking at the source of several ExtLib components, none of them use it.
The change I made was to the extension class. After reading Keith Strickland's blog (http://www.keithstric.com/A55BAC/keithstric.nsf/default.xsp?documentId=82770C11FA7B9B21852579C100581766 ) and examining com.ibm.xsp.extlib.component.layout.UIVarPublisherBase , I implemented the FacesComponent interface. In particular, I wanted to call the buildContents method which invokes the FacesComponentBuilder. The JavaDoc for FacesComponent has this to say about FacesComponent:
buildContents Build the component children and facets, the default implementation is usually: builder.buildAll(context, this, true); // includeFacets=true
My extension class ended up looking like this:
package com.example.component;
import javax.faces.FacesException;
import javax.faces.component.UIComponentBase;
import javax.faces.context.FacesContext;
import com.ibm.xsp.component.FacesComponent;
import com.ibm.xsp.page.FacesComponentBuilder;
public class CustomLayout extends UIComponentBase implements FacesComponent {
public CustomLayout(){
super();
setRendererType("com.example.applicationlayout");
}
@Override
public String getFamily() {
return "com.example.applicationlayout";
}
public void buildContents(FacesContext context, FacesComponentBuilder builder)
throws FacesException {
builder.buildAll(context, this, true);
}
public void initAfterContents(FacesContext context) throws FacesException {
}
public void initBeforeContents(FacesContext context) throws FacesException
}
}
It now seems to work as I wanted.
As an aside, other objects in the ExtLib (UIWidgetContainer and UIList specifically) have different ways of rendering their children without implementing FacesComponent. That probably requires a blog post.