There is no reRender
attribute for <a4j:commandButton>
for the version you are using. You can check the tag reference below
Based on the code I'm seeing (ui:composition
not taken into account), if you want to update the dataTable
simply change
reRender="ruleSearchPanel"
to
render ="searchForm:resultRuleSets"
If you are confused about determining ids check the link below
How can I know the id of a JSF component so I can use in Javascript
Note: Some of the attributes that you are using are not defined in some of the components that you are using. For example
lenght
for rich:popupPanel
selectedClass
for rich:extendedDataTable
label
for rich:column