it was just a small mistake in sending stanza , i missed the namespace
attribute in max
tag , following is the correct stanza
<iq type='get' id='334'>
<list xmlns='urn:xmpp:archive'
with='user4@mydomain.com'>
<set xmlns='http://jabber.org/protocol/rsm'>
<max xmlns='http://jabber.org/protocol/rsm'>30</max>
</set>
</list>
</iq>