First of all, the ||
is the logical or operator in perl. So the documentation snippet you showed means COUNT or ARRAY or HASH; you have to choose one. And if you use COUNT you'll just get back the number of tickets that match your search criteria.
Second, you need to provide the UserID of the agent that does the search, if you don't have one (because your script is for some kind of integration) you can always use ID number 1; which has access to all queues.
Lastly the QueueIDs parameter takes an array of queue IDs; so you can search for multiple queue IDs with the single parameter.
$result = $client->__soapCall("Dispatch", array($user, $pass,
"TicketObject", "TicketSearch",
"UserID", 1,
"Result", "ARRAY",
"QueueIDs", array(1),
"StateType", "Open"
));
Please note that the RPC interface is kind of low level, it exposes the raw OTRS API. It's not very polished and sometimes you need to do a lot of work to get the data you need, on the other hand it's very powerful.
The 'new' interface is the WSDL based web service interface; this is much more encapsulated and clean. It has two drawbacks, 1; PHP does not support using WSDLs as nice as for instance .NET or Java do, and 2; it only exposes a small part of the OTRS API's, only to create/search/modify tickets and to handle the FAQ (Knowledge Base). Stuff such as adding/removing queues, services or customers is not (yet) available.