I don't really understand why you want to populate your listBox using a popup.
But if you modify the underlying bean values, you need to rerender the <h:selectOneListbox/>
to reflect these changes. If you are using (or can use) the richfaces framework it can be done quite easily through an ajax call.
The ajax4jsf component called <a4j:jsFunction/>
can do the job. For example in your parent page add :
<a4j:jsFunction name="refreshListbox" reRender="booksListBox" limitToList="true"/>
and in your popup just call this js function when a new book value is selected :
<script type="text/javascript">
window.opener.refreshListbox();
</script>
You could also use a <rich:modalPanel>
instead of a popup to stay in the same page and interact more easily with your jsf components.