In my opinion use orangebox for popup. dont use jquery with ajax update panel. but if you insist then do the following
When you use Update panel it will render you page partially so you have to use something which will postback your page and take a look at the code below
<asp:UpdatePanel ID="AjaxPanel" runat="server" UpdateMode="Conditional" ClientIDMode="Static">
<ContentTemplate>
<!-- all your code goes here -->
</ContentTemplate>
<Triggers>
<asp:PostBackTrigger runat="server" ControlID="Button1" /> <!-- that will be the button which will fully postback by clicking on it. -->
</Triggers>
</asp:UpdatePanel>
let me know if this helps for further explanation read this article.