Here is a simple example on how I would achieve this:
HTML
<form method="post" onsubmit="return openModal()">
<input type="hidden" name="price" class="price" value="100.00">
<input type="submit" class="submit" value="Pay Now">
</form>
<div id="modal" style="display:none"></div>
Javascript
function openModal(){
// when you click the submit button
$('.submit').click(function(){
// Add an iframe inside the modal window, throw in the form value into the src URL
$('#modal').html('<iframe src="http://www.domain.com/checkout.php?price=' + $('.price').val() + '"></iframe>');
// Display the modal window
$('#modal').show();
}
// Don't submit this form (we don't want the page to reload)
return false;
}
So inside the iframe's source file, you would just need to grab the URL parameters and calculate them.
This is a simple example, no where near as involved as the example link you provided (they are using Ajax calls to achieve this), but this should get you started.