Here is the example of displaying web resource in such dialog box.
In the parent window:
var src = <Relative_Url_of_the_Webresource>;
var DialogOptions = new Xrm.DialogOptions();
DialogOptions.width = 500;
DialogOptions.height = 400;
Xrm.Internal.openDialog(src, DialogOptions, null, null, CallbackFunction);
function CallbackFunction(returnValue){ }
Also, you have to include ClientGlobalContext.js.aspx in your web resource, for example:
<script type="text/javascript" src="/webresources/ClientGlobalContext.js.aspx"></script>
For passing the return value from the dialog box to the parent window:
Mscrm.Utilities.setReturnValue(result);
try {
closeWindow(true); //Close the dialog box
}
catch (e) { }