I am working with a very old login system that my company used before on a website that used frames.
If this is a new website, don't use it. Login forms are a dime a dozen and can be done in your sleep. Start fresh and do it right.
So I need a few things to happen for this application. First, I need the initial popup window to stay up and the page should not submit if the combination of user/pass is wrong, and finally I need the error message to appear somewhere on the popup.
You're going to want to use an AJAX solution here, either write your own or use a good library like jQuery. Once you check the login values you can use jQuery or simple javascript to unhide or update the innerHTML of an empty element to display your error message.
<cflogin ...>
...
</cflogin>
CFLogin makes me sad. Another one of ColdFusion's tags meant to simplify something commonly done that doesn't really help much and sacrifices flexibility. You can get far more control over your application without it. instead of CFLogin, try something like this pseudo code
<cfcomponent>
<cffunction name = "onRequest" ...>
<cfargument name="targetPage" type="String" required = "true" />
<cfif !structKeyExists(session, "roles") and !findNoCase("loginHandler.cfm",cgi.script_name)>
<!--- notice I prevent the redirect on the form handler, otherwise the ajax will get redirected to the login.cfm page --->
<cfinclude template = "login.cfm">
<cfelse>
<cfinclude template = "#arguments.targetPage#">
</cfif>
</cffunction>
</cfcomponent>
Your login.cfm would then contain your form but your button would fire something like jQuery.post() to "loginHandler.cfm", then depending on the result of the login, your callback function may use jQuery.html() to display the error or window.location.replace / window.location.href if the login was successful. Of course, in the event of a successful login, your ColdFusion page would have to create their session variables and do whatever else you want it to do before sending the result back to your AJAX call.