Firstly, ensure that the application pool for the site is running in "Classic Mode". To set this, select the application pool for the site in IIS, click "Basic Settings..." and set "Managed pipeline mode" to "Classic".
Next, select the site in IIS and, click the "Error pages" icon. Click "Edit Feature Settings..." and make sure "error responses" is set to "Custom Error Pages".
You should then find you can set the 404 error to execute your ASP page.