The authentication Element (ASP.NET Settings Schema) is only valid at the application level. You cannot have different authentication modes in the same application.
You can, however, specify a location Element (ASP.NET Settings Schema) to a specific area and specify an authorization Element (ASP.NET Settings Schema) inside it that allows anonymous users. You can also place an authorization element in a web.config file under a child directory of the application.
In order to use two different authentication schemes, you'll have to convert that area into a separate application. It can, if you wish, still be a child of the parent application, but there are some caveats. One of the caveats, of course, is that session state will not be shared between the two applications.
Also, see the Nested ASP.NET 'application' within IIS inheriting parent config values? question and ASP.NET Configuration File Hierarchy and Inheritance.