The build file is set in its own element within the nant element. See below.
<nant>
<executable>c:\nantdir\nant.exe</executable>
<buildArgs>-D:blah_prop=foobar</buildArgs>
<nologo>false</nologo>
<buildFile>default.build</buildFile>
<targetList>
<target>the-nant-target</target>
</targetList>
<buildTimeoutSeconds>9000</buildTimeoutSeconds>
</nant>
buildArgs are just properties to send to Nant. They have nothing to do with the nant file you're trying to use.