You don't set it there - in SSMS, open the AS instance, then right-click and go to properties. You want to set the HardMemoryLimit/LowMemoryLimit/TotalMemoryLimit/VertipaqMemoryLimit properties.
It's percentages of TOTAL RAM ON THE BOX. So if you're running multiple instances (like classic and tabular), you need to share it between them. And you'll need to cap the SQL Server instance as well.