TeamCity has support for server-side and agent-side source control checkout. If you are getting JVM out of memory errors during checkout then you must have your build configuration configured for server-side checkout, because agent-side checkout requires git to be installed on the agent, rather than TeamCity using its own jgit based process server-side.
You are correct, you'll need to either edit the Tomcat configuration, or add the additional entires into <TeamCity Data Directory>/config/internal.properties
(documentation).