I solved it with this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.2</version>
<configuration>
<maxmemory>800m</maxmemory>
</configuration>
</plugin>
before the maven-release-plugin definition.