Currently (PrimeFaces 6+) p:rowToggler
uses Ajax loading, so it's lazy by default. After reading p:rowExpansion gets rendered even though it's not toggled, it seems that this wasn't always the case (up to PrimeFaces 5). So, if possible, just upgrade.
And for the multiple getter calls you should, as already pointed out, check: Why JSF calls getters multiple times.