You are not transforming the frame, only the layer. The layer is only the visual representation of the cell, and does not define size or position of the cell.
I believe, in a tableView, the frame of a cell is defined first by the heightForRowAtIndexPath:
method, but you should be able to change and animate it later using the "frame" property. Try setting it to zero, and animate to the final height when the cell appears.
Code, that finally gave the solution: https://www.cocoacontrols.com/controls/jtgesturebasedtableviewdemo