You may call your own function. To achieve this you should change your html like this:
<button type="button" ng-disabled="rowform.$waiting"
ng-click="cancelAdvice(rowform, $index)"
class="btn btn-default">
cancel
</button>
As you can see there is a new function with the form and the current index as parameter. In your controller you have to define this function:
$scope.cancelAdvice = function(rowform, index){
console.log(rowform, index);
$scope.removeUser(index);
rowform.$cancel();
}
Now you can do your own stuff and call the form $cancel if you are done.