The problem is in this line: -
if( !this->sceneRect().intersects(item->boundingRect()))
This is comparing the scene's rect, which is in scene coordinates, with the item's bounding rect, which is in the item's local coordinate system.
You need to convert one of them so that you're comparing within the same coordinate system.
QRectF itemSceneBoundingRect = item->mapRectToScene(item->boundingRect());
if( !this->sceneRect().intersects(itemSceneBoundingRect)
{
// remove the item.
}