Here is a method that should work for all grid sizes:
Count the number of dots in each row, and the number of dots in each column.
Then remove all dots in rows and columns that have less than the average number of dots.
You can construct such a histogram efficiently with a dictionary:
var numDotsPerRow = new Dictionary<int,int>();
var numDotsPerColumn = new Dictionary<int,int>();
foreach (var point in pointArray)
{
int count;
numDotsPerRow.TryGetValue(point.X, out count);
numDotsPerRow[point.X] = count+1;
numDotsPerColumn.TryGetValue(point.Y, out count);
numDotsPerColumn[point.Y] = count+1;
}