I ended up doing this in two steps:
foreach (var column in columnNumbers)
{
yield return data.Select(m => m[column].Value.ToString()).ToList();
}
Now I have the data I need, but with the rows and columns swapped, so i had to swap rows for columns and vice versa:
for (int i = 1; i < rowCount; i++)
{
var newRow = new List<string>();
foreach (var cell in list)
{
newRow.Add(cell[i]);
}
yield return newRow;
}