You could try this:
var results = collection.OrderBy(
row => Math.Abs(Convert.ToInt32(row["ColumnName"]) - given_value));
Or Convert.ToDouble, i don't know the type you are using.
Note: Is the column allows null, you should check it for DBNULL
first before Converting.