How do I find the average and standard deviation in ruby?

Question

I wrote a program which finds the average and standard deviation of a large data set which is in a separate txt file. I want this program to work with any data set. I tested my program by putting in two simple data points (year and month correlating to a temperature):

2009-11,20
2009-12,10

When running this it says that my the average is 20 and the standard deviation is 0 (obviously wrong).

Here is my program:

data = File.open("test.txt", "r+")
contents = data.read

contents = contents.split("\r\n")

#split up array
contents.collect! do |x|
  x.split(',')
end

sum = 0

contents.each do |x|
  #make loop to find average
  sum = sum  + x[1].to_f
end
avg = sum / contents.length
puts "The average of your large data set is: #{ avg.round(3)} (Answer is rounded to nearest thousandth place)"
#puts average

#similar to finding average, this finds the standard deviation
variance = 0
contents.each do |x|
  variance = variance + (x[1].to_f - avg)**2
end

variance = variance / contents.length
variance = Math.sqrt(variance)
puts "The standard deviation of your large data set is:#{ variance.round(3)} (Answer is rounded to nearest thousandth place)"

Answer 1

I think the problem comes from splitting the data using \r\n which is OS dependant: if you're on Linux, it should be contents.split('\n'). Either way, you probably would be better off using IO#each to iterate over every line in the file and let Ruby deal with line ending characters.

data = File.open("test.txt", "r+")

count = 0
sum = 0
variance = 0

data.each do |line|
  value = line.split(',')[1]
  sum = sum  + value.to_f
  count += 1
end

avg = sum / count
puts "The average of your large data set is: #{ avg.round(3)} (Answer is rounded to nearest thousandth place)"

# We need to get back to the top of the file
data.rewind

data.each do |line|
  value = line.split(',')[1]
  variance = variance + (value.to_f - avg)**2
end

variance = variance / count
variance = Math.sqrt(variance)
puts "The standard deviation of your large data set is: #{ variance.round(3)} (Answer is rounded to nearest thousandth place)"