I would try something like this:
group <- list(strftime(df$datetime, format = "%H"))
to convert the datetime strings into a list of hours to aggregate by.
I'd then aggregate the column of values whose means you want:
means <- aggregate(df$values, by = group, FUN = mean)
I don't know whether this a particularly good method but it works for me.