If your base dataset is temp and date, then this avoids manipulating the original data frame:
ggplot(df) +
geom_point(aes(x=strftime(date,format="%m-%d"),
y=temp,
color=strftime(date,format="%Y")), size=3)+
scale_color_discrete(name="Year")+
labs(x="date")
EDIT (Response to OP's comment).
So this combines the approach above with Henrik's, using dates instead of char for the x-axis, and avoiding modification of the original df.
library(ggplot2)
ggplot(df) +
geom_point(aes(x=as.Date(paste(2014,strftime(date,format="%m-%d"),sep="-")),
y=temp,
color=strftime(date,format="%Y")), size=3)+
scale_color_discrete(name="Year")+
labs(x="date")