ggplot2 Color Scale Over Affected by Outliers

Question

I'm having difficulty with a few outliers making the color scale useless.

My data has a Length variable that is based in a range, but will usually have a few much larger values. The below example data has 95 values between 500 and 1500, and 5 values over 50,000. The resulting color legends tend to use 10k, 20k, ... 70k for the color changes when I want to see color changes between 500 and 1500. Really, anything over around 1300 should be the same solid color (probably median +/- mad), but I don't know where to define that.

I'm open to any ggplot solution, but ideally lower values would be red, middle white, and higher blue (low is bad). In my own dataset, date is an actual date with as.POSIXct() in the ggplot aes(), but doesn't seem to affect the example.

#example data
date <- sample(x=1:10,size=100,replace=T)
stateabbr <- sample(x=1:50,size=100,replace=T)
Length <- c(sample(x=500:1500,size=95,replace=T),60000,55000,70000,50000,65000)
x <- data.frame(date=date,stateabbr=stateabbr,Length=Length)

#main plot
(g <- ggplot(data=x,aes(x=date,y=factor(stateabbr))) +
  geom_point(aes(color=as.numeric(as.character(Length))),alpha=3/4,size=4) + 
  #scale_x_datetime(labels=date_format("%m/%d")) + 
  opts(title="Date and State") + xlab("Date") + ylab("State"))

#problem
g + scale_color_gradient2("Length",midpoint=median(x$Length))

Adding trans="log" or "sqrt" doesn't quite do the trick either.

Thank you for your help!

Answer 1

Here's one slightly tricky options:

#Create a new variable indicating the unusual values
x$Length1 <- "> 1500"
x$Length1[x$Length <= 1500] <- NA

#main plot
# Using fill - tricky!
g <- ggplot() +
  geom_point(data = subset(x,Length <= 1500),
             aes(x=date,y=factor(stateabbr),color=Length),size=4) + 
  geom_point(data = subset(x,Length > 1500),
             aes(x=date,y=factor(stateabbr),fill=Length1),size=4)+
  opts(title="Date and State") + xlab("Date") + ylab("State")

#problem
g + scale_color_gradient2("Length",midpoint=median(x$Length))

So the tricky part here is using fill on points, in order to convince ggplot to make another legend. You can obviously customize this with different labels and colors for the fill scale.

One more thing, reading Brandon's answer. You could in principle combine both approaches by taking the outlying values, using cut to create a separate categorical variable for them, and then use my trick with the fill scale. That way you could indicate multiple outlying groups of points.

Answer 2

From my comment, see ?cut

x$colors <- cut(x$Length, breaks=c(0,500,1000,1300,max(x$Length)))

g <- ggplot(data=x,aes(x=date,y=factor(stateabbr),color=colors)) +
    geom_point() + 
    opts(title="Date and State") + 
    xlab("Date") + 
    ylab("State")

Answer 3

Get rid of the outliers. Quick and dirty, I know, but I think it was worth saying. You can always describe them in your text. Why let them ruin your analyses and graphs?

There's a paper referenced in this blog post which deals with ethically removing outliers:

http://psuc2f.wordpress.com/2011/10/14/is-it-dishonest-or-unethical-to-remove-outliers/

Another simple way of dealing with them would be to cap them:

df$Value[df$Value>1300]=1300

Again, you can describe that you did this in the text or even just edit the scale to say 1300+ instead of 1300