Parallel Processing in R using "parallel" package

Question

I have two data frames:

> head(k)
          V1
1 1814338070
2 1199215279
3 1283239083
4 1201972527
5  404900682
6 3093614019

> head(g)
  start    end      state      value
1 16777216 16777471 queensland 15169
2 16777472 16778239     fujian     0
3 16778240 16779263   victoria 56203
4 16779264 16781311  guangdong     0
5 16781312 16781823      tokyo     0
6 16781824 16782335      aichi     0

> dim(k)
[1] 624979      1
> dim(g)
[1] 5510305       4

I want to compare each value in data.frame(k) and match if it fits between the range of start and end of data.frame(g) and if it does return the value of state and value from data.frame(g)

The problem I have is due to the dimensions of both the data frame and to do the match and return my desired values it takes 5 hours on my computer. I've used the following method but I'm unable to make use of all cores on my computer and not even make it work correctly:

return_first_match_position <- function(int, start,end) {
  match = which(int >= start & int <= end)
  if(length(match) > 0){
    return(match[1])
  }
  else {
    return(match)
  }
}

library(parallel)
cl = makeCluster(detectCores())
matches = Vectorize(return_first_match_position, 'int')(k$V1,g$start, g$end)
p = parSapply(cl, Vectorize(return_first_match_position, 'int')(k$V1,g$start, g$end), return_first_match_position)
stopCluster(cl)

desired output is % number of times state and value show up for every match of the number from data.frame(k) in data.frame(g)

Was wondering there there is an intelligent way of doing parallel processing in R ? And can anyone please suggest (any sources) how I can learn/improve writing functions in R?

Answer 1

so instead of editing the last one a lot (pretty much making a new one).. is this what you want: I noticed that your end is always 1 before the next rows start, so what you want ( i think) is to just find out how many were within each interval and give that interval the state,value for that range. so

set.seed(123)
c1=seq(1,25,4)
c2=seq(4,30,4)
c3=letters[1:7]
c4=sample(seq(1,7),7)
c.all=cbind(c1,c2,c3,c4)

> c.all  ### example data.frame that looks similar to yours
     c1   c2   c3  c4 
[1,] "1"  "4"  "a" "3"
[2,] "5"  "8"  "b" "7"
[3,] "9"  "12" "c" "2"
[4,] "13" "16" "d" "1"
[5,] "17" "20" "e" "6"
[6,] "21" "24" "f" "5"
[7,] "25" "28" "g" "4"

k1 <- sample(seq(1,18),20,replace=T)

k1
 [1]  2  1 15 14  4 15  3 17 18  1  4  3 16 15  2  4  8 11  7 16

fallsin <- cut(k1,  c(as.numeric(c.all[,1]), max(c.all[,2])), labels=paste(c.all[,3],  c.all[,4],sep=':'), right=F)

fallsin
[1] a:3 a:3 e:6 e:6 a:3 e:6 a:3 f:5 f:5 a:3 a:3 a:3 e:6 e:6 a:3 a:3 c:2 d:1 b:7 e:6
Levels: a:3 b:7 c:2 d:1 e:6 f:5 g:4
prop.table(table(fallsin))

 a:3  b:7  c:2  d:1  e:6  f:5  g:4 
0.45 0.05 0.05 0.05 0.30 0.10 0.00 

where the names of the columns are the 'state:value' and the numbers are the percent of k1 that fall within the range of that label

Answer 2

I think you want to do a rolling join. This can be done very efficiently with data.table:

DF1 <- data.frame(V1=c(1.5, 2, 0.3, 1.7, 0.5))
DF2 <- data.frame(start=0:3, end=0.9:3.9, 
                  state=c("queensland", "fujian", "victoria", "guangdong"),
                  value=1:4)

library(data.table)
DT1 <- data.table(DF1, key="V1")
DT1[, pos:=V1]
#    V1 pos
#1: 0.3 0.3
#2: 0.5 0.5
#3: 1.5 1.5
#4: 1.7 1.7
#5: 2.0 2.0
DT2 <- data.table(DF2, key="start")
#   start end      state value
#1:     0 0.9 queensland     1
#2:     1 1.9     fujian     2
#3:     2 2.9   victoria     3
#4:     3 3.9  guangdong     4

DT2[DT1, roll=TRUE]
#   start end      state value pos
#1:     0 0.9 queensland     1 0.3
#2:     0 0.9 queensland     1 0.5
#3:     1 1.9     fujian     2 1.5
#4:     1 1.9     fujian     2 1.7
#5:     2 2.9   victoria     3 2.0