How do I create a DTM-like text matrix from a list of text blocks?

Question

I have been using the textmatrix() function for a while to create DTMs which I can further use for LSI.

dirLSA<-function(dir){
  dtm<-textmatrix(dir)
  return(lsa(dtm))
}

textdir<-"C:/RProjects/docs"
dirLSA(textdir)

> tm
$matrix
                  D1 D2 D3 D4 D5 D6 D7 D8 D9
1. 000             2  0  0  0  0  0  0  0  0
2. 20              1  0  0  1  0  0  1  0  0
3. 200             1  0  0  0  0  0  0  0  0
4. 2014            1  0  0  0  0  0  0  0  0
5. 2015            1  0  0  0  0  0  0  0  0
6. 27              1  0  0  0  0  0  0  1  0
7. 30              1  0  0  0  1  0  1  0  0
8. 31              1  0  2  0  0  0  0  0  0
9. 40              1  0  0  0  0  0  0  0  0
10. 45             1  0  0  0  0  0  0  0  0
11. 500            1  0  0  0  0  0  1  0  0
12. 600            1  0  0  0  0  0  0  0  0
728. bias          0  0  0  2  0  0  0  0  0
729. biased        0  0  0  1  0  0  0  0  0
730. called        0  0  0  1  0  0  0  0  0
731. calm          0  0  0  1  0  0  0  0  0
732. cause         0  0  0  1  0  0  0  0  0
733. chauhan       0  0  0  2  0  0  0  0  0
734. chief         0  0  0  8  0  0  1  0  0

Textmatrix() is a function which takes a directory(folder path) and returns a document-wise term frequency. This is used in further analysis like Latent Semantic Indexing/Allocation(LSI/LSA)

However, a new problem that came across me is that if I have tweet data in batch files (~500000 tweets/batch) and I want to carry out similar operations on this data.

I have code modules to clean up my data, and I want to pass the cleaned tweets directly to the LSI function. The problem I face is that the textmatrix() does not support it.

I tried looking at other packages and code snippets, but that didn't get me any further. Is there any way I can create a line-term matrix of sorts?

I tried sending table(tokenize(cleanline[i])) into a loop, but it wont add new columns for words not already there in the matrix. Any workaround?

Update: I just tried this:

a<-table(tokenize(cleanline[10]))
b<-table(tokenize(cleanline[12]))

df1<-data.frame(a)
df1
df2<-data.frame(b)
df2

merge(df1,df2, all=TRUE)

I got this:

> df1
    Var1 Freq
1           6
2      "    2
3    and    1
4   home    1
5   mabe    1
6 School    1
7   then    1
8   xbox    1
> b<-table(tokenize(cleanline[12]))
> df2<-data.frame(b)
> df2
        Var1 Freq
1              13
2          "    2
3  BillGates    1
4       Come    1
5       help    1
6        Mac    1
7       make    1
8  Microsoft    1
9     please    1
10   Project    1
11    really    1
12   version    1
13      wish    1
14     would    1
> merge(df1,df2)
  Var1 Freq
1    "    2
> merge(df1,df2, all=TRUE)
        Var1 Freq
1               6
2              13
3          "    2
4        and    1
5       home    1
6       mabe    1
7     School    1
8       then    1
9       xbox    1
10 BillGates    1
11      Come    1
12      help    1
13       Mac    1
14      make    1
15 Microsoft    1
16    please    1
17   Project    1
18    really    1
19   version    1
20      wish    1
21     would    1

I think I'm close.

Answer 1

Something that works for me:

textLSA<-function(text){

  a<-data.frame(table(tokenize(text[1])))
  colnames(a)[2]<-paste(c("Line",1),collapse=' ')
  df<-a

  for(i in 1:length(text)){
    a<-data.frame(table(tokenize(text[i])))
    colnames(a)[2]<-paste(c("Line",i),collapse=' ')
    df<-merge(df,a, all=TRUE)
  }

  df[is.na(df)]<-0
  dtm<-as.matrix(df[,-1])
  rownames(dtm)<-df$Var1

  return(lsa(dtm))
}

What do you think of this code?

Answer 2

Try something like this

  ll <- list(df1,df2)
  dtm <- xtabs(Freq ~ ., data = do.call("rbind", ll))