R package VLMC dies if state space size exceeds 27

Question

I am using VLMC to fit some Markov models and it dies as soon as the alphabet size reaches 28. I thought this was due to using a single letter in the alphabet by default, but it has the same behavior with "code1char = FALSE". This is true for me on real data as well as this fake example.

library(VLMC)

# works fine
ins <- sample(seq(1,27,1),50000,replace=T)
vlmc(ins, dump = 1,threshold.gen = 2, debug = TRUE)

#core dump
ins <- sample(seq(1,28,1),50000,replace=T)
vlmc(ins, dump = 1,threshold.gen = 2, debug = TRUE)

Any ideas?

The seg fault looks like this BTW. It looks to me like the alphabet after z is being mapped to NA which is causing an array bound issue.

library(VLMC)

sc <- 10
amp <- 13
x <- round(amp*sin(seq(0,2*sc*pi,0.01)))
x <- amp + x + rpois(NROW(x),1)
length(table(x))
length(x)

vlmc(x, dump = 1,threshold.gen = 2, debug = TRUE)

vlmc: Alpha = 'abcdefghijklmnopqrstuvwxyzNANANANANA' ; |X| = 31
vlmc: ctl.dump =  4 11 
vlmc: n = |data| = 6284, cutoff{prune} = 21.8865, threshold{gen} = 2
vlmc: |alphabet| = 31, alphabet = abcdefghijklmnopqrstuvwxyzNA
generating... 
*** caught segfault ***
  address 0x0, cause 'memory not mapped'

Traceback:
  1: .C("vlmc_p", data = Data, n = n, threshold.gen = as.integer(threshold.gen),     cutoff.prune = as.double(cutoff.prune), alpha.len = as.integer(alpha.len),     alpha = as.character(Alpha), debug = as.integer(as.logical(debug)),     dump.flags = as.integer(c(dump, ctl.dump)), size = integer(4),     PACKAGE = "VLMC")
2: vlmc(x, dump = 1, threshold.gen = 2, debug = TRUE)

Answer 1

As the maintainer of VLMC, I can tell you that one of the longest standing TODO entries for VLMC has been to raise the currently builtin limit of 26 for the maximal alphabet size should be raised..

Of course it it is a bug that I don't give an error message in the case of a larger alphabet, but rather pass things to C and do not check there. The next version of VLMC will not seg.fault for this anymore.

However, I'm not yet sure I'll find the time to allow a considerably larger alphabet.... Of course I'd happily accept patches ... it's free open source software.

Best regards, Martin Maechler, ETH Zurich