NLTK - how to find out what corpora are installed from within python?

Question

I'm trying to load some corpora I installed with the NLTK installer but I got a:

>>> from nltk.corpus import machado
      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      ImportError: cannot import name machado

But in the download manager (nltk.download()) the package machado is marked as installed and I have a nltk_data/corpus/machado folder.

How can I see from inside the python intepreter what are the installed corpora?

Also, what package should I install to work with this how-to? http://nltk.googlecode.com/svn/trunk/doc/howto/portuguese_en.html

I can't find the module nltk.examples refered to in the how-to.

Answer 1

try

import nltk.corpus
dir(nltk.corpus)

at which point, it probably told you something about __LazyModule__... so do dir(nltk.corpus) again.

If that doesn't work, try tab-completion in iPython.

Answer 2

The NLTK includes a package, nltk.corpus, that contains definitions of corpus readers (such as PlainTextCorpusReader). This package also includes a large list of pre-defined access points for corpora that can be downloaded with nltk.downloader(). These access points (e.g., nltk.corpus.brown) are defined whether or not the corresponding corpus has been downloaded.

1. To see which access points are defined in the NLTK, use dir(nltk.corpus) (after import nltk).

2. To see which corpora you have in your nltk_data area, try this:

   import os
   import nltk
   print( os.listdir( nltk.data.find("corpora") ) )
   

   This just dumps a list with the contents of the folder nltk_data/corpora. You can take it from there.

3. If you have installed your own corpus in the nltk_data/corpora area and the NLTK doesn't know about it, you need to fire up the appropriate reader yourself. E.g., if it's a plaintext corpus in corpora/mycorpus and all the files end in .txt, you'd do it like this:

   import nltk
   from nltk.corpus import PlaintextCorpusReader
   
   mypath = nltk.data.find("corpora/mycorpus")
   mycorpus = PlaintextCorpusReader(mypath, r".*\.txt$")
   

   But in that case you could put your own corpus anywhere, and point mypath to it directly instead of asking the NLTK to find it.