Lucene.NET and storing data on Azure Blob Storage

Question

The question I am asking is specifically because I don't want to use AzureDirectory project. I am just trying something on my own.

cloudStorageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=http;AccountName=xxxx;AccountKey=xxxxx");

        blobClient=cloudStorageAccount.CreateCloudBlobClient();


        List<CloudBlobContainer> containerList = new List<CloudBlobContainer>();
        IEnumerable<CloudBlobContainer> containers = blobClient.ListContainers();
        if (containers != null)
        {
        foreach (var item in containers)
        {
            Console.WriteLine(item.Uri);
        }
        }
        /* Used to test connectivity 
        */
        //state the file location of the index

        string indexLocation = containers.Last().Name.ToString();
        Lucene.Net.Store.Directory dir =
            Lucene.Net.Store.FSDirectory.Open(indexLocation);

        //create an analyzer to process the text
        Lucene.Net.Analysis.Analyzer analyzer = new
        Lucene.Net.Analysis.Standard.StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);

        //create the index writer with the directory and analyzer defined.

        bool findexExists = Lucene.Net.Index.IndexReader.IndexExists(dir);

        Lucene.Net.Index.IndexWriter indexWritr = new Lucene.Net.Index.IndexWriter(dir, analyzer,!findexExists, Lucene.Net.Index.IndexWriter.MaxFieldLength.UNLIMITED);
        //create a document, add in a single field
        Lucene.Net.Documents.Document doc = new  Lucene.Net.Documents.Document();
        string path="D:\\try.html";
        TextReader reader = new FilterReader("D:\\try.html");
        doc.Add(new Lucene.Net.Documents.Field("url",path,Lucene.Net.Documents.Field.Store.YES,Lucene.Net.Documents.Field.Index.NOT_ANALYZED));
        doc.Add(new Lucene.Net.Documents.Field("content",reader.ReadToEnd().ToString(),Lucene.Net.Documents.Field.Store.YES,Lucene.Net.Documents.Field.Index.ANALYZED));
        indexWritr.AddDocument(doc);
        indexWritr.Optimize();
        indexWritr.Commit();
        indexWritr.Close();

Now the issue is after indexing is completed I am not able to see any files created inside the container. Can anybody help me out?

Answer 1

You're using the FSDirectory there, which is going to write files to the local disk.

You're passing it a list of containers in blob storage. Blob storage is a service made available over a REST API, and is not addressable directly from the file system. Therefore the FSDirectory is not going to be able to write your index to storage.

Your options are :

1. Mount a VHD disk on the machine, and store the VHD in blob storage. There are some instructions on how to do this here: http://blogs.msdn.com/b/avkashchauhan/archive/2011/04/15/mount-a-page-blob-vhd-in-any-windows-azure-vm-outside-any-web-worker-or-vm-role.aspx
2. Use the Azure Directory, which you refer to in your question. I have rebuilt the AzureDirectory against the latest storage SDK: https://github.com/richorama/AzureDirectory

Answer 2

Another alternative for people looking around - I wrote up a directory that uses the azure shared cache (preview) which can be an alternative for AzureDirectory (albeit for bounded search sets)

https://github.com/ajorkowski/AzureDataCacheDirectory