Question

I'm using Apache JClouds to connect to my Openstack Swift installation. I managed to upload and download objects from Swift. However, I failed to see how to upload dynamic large object to Swift.

To upload dynamic large object, I need to upload all segments first, which I can do as usual. Then I need to upload a manifest object to combine them logically. The problem is to tell Swift this is a manifest object, I need to set a special header, which I don't know how to do that using JClouds api.

Here's a dynamic large object example from openstack official website.

The code I'm using:

public static void main(String[] args) throws IOException {
    BlobStore blobStore = ContextBuilder.newBuilder("swift").endpoint("http://localhost:8080/auth/v1.0")
            .credentials("test:test", "test").buildView(BlobStoreContext.class).getBlobStore();
    blobStore.createContainerInLocation(null, "container");

    ByteSource segment1 = ByteSource.wrap("foo".getBytes(Charsets.UTF_8));
    Blob seg1Blob = blobStore.blobBuilder("/foo/bar/1").payload(segment1).contentLength(segment1.size()).build();
    System.out.println(blobStore.putBlob("container", seg1Blob));

    ByteSource segment2 = ByteSource.wrap("bar".getBytes(Charsets.UTF_8));
    Blob seg2Blob = blobStore.blobBuilder("/foo/bar/2").payload(segment2).contentLength(segment2.size()).build();
    System.out.println(blobStore.putBlob("container", seg2Blob));

    ByteSource manifest = ByteSource.wrap("".getBytes(Charsets.UTF_8));
    // TODO: set manifest header here
    Blob manifestBlob = blobStore.blobBuilder("/foo/bar").payload(manifest).contentLength(manifest.size()).build();
    System.out.println(blobStore.putBlob("container", manifestBlob));

    Blob dloBlob = blobStore.getBlob("container", "/foo/bar");
    InputStream input = dloBlob.getPayload().openStream();
    while (true) {
        int i = input.read();
        if (i < 0) {
            break;
        }
        System.out.print((char) i); // should print "foobar"
    }
}

The "TODO" part is my problem.


Edited:

I've been pointed out that Jclouds handles large file upload automatically, which is not so useful in our case. In fact, we do not know how large the file will be or when the next segment will arrive at the time we start to upload the first segment. Our api is designed to make client able to upload their files in chunks of their own chosen size and at their own chosen time, and when done, call a 'commit' to make these chunks as a file. So this makes us want to upload the manifest on our own here.

Was it helpful?

Solution

According to @Everett Toews's answer, I've got my code correctly running:

public static void main(String[] args) throws IOException {
    CommonSwiftClient swift = ContextBuilder.newBuilder("swift").endpoint("http://localhost:8080/auth/v1.0")
            .credentials("test:test", "test").buildApi(CommonSwiftClient.class);

    SwiftObject segment1 = swift.newSwiftObject();
    segment1.getInfo().setName("foo/bar/1");
    segment1.setPayload("foo");
    swift.putObject("container", segment1);

    SwiftObject segment2 = swift.newSwiftObject();
    segment2.getInfo().setName("foo/bar/2");
    segment2.setPayload("bar");
    swift.putObject("container", segment2);

    swift.putObjectManifest("container", "foo/bar2");

    SwiftObject dlo = swift.getObject("container", "foo/bar", GetOptions.NONE);
    InputStream input = dlo.getPayload().openStream();
    while (true) {
        int i = input.read();
        if (i < 0) {
            break;
        }
        System.out.print((char) i);
    }
}

OTHER TIPS

jclouds handles writing the manifest for you. Here are a couple of examples that might help you, UploadLargeObject and largeblob.MainApp.

Try using

Map<String, String> manifestMetadata = ImmutableMap.of(
    "X-Object-Manifest", "<container>/<prefix>");
BlobBuilder.userMetadata(manifestMetadata)

If that doesn't work you might have to use the CommonSwiftClient like in CrossOriginResourceSharingContainer.java.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top