Publish files in open splice DDS

Question

I have successfully created a publisher and subscriber with primitive types. Now I have a problem when trying to publish a file (like xml, txt, video, audio etc) to DDS. I don't know how to specify the types in IDL file and how to publish the whole file to DDS (also how to receive the file on the subscriber side).

And one more question: "Are there any memory limitations to publishing in DDS?"

Answer 1

There are several routes you can go for distributing a file using DDS. The most straightforward (but not necessarily the best) is using a sequence of octets (bytes) to capture the contents of the file. This can be achieved by an IDL definition like this:

const long MAX_NAME_LEN = 128;
const long MAX_FILE_SIZE = 100000;

typedef string<MAX_NAME_LEN> NameType;
typedef sequence<octet,MAX_FILE_SIZE> BinaryContentsType;

struct BinaryFile {
    NameType name; //@key
    BinaryContentsType contents;
};
#pragma keylist BinaryFile name

Since your question is not specific for OpenSplice DDS, but could be applied to any DDS implementation, I have included syntax for defining the key attribute such that it works with several implementations.

In your application, you will have to instantiate the type and fill the values of the members. This will look something like

BinaryFile instance = new BinaryFile();
instance.name = "SomeFileName";
// fill instance.contents by reading file into array of bytes.

The contents attribute will be an array of bytes. After the contents have been filled, just invoke the write method on a BinaryFileDataWriter, similar to what other examples do.

There is no real limit to the size of the file distributed this way other than the limits of the shared memory as configured. However, it is good practice to impose a limit on the size, which is why the sequence in the BinaryContentsType type is bounded.

You have not mentioned what programming language you are using, so it is hard to give any coding details. But since you mentioned that you did get started with basic types, it should be easy to figure out how to publish the BinaryFileType type, especially if you take a look at the provided documentation and examples and apply that to your own type.

As a note on the side, are you sure that you want to distribute complete files? Depending on the situation, a better approach might be to analyse the structure of the contents of the file and create a data-model that matches that. You would be reading the file on the publisher side and translate it into meaningful data-items as opposed to a blob. That way, subscribers can take advantage of more advanced data management features, like subscribing to a subset of the contents by subscribing to a subset of the available Topics only, or by using a filter based on content. Whether this makes sense all depends on your use case though.

Answer 2

But the real question is -- why? Transporting opaque data via DCPS is one thing, but entire files? Aren't there several protocols much better suited to this? Do you keep lots of files in memory such that you consider file transfers akin to IPC? My answer to you is to seek out FTP, NFS, SSHFS, SMB, or HTTP. All of these can peacefully coexist with your DDS traffic, and you would likely see no net benefit to using DDS for the file transfer.

Reinier's answer is correct, per the question. (BTW you should really consider accepting some of the answers -- you have two outstanding! You might not get anyone on SO to answer your next question).

Answer 3

One benefit that DDS can bring to sending files is for the case where you want to send a file to multiple recipients. Through the use of a reliable protocol built on top of UDP with Multicast enabled, you can then send the file to several recipients at the same time without incurring the bandwidth usage to perform multiple file transfers.