Couchbase: is there a base64-encoding overhead on storing binary data?

Question

It's known (see this answer here), that Couchbase provides binary data as base64-encoded document, when using MapReduce queries. However, does it stores it as base64 too? From libcouchbase's perspective, it takes a byte array + length, does it gets converted to base64 later?

Answer 1

The Couchbase storage engine stores your data exactly as-as (i.e. the stream of bytes of the length you specify) internally. When reading that data using the CRUD key/value API at the protocol level you get back the exact same stream of bytes.

This is possible because the low-level key-value protocol is binary on the wire, and so there are no issues with using all 8 bits per byte.

Different Client SDKs will expose that in different ways to you. For example:

• The C SDK (being low-level) directly gives you back a char* buffer and length.
• The Python SDK provides a Transcoding feature where is uses a flag in the documents' metadata to encode the type of the document, so it can automatically convert it to the original type, for example a Python serialised object or a JSON object.

On the other hand, the Views API is done over HTTP with JSON response objects. JSON cannot directly encode 8-bit binary data, so Couchbase needs to use base64 encoding for the view response objects if they contain binary data.

(As an aside, this is one of the reasons why it is recommended to have an index emit the minimum amount of data needed, for example just the key of the document(s) of interest, then use the CRUD key/value interface to actually get the document - the Key/Value interface doesn't have the base64 overhead when transmitting data back.