GobEncoder for Passing Anonymous Function via RPC

Question

I'm trying to build a system that will execute a function on multiple machines, passing the function anonymously via RPC to each worker machine (a la MapReduce) to execute on some subset of data. Gob doesn't support encoding functions, though the docs for GobEncoder say that "A type that implements GobEncoder and GobDecoder has complete control over the representation of its data and may therefore contain things such as private fields, channels, and functions, which are not usually transmissible in gob streams" so it seems possible.

Any examples of how this might work? I don't know much about how this encoding/decoding should be done with Gob.

Answer 1

IMHO this won't work. While it is true that if your type implements Gob{En,De}coder it can (de)serialize unexported fields of structs it is still impossible to (de)serialize code: Go is statically compiled and linked without runtime code generation capabilities (which would circumvent compile time type safety).

Short: You cannot serialize functions, only data. Your workers must provide the functions you wan't to execute. Take a look at encoding/rpc.

Answer 2

You may want to try GoCircuit, which provides a framework that basically lets you do this:

http://www.gocircuit.org/

It works by copying your binary to the remote machine(s), starting it, then doing an RPC that effectively says "execute function X with args A, B, ..."