Conduit and creating conduit based libraries

Question

I know that Conduit is a solution to the streaming data problem for handling data in constant memory.

I see lot of libraries in hackage with conduit name attached to it. Some examples being cvs-conduit, zlib-conduit, attoparsec-conduit, etc.

So, how to make some ordinary library conduit based ? Just by using the main conduit package, does it qualify of giving the solution to streaming data problem ?

I tried to dig into the source of csv-conduit and it was way too complex. (Well way bigger (much bigger) than the complete csv parser implemented in RWH book.) Also this github library actually uses Conduit, so does that also handle streaming data in constant memory ?

Answer 1

Yes, main conduit package is pretty much everything you would need to start.

The idea is that you organize your workflow in Conduits. Take some elements in - push some elements out. On the basic level you can go pretty far just by using await and yield. Just use them in a monad, like this:

idConduit =
  do mElt <- await
     case mElt of
       Nothing -> return ()
       Just element ->
         do yield element
            idConduit

Then you put them between Sink and Source like mySink $= myConduit1 $= myConduit2 $$ mySource and you are done.

If you want to read files with them, try attoparsec-conduit as well.