I was also very interested by tessellation, and specifically subdivision surfaces some time ago. This is a very complicated topic. It was researched from early 70th and it's still in research.
It isn't clear if you want to re-implement whole shader tessellation pipeline (which I think will take years for a single programmer) or just a single subdivision algorithm (or even algorith that isn't subdivision).
Anyway, there is some links about subdivision:
Theory
Typically we implement subdivision surfaces in tessellation shaders using Catmull–Clark subdivision surface algorithm. You can find some papers from original authors in Google. There is main one:
"Recursively generated B-spline surfaces on arbitrary topological meshes" (year 1978, PDF)
Closer to code
Check papers of those cool guys from Microsoft Research:
- Charles Loop - (BTW, author of another subdiv algorithm), purely mathematical stuff
- Hugues Hoppe - Look for "progressive meshes", much more close to software
Even more closer to code.
You can find some libraries on the web. When I searched a while ago, there was a dozen of libs, that implemented subdivision on CPU. I didn't look they much, because I was interested in GPU implementation. The keyword for search is "subdivision lib" =)
The most interesting one is Pixar's OpenSubdiv. Take a look at their code.
Also look at "NVIDIA Instanced Tessellation" Sample (DirectX, OpenGL). They've implemented tessellation pipelne in Vertex and Geometry shaders.
Hope it helps!