You probably need to implement arbitrary transformations on primitives using homogenous matrices, rather than support arbitrary orientation for each primitive.
For example, it's not uncommon for ray tracers to only support cones that have their base on the origin, and that point along the vertical axis. You would then use affine transformations to move the cone to the right place and orientation.
My own ray tracer (which thus far only supports planes, boxes and spheres) has the same problem, and implementation transformation matrices is my next task.