This looks pretty familiar, and is called "surface acne". A common error with ray-tracing shadows is that the shadow ray intersects the object the shadow ray is emanating from. Because of finite floating-point precision, the intersection point isn't always exactly on a surface - it ends up being slightly above or below. When you cast your shadow ray, you can intersect that surface again. It looks noisy because some locations do self-intersect and some don't
Typical ways to handle this are to move the origin of your shadow ray slightly back toward the eye point, or to ignore shadow intersections that occur within a very short distance, or to ignore all intersections with that surface (if your surface is planar or convex).
As for the color of shadowed areas, you'll likely have to post actual code but it looks like you're using the material of the object the shadow ray intersects instead of the primary ray.