My feeling is that you haven't provided enough information in your question. The actual pthread mutex implementation in glibc is a very complex beast (it's meant to support many use cases and policies) and is hardly suitable for training assignments.
Therefore, you may actually be required to do something different then digging into pthread mutex internals.
One possibility: it may be enough for you to use an alternative to pthread_mutex_trylock
. Such alternative surely exists and is called pthread_mutex_timedlock
(invoke it with timeout of 0).
Another possibility: you may need to come with your own mutex implementation (based on low level primitives, such as futex
on Linux). This is not too complicated either: http://locklessinc.com/articles/mutex_cv_futex/.