After more looking around I finally managed to come across it.
In the request_rec there is a variable called server which is a server_rec*. So to access the config file you would use
ap_get_module_config(r->server->module_config, &my_module)
where r is the request_rec passed into the handler.