From your description it sounds like users belong to roles, and then the combination of userRoles belongs to a conference. So, without the fancy diagram, your entities would be something like this...
Users
Id
Roles
Id
Conferences
Id
UserRoles
UserId
RoleId
ConferenceUserRoles
ConferenceId
UserId
RoleId
You may need to add a "UserRoleId" to UserRoles and use that in ConferenceUserRoles. I'm not exactly sure how EF will handle the three-way relation table.
Hope this helps!