Not sure if you found a solution for this. If not, try the following:
$groups = $web.SiteGroups
$clientContext.Load($groups)
$clientContext.ExecuteQuery()
$group = $groups.GetByName("Your Visitors Group Name")
$clientContext.Load($group)
$web.AssociatedVisitorGroup = $group
$web.Update()
$clientContext.ExecuteQuery()
Let me know how that works.