Just use the other side of the connection as the where clause
var contacts = from ct in xrm.ContactSet
join cn in xrm.ConnectionSet on ct.Id equals cn.Record2Id.Id
where cn.Record1Id.Id == account.Id
select ct;
This will actually get both sides of any connection, where the Account and Contact are related any which way round