Question about BERT embeddings with high cosine similarity

Question

Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?

Answer 1

It's really hard to say if the BERT assigns similar embeddings to the same words in the similar context. Most likely it will not. This is because, the embeddings, is not just function of context (other words in the sentence), but the position as well. Hence, even if the words occurs in similar context, with different positions, they can have different embeddings.