Do we really need `<unk>` tokens?

datascience.stackexchange https://datascience.stackexchange.com/questions/33444

Question

I am wondering, do we really need <unk> tokens? Why do we limit our vocabulary?

Is it for speed? Accuracy?

If we disable all limitations, what do you predict happens?

No correct solution

Licensed under: CC-BY-SA with attribution
Not affiliated with datascience.stackexchange
scroll top