How do NLP tokenizers handle hashtags?

datascience.stackexchange https://datascience.stackexchange.com/questions/29162

  •  31-10-2019
  •  | 
  •  

Question

I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?

No correct solution

Licensed under: CC-BY-SA with attribution
Not affiliated with datascience.stackexchange
scroll top