Scaling label encoded values for Linear Algorithms

Question

I have encoded categorical variables to numerical values. As we know that for feeding values to Linear Algorithms like SVM or KNN, we scale the values for columns having large variations.

I have three label encoded columns, one of which has unique values from 1-3, another has 1-150 and another has 0,1,-1 where -1 represents missing values. How will the MinMaxScaler affect here? Or it is not needed to scale these columns? If not, how can I avoid these specific columns? Generally we scale the whole dataframe.

P.S. I'm using LinearSVC algorithm.