Python can't take input while using functions
-
02-11-2019 - |
Question
While building my model, I figured that it makes sense to reuse some of the code that I've been using for the train dataset, on the test set as well so I took the code performing mutual operations into one function definition. In this function, I am handling missing values and using its return to perform one-hot-encoding and using it on Random Forest Regression. However, its throwing the following error:
Traceback (most recent call last):
File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 56, in <module>
sel.fit(x_train, y_train)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\feature_selection\from_model.py", line 196, in fit
self.estimator_.fit(X, y, **fit_params)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\forest.py", line 249, in fit
X = check_array(X, accept_sparse="csc", dtype=DTYPE)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 542, in check_array
allow_nan=force_all_finite == 'allow-nan')
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 56, in _assert_all_finite
raise ValueError(msg_err.format(type_err, X.dtype))
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
I did not have this problem while using the same code without organizing it into a function. def feature_selection_and_engineering(df)
is the function in question. The following is my entire code.
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")
def feature_selection_and_engineering(df):
# Creating a series of how many NaN's are in each column
nan_counts = df.isna().sum()
# Creating a template list
nan_columns = []
# Iterating over the series and if the value is more than 0 (i.e there are some NaN's present)
for i in range(0, len(nan_counts)):
if nan_counts[i] > 0:
nan_columns.append(df.columns[i])
# Iterating through all the columns which are known to have NaN's
for i in nan_columns:
if df[nan_columns][i].dtypes == 'float64':
df[i] = df[i].fillna(df[i].mean())
elif df[nan_columns][i].dtypes == 'object':
df[i] = df[i].fillna('XX')
# Creating a template list
categorical_columns = []
# Iterating across all the columns,
# checking if they're of the object datatype and if they are, appending them to the categorical list
for i in range(0, len(df.dtypes)):
if df.dtypes[i] == 'object':
categorical_columns.append(df.columns[i])
return categorical_columns
# take one-hot encoding
OHE_sdf = pd.get_dummies(feature_selection_and_engineering(train))
# drop the old categorical column from original df
train.drop(columns = feature_selection_and_engineering(train), axis = 1, inplace = True)
# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)
# Dividing the training dataset into train/test sets with the test size being 20% of the overall dataset.
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.2, random_state = 42)
randomForestRegressor = RandomForestRegressor(n_estimators=1000)
# Invoking the Random Forest Classifier with a 1.25x the mean threshold to select correlating features
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)
selected = sel.get_support()
# linearRegression.fit(x_train, y_train)
randomForestRegressor.fit(x_train, y_train)
# Assigning the accuracy of the model to the variable "accuracy"
accuracy = randomForestRegressor.score(x_train, y_train)
# Predicting for the data in the test set
predictions = randomForestRegressor.predict(feature_selection_and_engineering(test))
# Writing the predictions to a new CSV file
submission = pd.DataFrame({'Id': test['PassengerId'], 'SalePrice': predictions})
filename = 'Boston-Submission.csv'
submission.to_csv(filename, index=False)
print(accuracy*100, "%")
No correct solution
Licensed under: CC-BY-SA with attribution
Not affiliated with datascience.stackexchange