How to Evaluate and Analyse Machine Learning algorithm Performance?

Question

Sorry if my question sounds too naive... i am really new to machine learning and regression i have recently joined a machine learning lab as a master student . my professor wants me to write "the experiments an analysis" section of a paper the lab is about to submit about a regression algorithm that they have developed. the problem is i don't know what i have to do he said the algorithm is stable and completed and they have written the first part of paper and i need to write the evaluation part . i really don't know what to do . i have participated in coding the algorithm and i understand it pretty well but i don't know what are the tasks i must take in order to evaluate and analysis its performance. -where do i get data? -what is the testing process? -what are the analysis to be done? i am new to research and paper writing and really don't know what to do. i have read a lot of paper recently but i have no experience in analyzing ML algorithms. could you please guide me and explain (newbie level) the process please. detailed answers are appreciated
thanks

Answer 1

• You will need a test dataset to evaluate the performance. If you don't have that, divide your training dataset (that you're currently running this algorithm on) into training set and cross validation set (non overlapping).

  Create the test set by stripping out the predictions (y values) from the cross validation set.

  Run the algorithm with the training dataset to train the model.

  Once your model is trained, test it's performance using the stripped off 'test set'.

  To evaluate the performance, you can use the RMSE (Root Mean Squared Error) metric. You will need to use the predictions that your algorithm made for each sample in the test set and their corresponding actual predictions (that you stripped off earlier to feed in the test set). You can find more information here.

Answer 2

Machine learning model evaluation

Take a look at this paper. It has been written for people without a computer science background, so it should be fairly easy to follow. It covers:

• model evaluation workflow

• holdout validation

• cross-validation

  • k-fold cross-validation
  • stratified k-fold cross-validation
  • leave-one-out cross-validation
  • leave-p-out cross-validation
  • leave-one-group-out cross-validation

• nested cross-validation