Scikit learn linear regression - learning rate and epoch adjustment

Question

I am trying to learn linear regression using ordinary least squares and gradient descent from scratch.

I read the documentation for the Scikit learn function and I do not see a means to adjust the learning rate or the epoch with the sklearn.linear_model.LinearRegression class.

Is there a standard learning rate for the linear regression model?

Epochs I am assuming are determined by the change in the error function and are thus dependent on the dataset, with some predetermined maximum amount before convergence.

Are there any other good packages that have a linear regression model that I can test my own model against with adjustable learning rates and epochs?

Answer 1

A linear regression model $y=\beta X+u$ can be solved in one "round" by using $(X'X)^{-1}X'y=\hat{\beta}$. It can also be solved using gradient descent but there is no need to adjust something like a learning rate or the number of epochs since the solver (usually) converges without much trouble.

Here is a minimal example in R:

x0 <- c(1,1,1,1,1) 
x1 <- c(1,2,3,4,5)
x2 <- c(8,4,3,1,8)
x <- as.matrix(cbind(x0,x1,x2))
y <- as.matrix(c(3,7,5,11,14))

x
y

# (X'X)^-1 X'y
beta1 = solve(t(x)%*%x) %*% t(x)%*%y 

# R's regression command
beta2 = summary(lm(y ~ x[, 2:3]))

# Gradient decent
m <- nrow(y)
grad <- function(x, y, theta) {
  gradient <- (1/m)* (t(x) %*% ((x %*% t(theta)) - y))
  return(t(gradient))
}

# define gradient descent update algorithm
grad.descent <- function(x, maxit){
  theta <- matrix(c(0, 0, 0), nrow=1) # Initialize the parameters
  
  alpha = .05 # set learning rate
  for (i in 1:maxit) {
    theta <- theta - alpha  * grad(x, y, theta)   
  }
  return(theta)
}

# results without feature scaling
print(grad.descent(x,2000))
beta1
beta2

Answer 2

You can use SGDRegressor available in scikit learn for adjusting learning rate. It has a variety of parameters you can adjust.