Skip to content

Optimal coefficients, fixes #8847 #8885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Optimal coefficients, fixes #8847 #8885

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
51c8190
Fixes #8847
liakdimi1 Jul 22, 2023
4df600e
Fixes coefficients
liakdimi1 Jul 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions machine_learning/linear_regression.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ def run_steep_gradient_descent(data_x, data_y, len_data, alpha, theta):
curr_features - alpha_ * gradient(w.r.t. feature)
"""
n = len_data

prod = np.dot(theta, data_x.transpose())
prod -= data_y.transpose()
sum_grad = np.dot(prod, data_x)
Expand Down Expand Up @@ -70,8 +69,8 @@ def run_linear_regression(data_x, data_y):
:param data_y : contains the output (result vector)
:return : feature for line of best fit (Feature vector)
"""
iterations = 100000
alpha = 0.0001550
iterations = 250000
alpha = 0.000326
Comment on lines +72 to +73
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't give the optimal coefficients. Like I said in my original issue, gradient descent is an approximation algorithm by definition, so tweaking the parameters will only give you a better approximation. My point with the original issue is that we should add a different algorithm, one that is a direct method rather than an iterative method.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also doesn't fix the fact that the code is still calculating SSE incorrectly.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you elaborate on what do you mean by direct method?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A direct method gives an exact solution. By contrast, an iterative method (such as gradient descent) gives increasingly accurate approximations. With an iterative method, you get closer and closer to the exact solution but you may never actually reach that exact solution.

For example, let's say you want to find the roots of a quadratic polynomial $ax^2 + bx + c$. You could use Newton's method to approximate the roots, refining your solutions to be more and more accurate—this is an iterative method. Alternatively, you could just use the quadratic formula to just get the exact solutions—this is a direct method.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, explain with facts why the SSE is calculated incorrectly because the code does the predictions finds the deviation and takes the squares of the deviations, and sums the errors, which is the definition of SSE.

Copy link
Contributor

@tianyizheng02 tianyizheng02 Jul 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already explained this with my code sample in the original issue, but sure, let's go over the math for the SSE function:

Python/machine_learning/linear_regression.py

Lines 52 to 64 in 9e08c77

def sum_of_square_error(data_x, data_y, len_data, theta):
"""Return sum of square error for error calculation
:param data_x : contains our dataset
:param data_y : contains the output (result vector)
:param len_data : len of the dataset
:param theta : contains the feature vector
:return : sum of square error computed from given feature's
"""
prod = np.dot(theta, data_x.transpose())
prod -= data_y.transpose()
sum_elem = np.sum(np.square(prod))
error = sum_elem / (2 * len_data)
return error

Likes 60–62 are fine because they calculate the SSE exactly as we expect: it calculates the differences between the predicted response and actual response values, squares them, and then sums them:
$$\mathrm{SSE} = \sum_{i = 0}^{N} (\mathbf{x}_i^{\top} \boldsymbol{\theta} - \mathbf{y}_i)^2$$

However, for some reason, the function then divides the SSE by 2 * len_data. Dividing the SSE (sum of squared errors) by the number of datapoints gives you the MSE (mean squared error):
$$\mathrm{MSE} = \frac{1}{N} \sum_{i = 0}^{N} (\mathbf{x}_i^{\top} \boldsymbol{\theta} - \mathbf{y}_i)^2$$

Naturally, it follows that the "SSE" function is actually calculating half of the MSE because it's dividing the SSE by $2N$.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i appreciate your response


no_features = data_x.shape[1]
len_data = data_x.shape[0] - 1
Expand Down