Calculating a Least Squares Regression Line: Equation, Example, Explanation

Suppose we wanted to estimate a score for someone who had spent exactly 2.3 hours on an essay. I’m sure most of us have experience in drawing lines of best fit, where we line up a ruler, think “this seems about right”, and draw some lines from the X to the Y axis. In a room full of people, you’ll notice that no two lines of best fit turn out exactly the same. What we need to answer this question is the best best fit line. 

Through the magic of least sums regression, and with a few simple equations, we can calculate a predictive model that can let us estimate grades far more accurately than by sight alone. Regression analyses are an extremely powerful analytical tool used within economics and science. There are a number of popular statistical programs that can construct complicated regression models for a variety of needs. A simpler model such as this requires nothing more than some data, and maybe a calculator. It’s worth noting at this point that this method is intended for continuous data.  

The premise of a regression model is to examine the impact of one or more independent variables (in this case time spent writing an essay) on a dependent variable of interest (in this case essay grades). Linear regression analyses such as these are based on a simple equation:  

 

There’s a couple of key takeaways from the above equation. First of all, the intercept (a) is the essay grade we expect to get when the time spent on essays is zero. You can imagine you can jot down a few key bullet points while spending only a minute on an essay and still get a few points here and there. Every essay will have at least this score according to our model. On top of that, every hour we spent on our essays (X) leads to an increase of b in the grade the essay gets.  We can work out b through the following, slightly scary equation:

When calculating least squares regressions by hand, the first step is to find the means of the dependent and independent variables. We do this because of an interesting quirk within linear regression lines - the line will always cross the point where the two means intersect. We can think of this as an anchor point, as we know that the regression line in our test score data will always cross (4.72, 64.45).  

The second step is to calculate the difference between each value and the mean value for both the dependent and the independent variable. In this case this means we subtract 64.45 from each test score and 4.72 from each time data point.  Additionally, we want to find the product of multiplying these two differences together.

You should notice that as some scores are lower than the mean score, we end up with negative values. By squaring these differences, we end up with a standardized measure of deviation from the mean regardless of whether the values are more or less than the mean.

Let's remind ourselves of the equation we need to calculate b. 

The symbol sigma (∑) tells us we need to add all the relevant values together.

If we do this for the table above, we get the following results:

∑(x-x ̅ ) * (y-y ̅ ) = 611.36

And 

∑(x-x ̅ ) ^2 = 94.18

Slotting in the information from the above table into a calculator allows us to calculate b, which is step one of two to unlock the predictive power of our shiny new model:

The final step is to calculate the intercept, which we can do using the initial regression equation with the values of test score and time spent set as their respective means, along with our newly calculated coefficient.

64.45= a + 6.49*4.72

We can then solve this for a:

64.45 = a + 30.63

a = 64.45 – 30.63

a = 30.18

Now we have all the information needed for our equation and are free to slot in values as we see fit. If we wanted to know the predicted grade of someone who spends 2.35 hours on their essay, all we need to do is swap that in for X. 

y=30.18 + 6.49 * X

y = 30.18 + (6.49 * 2.35)

y = 45.43

Drawing a least squares regression line by hand

If we wanted to draw a line of best fit, we could calculate the estimated grade for a series of time values and then connect them with a ruler. As we mentioned before, this line should cross the means of both the time spent on the essay and the mean grade received. 

 

And there we have it! A perfect* predictive model that will make our teachers’ lives a lot easier.

What are the disadvantages of least-squares regression?

*As some of you will have noticed, a model such as this has its limitations. For example, if a student had spent 20 hours on an essay, their predicted score would be 160, which doesn’t really make sense on a typical 0-100 scale. It’s always important to understand the realistic real-world limitations of a model and ensure that it’s not being used to answer questions that it’s not suited for. 

Outliers such as these can have a disproportionate effect on our data. In this case, it's important to organize your data and validate your model depending on what your data looks like to make sure it is the right approach to take. 

This article was originally uploaded on October 2, 2018 and was last updated on August 21, 2020.