We've updated our Privacy Policy to make it clearer how we use your personal data.

We use cookies to provide you with a better experience. You can read our Cookie Policy here.

Advertisement
Rectangle Image
Article

Calculating a Least Squares Regression Line: Equation, Example, Explanation

Rectangle Image
Article

Calculating a Least Squares Regression Line: Equation, Example, Explanation

Read time:
 

Being able to make conclusions about data trends is one of the most important steps in both business and science. It’s the bread and butter of the market analyst who realizes Tesla’s stock bombs every time Elon Musk appears on a comedy podcast, as well as the scientist calculating exactly how much rocket fuel is needed to propel a car into space. 

How to Find a Least Squares Regression Line

Often the questions we ask require us to make accurate predictions on how one factor affects an outcome. If a teacher is asked to work out how time spent writing an essay affects essay grades, it’s easy to look at a graph of time spent writing essays and essay grades say “Hey, people who spend more time on their essays are getting better grades.” What is much harder (and realistically, pretty impossible) to do by eye is to try and predict what score someone will get in an essay based on how long they spent on it. Sure, there are other factors at play like how good the student is at that particular class, but we’re going to ignore confounding factors like this for now and work through a simple example.

Our teacher already knows there is a positive relationship between how much time was spent on an essay and the grade the essay gets, but we’re going to need some data to demonstrate this properly.  

Least Squares Regression Line Example

A graph comparing grades with hours spent on an essay.
Suppose we wanted to estimate a score for someone who had spent exactly 2.3 hours on an essay. I’m sure most of us have experience in drawing lines of best fit, where we line up a ruler, think “this seems about right”, and draw some lines from the X to the Y axis. In a room full of people, you’ll notice that no two lines of best fit turn out exactly the same. What we need to answer this question is the best best fit line. 

A graph that plots possible regressions between grade and hours spent on essay.


Through the magic of least sums regression, and with a few simple equations, we can calculate a predictive model that can let us estimate grades far more accurately than by sight alone. Regression analyses are an extremely powerful analytical tool used within economics and science. There are a number of popular statistical programs that can construct complicated regression models for a variety of needs. A simpler model such as this requires nothing more than some data, and maybe a calculator. It’s worth noting at this point that this method is intended for continuous data.  

Least Squares Regression Equations

The premise of a regression model is to examine the impact of one or more independent variables (in this case time spent writing an essay) on a dependent variable of interest (in this case essay grades). Linear regression analyses such as these are based on a simple equation:  
 
Y = a + bX 

Y – Essay Grade    a – Intercept     b – Coefficient      X – Time spent on Essay

There’s a couple of key takeaways from the above equation. First of all, the intercept (a) is the essay grade we expect to get when the time spent on essays is zero. You can imagine you can jot down a few key bullet points while spending only a minute on an essay and still get a few points here and there. Every essay will have at least this score according to our model. On top of that, every hour we spent on our essays (X) leads to an increase of b in the grade the essay gets.  We can work out b through the following, slightly scary equation:
An equation for calculating the coefficient, b.

But we’re getting ahead of ourselves. To calculate b, and make sense of that creepy equation, we’re going to need to know the values for our data:

A table showing values of data.

How to Calculate Least Squares Regression Line by Hand


When calculating least squares regressions by hand, the first step is to find the means of the dependent and independent variables. We do this because of an interesting quirk within linear regression lines - the line will always cross the point where the two means intersect. We can think of this as an anchor point, as we know that the regression line in our test score data will always cross (4.72, 64.45).  

A graph contrasting grade with hours spent on an essay.


The second step is to calculate the difference between each value and the mean value for
both the dependent and the independent variable. In this case this means we subtract 64.45
from each test score and 4.72 from each time data point.  Additionally,
we want to find the product of multiplying these two differences together.


A data table with dependent and independent variable calculations.

You should notice that as some scores are lower than the mean score, we end up with negative values. By squaring these differences, we end up with a standardized measure of deviation from the mean regardless of whether the values are more or less than the mean.

Let's remind ourselves of the equation we need to calculate b

An equation for calculating the coefficient, b.

The symbol sigma () tells us we need to add all the relevant values together.

If we do this for the table above, we get the following results:



∑(x-x ̅ ) * (y-y ̅ ) = 611.36


And 


∑(x-x ̅ ) ^2 = 94.18


Slotting in the information from the above table into a calculator allows us to calculate b, which is step one of two to unlock the predictive power of our shiny new model:



The final step is to calculate the intercept, which we can do using the initial regression equation with the values of test score and time spent set as their respective means, along with our newly calculated coefficient.


64.45= a + 6.49*4.72

We can then solve this for a:


64.45 = a + 30.63

a = 64.45 – 30.63

a = 30.18


Now we have all the information needed for our equation and are free to slot in values as we see fit. If we wanted to know the predicted grade of someone who spends 2.35 hours on their essay, all we need to do is swap that in for X


y=30.18 + 6.49 * X

y = 30.18 + (6.49 * 2.35)

y = 45.43


Drawing a Least Squares Regression Line by Hand


If we wanted to draw a line of best fit, we could calculate the estimated grade for a series of time values and then connect them with a ruler. As we mentioned before, this line should cross the means of both the time spent on the essay and the mean grade received. 


A least-squares regression line is shown to link grade with hours spent on essay.

 

And there we have it! A perfect* predictive model that will make our teachers’ lives a lot easier.

Disadvantages of least-squares regression


*As some of you will have noticed, a model such as this has its limitations. For example, if a student had spent 20 hours on an essay, their predicted score would be 160, which doesn’t really make sense on a typical 0-100 scale. It’s always important to understand the realistic real-world limitations of a model and ensure that it’s not being used to answer questions that it’s not suited for. 

Outliers such as these can have a disproportionate effect on our data. In this case, it's important to organize your data and validate your model depending on what your data looks like to make sure it is the right approach to take. 

This article was originally uploaded on October 2, 2018 and was last updated on August 21, 2020. 



Advertisement