The Mortgage Paydown Dilemma

I bought my first house in April 2012. We paid $185,000. Our loan was for $177k at 3% for 15 years.

We hated debt, made decent money, and at some point set a goal to pay this mortgage off in about 5.5 years. It was an FHA loan, which required dreaded Mortgage Insurance (MI) payments, at least until our mortgage balance was below $148k. I believe MI was around $63/month.

On our quest to pay off the mortgage, we worked towards a few milestones:

  1. Get rid of mortgage insurance… By making minimum payments, we would’ve dropped MI in May 2015. We got there in March of 2014. A year ahead of schedule! 🙌
  2. Get balance below $100k… With minimum payments, this would’ve happened in Oct 2019. I think we hit that number in January 2016. That’s $48k paid off in about 22 months. Not bad right?! Almost 4 years shaved off the mortgage!
  3. Pay the whole thing off… We were shooting for the end of 2017. We still had cash in the bank, we kept getting raises at work, so $100k in 2 years felt feasible. However, plans changed when we both quit our jobs in July 2016. I made a career change to be a real estate agent, while my wife took a couple of months off to find something less stressful. We went from dual-income to no income in a 2 week period.

In February 2017 we decided to buy a condo and move. Originally we planned on turning the house into a rental, but it just didn’t feel right. We bought the condo before our house sold. We sold it for $300k and had a remaining balance of $60k. This was a magical moment for us. Over $225k hit our bank account, and we’d already bought our new place.

Was it worth it?

We bought our new place with plans of turning it into a rental in the next year. Paying it off wasn’t really of interest. At 30 years old, I wondered “what should I do with my money now?” I didn’t discover the Financial Independence concept for about another 6 months, so I didn’t really have an investor mindset.

I want to look back and analyze these questions:

Would I have been richer by making minimum payments and investing all my surplus? Was aggressively paying down my mortgage the best financial decision?

Aside from the course I ran, I think there are 2 alternative plans I could have chosen. I’d like to measure both to see what would’ve put me in the best financial position and compare to what I actually did:

  1. Instead of making extra payments towards principal, invest the cash into VFIAX.
  2. Use a 30 year mortgage versus 15 year, and put the extra cash into VFIAX. The 30 year loan would’ve had a 3.25% interest rate. I’ll assume the MI would’ve been $83/month. A side note here… at the time FHA loans would drop MI on 15 year loans when you hit an 80% loan to value ratio. MI would never go away on the 30 year loans.

Many investors probably look at the 3% mortgage and think “I’ll borrow as much as I can get, because I can get more than 3% returns in the market.” Me in 2020 tends to think that way too. I didn’t in 2012.

Another way to think about this question: “is it better to pay the 3% interest on the mortgage, so you can invest your extra cash with hopes to earn greater than 3% returns?”

Payment History Details

My mortgage payment (P+I) was $1,223/month. Add another $63 when the mortgage insurance was in play. At Milestone 1 (get rid of Mortgage Insurance), we had paid an extra $11,500 towards principal in the first 22 months of the mortgage. So about $520 extra each month. My total monthly payments were $1223+$63+$520 = $1,806

From the time we dropped MI ($148k) to the time we hit $100k (Milestone 2) was 22 months. With minimum payments, the balance would’ve been paid down from $148k to $129k. So we dropped an extra $29k in principal in 22 months. About $1,300 in extra payments per month. My total monthly payments were $1223+$1300 = $2,523

From $100,000 to $60,000 (when we sold) took about 15 months. With minimum payments, the balance would’ve been paid down from $100k to $86k. That equates to $1,730 in extra payments per month. My total monthly payments were $1223+$1730 = $2,953.

Over the course of 5 years, if I had only made minimum payments, my balance at sale would’ve been about $128,000. In 5 years, I paid an extra $64,500 towards principal. My best guess is that I saved about $4,700 in interest payments. So across 60 months of payments, I averaged just over $1,000 in extra payments towards principal.

Investment Growth

Now I’ll look at the historical returns of the S&P 500 for each of the Milestones mentioned above. Then determine how much I could have contributed to investing during those periods with my 2 hypothetical financial plans.

Financial Plan 1: Keep 15 year loan, and invest extras into S&P 500

During Milestone 1 (May 2012 to Mar 2014) the annualized S&P 500 return was 22% (with dividends reinvested). If I invested $520/month for 22 months, my ending balance would’ve been $14,210.

During Milestone 2 (Mar 2014 to Jan 2016) the annualized S&P 500 return was 3.66%. With regular payments, MI would’ve gone away starting with my June 2015 payment. If I started with $14,210 and invested $1,237/month for 15 months, then $1,300/month for 7 more months my ending balance would’ve been $43,828.

During Milestone 3 (Jan 2016 to April 2017) the annualized S&P 500 return was 20.47%. If I started with $43,828 and invested $1,730/month for 15 months, my ending balance would’ve been $86,278.

My mortgage balance would’ve been $127,706. So my combined net worth would’ve been $86,278 – $127,706 = -$41,428.

Financial Plan 2: 30 year loan, and invest extras into S&P 500

With this plan, my monthly mortgage payment would’ve been significantly lower ($772 vs $1,223), so I’d be able to invest more cash into the market. It would’ve had a 3.25% interest rate vs 3% on the 15 year. Also, I never would’ve dropped the $83 monthly MI.

During Milestone 1, if I invested $951/month for 22 months, my ending balance would’ve been $25,430.

During Milestone 2, if I invested $1,668/month for 22 months, my ending balance would’ve been $64,993.

During Milestone 3, If I invested $2,098/month for 15 months, my ending balance would’ve been $119,467.

My mortgage balance would’ve been $158,683. So my combined net worth would’ve been $119,467 – $158,683 = –$39,216.

Synopsis

What I Did15 year + VFIAX30 year + VFIAX
DateMortgageMortgageInvestmentsNet WorthMortgageInvestmentsNet Worth
April 2012-$177,300-$177,300$0-$177,300-$177,300$0-$177,300
March 2014-$148,000-$159,656$14,210-$145,446-$170,703$25,430-$145,273
Jan 2016-$100,000-$141,016$43,828-$97,188-$163,701$64,993-$98,709
April 2017-$60,000-$127,706$86,278-$41,428-$158,683$119,467-$39,216

By paying my mortgage down aggressively, I ended up with about a $60,000 loan balance at the end of 5 years. So $60k in debt, $0 invested. Net worth = -$60,000.

If I would’ve kept my 15 year loan and invested all the extra payments, I would’ve ended up with a $127,706 loan balance and $86,278 investment balance. Net worth = -$41,428.

If I would’ve used a 30 year loan with a slightly higher interest rate and persistent MI, I would’ve ended up with a $158,683 loan balance and $119,467 investment balance. Net worth = -$39,216

I think we have a winner. Over the course of five years, if I would have invested with the same discipline I had used to pay down my mortgage, I would’ve ended up over $20,000 ahead with a 30 year mortgage.

Getting into the Weeds

A few notes should be made about this experiment. During those 5 years, the S&P 500 had an annualized return of 14.49% if you reinvested dividends. Pretty exceptional returns!

If you were to re-run this experiment in the previous 5 years (2007-2012), you would have ended up with a -1% return in the S&P 500. Your ending financial situation would’ve been better by just paying off the mortgage. My backwards looking scenario should not be taken as a silver bullet for what will always work.

If I used the 15 year loan and didn’t pay any extra towards my principal and invested $1000/month, I would’ve needed at least a 5.7% annualized return do better than my accelerated paydown. On the 30 year loan, I would’ve needed at least an 8% return to do better.

Taxes are another fickle subject. Mortgage interest and mortgage insurance are both tax deductible. Up through 2017, the standard deduction for a married couple was $12,700. I easily would’ve surpassed that number, so the extra expenses paid could’ve put me even further ahead. After Trump simplified the tax code, lowered tax brackets, and increased the standard deduction to $24,000, that extra interest and mortgage insurance likely would’ve had a $0 benefit to my taxes.

Also speaking of taxes, dividends paid out on the mutual funds would be subject to taxes. Those likely would’ve had a negligible impact.

This is a small sample period. Trying to pay off a mortgage in 5-6 years is kind of a tall order. I think most people’s accelerated payoffs are maybe 10-12 years. I suspect in those scenarios, the 30 year loan option has an even better outlook. This is because the investment balances have even more time to compound.

Personal Preference and Flexibility

I can’t tell you how many people have talked to me about how great it would be to not have a house payment and be debt free. The Dave Ramsey plan if you will… I’m not trying to downplay that objective, it’s an admirable goal and a better plan than 98% of homeowners have. But debt isn’t necessarily a bad thing. Just look back at the scenarios I outlined…

Would you rather:

A. Owe $60,000 on your mortgage and have $0 in the bank?

B. Owe $160,000 on your mortgage and have $120,000 in the bank?

EVERYONE reading this should be screaming for Option B.

In reality, both scenarios could cost you the same amount of money upfront. They would’ve for me.

Option B also gives you flexibility… If you wanted to spend $50,000 to buy a rental property, you’d have the liquidity to do so. Or if you wanted to put all your savings towards your mortgage, you’d end up $20k better off than Option A.

However, I suspect if you’ve gotten this far, you probably won’t want to do that. Why? Because you’ve grown to understand the power of compound interest. In those early years with a savings balance of say $10-20k, 6% annual returns may only mean $600-1,200/year. Returns like that can seem futile when you’re paying $5,000/year in mortgage interest charges. However, when your $20k investment balance grows to $100,000, a 6% return will earn you $6,000/year. Even if you don’t contribute another $1, in the next year that 6% will earn $6,360 in interest.

Put another way, a $200,000 mortgage at 3% interest would cost you under $6,000 in interest in its first year. The interest charges will decrease every year after that. A $100,000 investment account earning 6% interest/return would earn you $6,000 in its first year. The returns would increase every year after that. Feed the investment account! The more you pay towards your mortgage, the less it’ll save you in interest. The more you put into an investment, the more it can make you. If you’re looking at a 15-20 year investment period, odds are good you’ll earn greater than 3% returns. Some years you’ll lose money on your investing. Some years you’ll gain more than 6%.

If you really want to pay off your mortgage, do it. No one is going to give you a hard time when your house is paid off. However, with interest rates under 4%, I am going to pay the minimums, invest the rest, and let my investments compound. I am young, time is on my side, and the growth potential of my assets is greater than the certainty of my interest payments.

One thought on “The Mortgage Paydown Dilemma

  1. In hindsight, I feel the same way about my student loan debts. I aggressively paid down my loans at interest rates of around 5%, so a little higher than a mortgage these days. But I missed out on some huge returns in the market, and wasn’t really putting ANYTHING toward investments at the time. I just didn’t know any better. But ultimately, I’m debt free and investing now, so not the worst outcome. But the opportunity cost still gnaws at me looking back.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s