The hit to your nest egg may be greater—and last longer—than you think. Thinkstock By Anne Kates Smith, Executive Editor From Kiplinger's Personal Finance, December 2014 Household debt is on the rise, driven by increases in student loans. A Charles Schwab survey found that borrowers are tapping nest eggs, too, with one in four 401(k) participants surveyed having borrowed against their accounts.See Also: 10 Things You Must Know About 401(k)s Some loans were hardly essential: 9% of borrowers tapped a retirement fund to “buy something special”; 4% did so for a vacation. Borrowing against your 401(k) is rarely a good idea. Here are four reasons: 1. You may rationalize that you’re paying interest to yourself, so the loan doesn’t cost you anything. But if you’re earning 10% on the investments in your 401(k) (the long-term average for big-company stocks) and paying 4.25% interest (the current average) on the loan, you’re shaving your return by nearly six percentage points, plus all of the compounded growth on those earnings. 2. Borrowers often stop contributions while loans are outstanding, compounding the long-term hit to their nest egg. 3. You’ll repay the loan with after-tax money, which means, unless you’re borrowing against a Roth 401(k), the interest you pay will get taxed again when you withdraw it at retirement. 4. And if you leave your job for any reason, you’ll have to pay the loan back, typically within 60 days of leaving. If you can’t, you’ll owe taxes on the balance and, if you’re younger than 55, a 10% penalty as well.