Should you pay off your mortgage when you retire?

Should I pay off my mortgage early to get rid of the monthly payments?
I retired five years ago at age 55 with a pension and a 401(k) from which I took fixed withdrawals through age 59½. Then, I rolled over the 401(k) to a low-risk annuity with a $200,000 balance. I have $50,000 left to pay on my mortgage (five years left on a 10-year loan with a 5.375% interest rate). With most of the interest paid on this loan, I'm wondering if it would be better to pay off the balance with an annuity withdrawal or continue to make monthly payments. I realize I would have to pay taxes on the annuity withdrawal, but I could use the extra $1,000 a month now rather than in five years.