The 62-year-old comedian is set to host the last-ever episode of The Late Show on Thursday, after CBS announced in 2025 that it was axing the late-night TV talk show, and Colbert is now feeling philosophical about the decision.
"I mean, you can't do this forever. You have to think about, well, when is the right time to end your tenure? I didn't think this soon, but I mean, who knows?, he told People.
"Maybe CBS saved my life because it takes a lot of bone marrow to do the show every day, and now I'll be stepping down with enough time, enough energy to do other things that I want to do."
Colbert also revealed how he'll be spending his first day after The Late Show ends.
"I'm going to my brother's wedding in Washington DC," he said.
"My [eight] brothers and sisters and [their] husbands and wives are coming to the last show, and then we're all getting on the train the next day and going down to DC to go to Tommy's wedding."
Colbert is looking forward to the focus being on his brother, rather than himself for once.
"It's great, I love it. The next day focus is not on me, focus is on my brother. So much better. We'll get drunk and we'll sing. We all think we have good voices. That's the great danger of our family, especially the men. It'll be great," he said.
Colbert announced the axing of The Late Show in July.
The comedian - who replaced Letterman at the helm in 2015 - thanked CBS and TV viewers at the time for their support of the show.
"I'm not being replaced. This is all just going away. I do want to say that the folks at CBS have been great partners … And I'm grateful to the audience, you, who have joined us every night, in here, out there, and all around the world."