Adding my two cents here. I realized some time ago that I'm usually annoyed about clutter when my mind is a mess. The mind clutter overwhelms, and then I start trying to organize/trash things to organize the mind. Sometimes it works, sometimes it doesn't.
When life is good, clutter doesn't matter much.