Too simplify or not?

I have adhd and frustrated because its hard to keep up with every thing. I work hard and make a very good income. I have a vey loving wife and 4 children. My baby is a senior in high school. I am 44 yrs old yet I have a strong desire to downsize to a condo or something smaller which would free up time and money and stress on myself. I thought maybe we as a couple would have more freedom and less cost thus being to travel and go out and play more. On the other hand my wife likes the big house and pool for the family parties and for the kids to come to as their families grow. Am I being selfish or should keep pushing forward for another 15 yrs until retirement?