How to communicate the need for space

I have an issue.  I've been invited to an impromptu dinner this evening with some friends from my church group.  I would like to go.  DH does not have plans, and I hate that I have to "ask permission" to do something like this.  Now thinking this through, I know that an ADHD mind may see this as a problem because it deviates from the normal schedule.  Also, I know that my husband will translate it into something like "she would rather be with these people than with me" or sometimes "is she up to no good?".  In reality, I haven't seen this group of friends since the last retreat we did together nearly a year ago and I'm eager to reconnect with other women of my faith.  I have never cheated or deviated from our marriage vows.  I love being with DH and our 3 children at home, but I also need some "away" time.

This is actually making me nervous because I know he will react with negativity!  What's worse, when I go the children will get an ear-full.  How do I approach it without a major argument and "tantrum"?  I just feel like this is so unhealthy and I'm so alone.