Understanding Implied Meaning--That's not what I meant!

I'm new to this site and this is my first post. I could put together a decent list of ADHD issues that impact (or could seriously impact, if we lived in the same house and/or were legally married) our relationship, but I wanted to ask first about the ADHD partner misunderstanding communication. This one is going to tear us apart and be our downfall in the end (which may be here soon....).  My partner of 8 years (on and off) will misinterpret something I've said---usually something lighthearted---and become impatient with me, even to the point of getting angry. Each time it happens, I feel like I'm going down a rabbit hole of crazy.

He isn't a mean person; he actually has a big heart and can be a softie. But, for some reason, he takes things I say "wrong" then gets very upset when I try to tell him "that's not what I meant at all". I'm rarely (if ever) able to explain myself---it just makes things worse. So I give up quickly and withdraw until its safe to "reappear". I haven't attempted to "fix" these efforts because it so easily devolves into another negative exchange. The saddest part is he's angry over something SILLY I said that he somehow took wrong. Here's one of my favorite examples:

     I was finishing up a month of skin pre-cancer treatment on my face. It was broken out in red splotches and too sore to try to wear makeup, in fact, I was wearing Aquaphor all over my face to work, which looks pretty attractive. During a phone conversation he asked if I wanted to go out that night to eat. I, KIDDINGLY and in a self deprecating way said "Sure if you want to be seen with me wearing a bag over my head" and then laughed. I was saying "yes" but warning him I'm lookin' pretty bad. I was also just joking about my face treatment. And I was in a good mood.

     Instead of saying "OK where do you want to go?" or "Oh I don't mind you wearing a bag over your head, haha", he says THIS:  "I ASKED you if you wanted to go out! Of COURSE I want to go out and I don't care what your face looks like! I wouldn't have ASKED you!!". He is impatient and irritable and quickly raising his voice. As usual, even though you'd think I'd be used to it, I'm thinking "Omg, here we go again. Why is he saying that? Why is he acting like I said some horrible thing? I'm being silly about how I look".   I go into a bit of shock mode when this happens.

     I try to calmly tell him "I know you asked me, [partner's name]. I'm just being silly, you know, self deprecating. Why are you so mad?" He's still "yelling" at me about how I could even ask him if he really wanted to go out. I raise my voice enough for him to hear me try to "explain myself" but he raises his louder. Then he begins saying "SEE, NOW we're ARGUING!"  I now know the conversation is doomed, so I say "OK, [partner's name], this isn't a big deal, I was just being silly, you're mad, let's just hang up, OK?"  He's still mad. He goes on another minute. I repeat that we should hang up. He hangs up without saying goodbye.

     Nice. And, no, we didn't go out that night.  This happens regularly in slightly different variations, but always the same theme. The added frustration is if I try to explain myself, he is just as angry because now I am telling him he's "wrong" once again and that he just can't understand anything. I realize this last part is probably due to his being criticized in his life for not understanding things (I assume). He wasn't diagnosed until his early forties.

Bottom line: he can't seem to roll with the flow of some conversations and gets upset over the lightest of things at times. Is this an ADHD thing and does anyone have this experience? I'm at a loss. I've wanted so many times to talk about it calmly (at a later time), but am afraid to try as he quickly gets impatient and irritable.