How to get through to ADHD partner

I have been with my boyfriend for 4 years, and I have always known that he has severe ADHD. He is extremely hyperactive, but it never really bothered me too much. However, we have had a lot of problems in our relationship that I think are stemming from his ADHD. Recently I left work at 5:45 instead of 5:30, and he became extremely suspicious of why I left 15 minutes late. It turned into a huge argument because I am such an honest person and I love him so much, but he constantly thinks I'm "up to something", or he "smells something fishy". I have tried to talk to him so many times about this issue, that he needs to trust me and that he has no reason to doubt me. I have just gotten to the point where I feel like giving up, because every time I try to approach the situation, he gets defensive and puts up a wall. Everything is an attack on him.

He doesn't take medication usually. He only takes it when he has to study or has a test because it makes him extremely unfriendly, and he won't eat. His mom has told me how difficult it was to deal with him as a child, like everyone was against him. She read a lot of books and they even tried therapy. I recall him saying that he knew what the therapist was doing and the games that he was playing... I just think that is the way he views everything.

It's like he picks apart what he thinks other people are doing or thinking. He thinks they have ulterior motives. He then changes his actions so that he has the upper hand. It's almost like a game of manipulation with him, but he's the only one playing...

I am seriously at my breaking point and don't know how to get through to him! I want him to realize what he is doing, but he just feels attacked when I try to address it. Does anyone have a suggestion of how I could approach the situation? I have even thought about asking his mom about it, but then I feel like he might feel even more attacked...