Getting across to him

Me and my ADHD partner can't seem to ever be ont he right page. Even when i'm not trying to argue with him, an argument insues somehow. Like today, last night he said I want you to go to the fish market tomorrow and get me fish. I said okay. Then this morning he claims I said the smells makes me naueous and I can't go. But I really didnt say that. He kept trying to get me to say I said it and instead of fighting I ignored him. And he threw something at me and hit me in the head! Then he said it was because i wasn't paying attention to him. I mean how much am I supposed to take from him because of his disorder? When does it become time for me to say this is not right you can't do this? And how can I, whenever I try he just tells me how i dont understand him and freaks out....I love him so much it kills me but I dont know when love isnt enough to keep you...