Help! Should I leave Him??

We have been together for four years. We argue about petty stuff and he doesn't acknowledge me or my feeling. He tends to misread and misinterpret things I say and I how say it. He doesn't even second guess himself. Instead of talking to me like an adult. He lashes out, acts rude and agrumentive. Even when I'm crying in front of him. I try to tell him my feelings but a lot of time he just keeps interrupting me to tell me excuses or just say nasty things to me. I try to be patient because I know he has ADD/ADHD but I feel like I'm losing my mind. If he apologizes at its takes a lot of the time and its forced apology. He comes off really self centered. If I don't ask one I mostly will not get one. Whenever he feels blamed for anything he is not shy about making you feel blamed for it. He will blame himself and then blame me for remembering something. He has a bad memory. He tends to overreact. He can't do simple task sometimes. He has no problem with yelling at me in public. Its so embarrassing. I've actually had someone in the past approach me and she was worried about safety.This is a middle aged man acting like this. I don't think I should be treated like this or put up with this just because it might be part of his ADD. He doesn't have a job. I work and I'm in the middle of getting a new apartment but he would be homeless and live in his car. And doesnt talk to his family. I dont think he has appreciate things I have done for him. He is dependant on me. So I feel bad.. I'm a bad person if I leave him?? I really hate to say this but I feel like a would be abandoning a child because he can't take care of himself.