When to stand your ground and when to cede

My husband has ADHD and other issues. Well, we just found out that someone passed away in The Bronx, 2.5 hours away from us. We both knew her from our younger years but my H was closer to her. He hasn't really kept in touch with anyone from his past since he forgets and hyperfocuses on things rather than people. Well we were both going to go to the service and he found out he has to work. It's something he can't get out of. I get very anxious when he drives in the city since he has a temper and I get nervous driving myself in the city so I knew I wasn't going alone. Well, we have a mutual friend that knows the person that died also and she may be going. He mentioned that I could ride with her. That would be fine if she weren't accident prone. So I told him I would not be comfortable with that and he got upset. He said that he wanted me there to represent him. Again I said that I didn't feel comfortable driving myself or being driven by this person because I didn't want to put my life in jeopardy, especially since I wasn't really close to the deceased. He then said, "I am asking you as a favor". 

Ok, I am not tit for tat but this man has on many occasions has flat out not done what I asked as a favor and it only required calling a family member to console them and he said he was not equipped for that. I was so angry when he said that and I told him how I felt and he still didn't do it for me. He has made MANY financial mistakes after not listening to me as well as illogical decisions that have hurt our marriage even after I tell him I can see it coming. So he does what he wants at our expense all the time. I have had my share of suffering at his hand. 

So a second time he said he felt bad for not being able to go and he asked if I was going and I reminded him about how I felt. He said, "Nothing is going to happen! Never mind!" and then he stormed off and left me in bed for hours alone. 

I understand what a favor is. I do them for him all the time and I inconvenience myself more than he ever will in a lifetime since he often forget things and calls me to deliver them to him and things like this. I am that girl that sees him in pain and give a massage without asking, brings bfast in bed, etc. He does NONE OF THAT unless I ask of course and then with a stank attitude. So here we are, I am put in a situation where I seem so horrible because I won't rep him at a funeral for someone he didn't even keep in touch with but brings back memories for him. Mind you, he said not too long ago that there was no one in NY left for him to see or make effort to see, one being his "best friend" who he emotionally disconnected himself from since his friend sorta sided with me when I left my H for a month due to issues we were having. He is so quick to shut people out but now he wants to make me a nervous wreck either driving myself or driving with someone that may kill me. He even told me last year he didn't want me driving with this person due to her accidents. 

I am feeling like maybe I should give in but then I also want to stand my ground since a favor is not a demand. 

This is the second time in our marriage that he has been upset that I don't drive in the city. The first time was about 15 years ago and I ended up not going to the city but paid for it. He called me unsubmissive and disrespectful even though I was scared to death of driving. 

Any suggestiions?