My ex has ADHD - I still care about him but I’m confused

I met my ex last year and it was amazing. From our first date there was chemistry like I had never had before with anyone. He was very caring, attentive, communicative. It seemed too good to be true. He would travel to London twice a week to see me and made a lot of effort. We seemed to have a lot in common and want the same things. He asked me after the first date to be exclusive and after our second date to be in a relationship. He seemed a bit intense - but everything was going really well. Early on he told me that he had recently realised he had ADHD but had not been officially diagnosed (he is 38).

After about a month of dating we got into an argument at my house because my dog woke him up and he ended up walking out and leaving. He later apologised and explained it wasn't about me. A few days later he came back to see me and assured me that everything was okay. We continued seeing each other as normal and another month later it happened again. We were talking and not agreeing on something and he packed his stuff and left but this time he said it was over. Again, he came back the next week wanting to make things up and assuring me it was all okay. Telling me he loved me as a person and did I love him. Asking me if when I would move in with him etc. Then he said he would spend one day of Christmas with me so again he came over, I was annoyed that he was so late and he took his things, walked out and ended things again. He then apologised and kept messaging me to make sure I was ok. 

About a week later he explained it was because he was having a hard time because he couldn't be with his daughters over Christmas and his grandma was terminally ill. A couple more weeks went by and he kept arranging to see me and then cancelling last minute. This caused us to argue as I thought he couldn't be bothered to see me. He avoided talking about what was going on over the phone and I felt totally disconnected from him. He said the way he felt about me had changed. We then agreed to meet and talk about what was wrong and how we could get things back on track.

When I saw him, instead of talking about what happened and telling me how he felt, he acted as if everything was ok. He behaved exactly how he had when we were a couple and stayed for the weekend but I knew something was wrong. A week later he told me he wouldn't be coming to see me and I got totally frustrated and gave him an ultimatum, either he finally told me what was going on or it was over. He refused to tell me so I called it quits. The next day he told me he had found out that his daughters had been taken into foster care and he had to go to court the next week and wasn't in a place to talk about us. He refused to see me to talk after that and told me he wanted to stay friends. I gave him space and left things friendly but didn't get in touch.

He recently contacted me after two months of not speaking, asking if I would come and stay with him for a few weeks. He then changed his mind and asked me to come for a few days. Then, he told me that if I saw him it would be just for sex and he didn't want to be together, it couldn't happen. I still don't know to this day why he ended it/felt differently about things or why he kept changing his mind about me. He doesn't want to be with me, but he doesn't seem to want to lose me either. I don't know whether to just leave it and move on or to try and be his friend and be more patient with him.

I now understand that the first part of the relationship and intensity was largely because of the ADHD. I think with the added stresses of his personal life and a new relationship maybe it was too much, but I still can't understand how he was so into me and wanted to be together so much and has now completely changed: any advice would be appreciated! I'm totally confused by it all and am struggling to move on as I cared for him so much. I don't really understand how he can say he cares for me but we can't be together.