Should I call it quits?

Maybe he's just not that into you. Perhaps after going out there were things that he saw as red flags and decided that it wouldn't make a good match. Either way both of you are not ready for a relationship till you fix the issues that both of you have. Work on yourself and find a person ready and willing to give you what you desire.