I really need honest opinion and advice about this guy I like?

Sam is a senior and I'm a sophomore in college. I met him officially through Cory, who's a classmate of mine, and who's also crazy for me, but I sometimes hang out with him just for fun. But both Sam and Cory knows that I don't like Cory even though he is nuts for me. Sam asked me out the next night, without letting Cory know about it, and we ended up having s*x but promised that he won't breathe a word to this to anyone. 2 days later, he called me drunk and accidently spilled the fact that we did it, while he was out with Cory and some other guy friends. Cory got mad at Sam because they were close buddies and almost got into a fight. Sam called me back again, and says that maybe we should stop seeing since Cory loves me (eventhough he already knows that I do not love Cory). So 2 weeks later without any communication, Sam randomly calls me up again apologized for what hapenned the last time and asked to hang out with him that night (He asked me to come over to this hotel). This time I refused because by the 2 week time, I thought a lot about him and I realized that I like him a lot, and I do not ONLY want a PHYSICAL relationship with him. He got kinda pissed and mentions that 'maybe next time we will hang out'. So there was no news of him for over a week, so I thought I'd ask him out myself next. He blew me off that night saying he had plans with his guy friends but mentions casually that 'maybe we might hang out again next time'. Do you think he blew me off because 'I blew him off?' I told him that night HE asked me out that I wasn't feeling too good and that I had a headache, which is a valid reason.. So now its been 4 days and I want to ask him out again, but I don't want to sound clingy or desperate for him. He wouldn't know that I like him, but I want to let him know. I heard many stuff about him that he's a drunkard/uses drugs and that he gets laid a lot, and that he broke up with his 4 year old relationship less than a month ago and also used to cut himself over her. But he was extremely nice to me and respectful towards the first night we hanged out. I want to let him know that I like him. Should I ask him out again or wait? Any good ideas? I need honest advice and opinions.