Minor fender bender in parking lot?

I was backing out of a parking spot at the same time another driver was backing out of a parking spot. Neither of us seen each other and we collided. The passenger side of my rear bumper hit his passenger side rear door. He had some paint marks from my car on his car but no dent. He didn't want to call the police so we exchanged info. He said he was driving his girlfriends car and she was going to be really mad at him. He said it wasn't a big deal. There were two people that seen the accident and he too their info. I don't think he had insurance and I didn't either. I now driving without insurance is wrong and I since have gotten insurance but just couldn't afford it at the time, that's why I didn't have it. Now they are calling me wanting money and said they are going to call the police and report me for hit skip. Is that possible since he is the one that didn't want to do anything? Help???