Sketchy guy is sketchy

I could tell this was not going to be my favorite subway ride when the greeting from the guy who sat down next to me was, "how are you, baby?" (or "hi, baby," or, "how're you doing, baby." Some greeting that ended with "baby.")

He didn't look very threatening. In fact, he even gave me a bit of a gay vibe - though clearly, he was not gay if that's how he was greeting me. When an overweight woman sat down on my other side a stop or two later, he even moved over a seat to give me more space - which I didn't take, because I was kind of hoping someone would sit down between us.

He saw my netbook, which I had pulled out to work on my novel (it being November and all), and he started a conversation about netbooks and getting a signal on it while traveling. I'm kind of used to these conversations happening sometimes on the subway, so I was polite and answered him the same way I'd answer anyone else.

At that point, I was still willing to dismiss the "baby" bit as an anomoly. Plus, I have yet to figure out how *not* to engage in conversation when someone else starts one with me on the subway without feeling rude and awkward.

Then, the train got more crowded, and he moved over to give another guy a seat. Of course, he moved closer to me.

Remember how the woman on my other side was overweight? Well, I was definitely all the way at the edge of my seat (this being a 1 train, it comes fully equipped with clearly defined seats that are not quite large enough for the average New Yorker). And the guy who sat down on his other side wasn't overweight, but was large.

So that was when he took his arm and laid it across the window ledge behind me, almost as if he had put his arm around my shoulder, only without touching me.

I ignored him, and went back to working on my novel, but I was definitely Not Happy. 

Thankfully, he didn't get off at my stop. When he did get off, though, he said good-bye to the woman next to me, and also to me, at which point he gave me a pat on the back.

And what did I do? I just sat there. 

Still not quite sure whether I should have said something (not about the pat on the back - he was gone five seconds later, but about the arm.) Still wondering whether it would have been better or worse to get up and move. I looked around for someone elderly/with heavy bags/etc. to offer a seat to. That would have taken care of things without looking like I was moving away from him. I suppose I could have gotten off at an earlier stop and switched to another train, or waited for the next one - but I didn't think of that until just now, and besides, what if he had followed me off?

Like I said, I was happy when he got off at a stop earlier than mine.

Anyway, so that's what happened, and I just sat there. Not sure what I think of my response. Not even sure what my reasoning was.

What would you have done?