overview for xkcd: It's more that you should be aware when you're making these jokes, or bringing up the topic, that just the word 'rape' feels like a punch in the stomach to a pretty large group of people, and these are often people who would never in a million years give you feedback. The people in this group aren't just rape victims. A lot of them are just ordinary members of a gender that's always been a little more afraid to walk around at night or go to parties by themselves, knowing in the back of their minds that no matter how they dress or what they carry, they always have something that some bigger, stronger person might decide to take violently. Guys making jokes who weren't brought up with this fear don't always appreciate that.
So if you want to be a decent human being, take that into account when deciding what jokes to make to what audience.
If you want to go for it anyway, that's completely up to you. You have the freedom to say anything you want. I just suspect there are a lot of people who cheerfully make these remarks and don't realize how their words are going to be received. (And if you aren't interested in how your words are going to be received, why are you talking in the first place?)