“To jostle” means to push, elbow, or bump against someone roughly while moving, typically in a crowd, in order to move past people or get more space. When the narrator says "...so that we would have had to jostle them to get by", they mean they had to push women and men in the crowd in order to be able to move forward.