I think it is his social standing. Zaroff comes off from a very well off family. This idea makes him look very intimidating. This idea also makes it easier for us to understand why he rather hunts human instead of hunting bigger games such as lions, tigers, and other dangerous animals. He sees humans as a better prey because of their intellectual ability which makes things more challenging.
