Our attention is pulled in the direction of where we see other people looking. It happens so automatically that experts have assumed it's a reflex response, impervious to conscious factors, such as the particular identity of the gazer whose line of sight we're following.