If something else in the area attacks it (while you're close) it will agro on you for being there too.

More Vulture Encountering Tips