Use a torch. They have very little health but are difficult to target. Using fire lights them up and kills them quickly and safely.
Bonus points if they are attacking at night, as you will have several extra, flying torches for a few seconds!
I’ve found that when there is just one they are neutral but if there’s 2 or more they will aggro. When they swoop down to attack you stab with a spear and it’s an easy kill. In a swarm punching won’t save you.