Use a whip on small cacti (fastest method) or a sword on large cacti.
Cacti seem to respawn slower than metal rocks frustratingly