They get angry at any raft that isn’t close to the shore. As far as I can tell, only two exist on the map at a time. If you learn what their locations are, you can tell when to pull back to the shore. One of these guys destroyed my raft in the icebergs and killed me and my friend. I have a burning hatred for them.