If your taming effectiveness gets too low you can knock it out and wait for it to wake up. Doing so will cancel the tame process and allow you to try again from scratch. Just keep an eye on the health with a magnifying glass to make sure it doesnt die while applying torpor