Even within his own party, confidence in his leadership appears to be waning