Duck is a sap for compliments and is a pushover to manipulate if you treat him as oh so awful"special".
Joe is not that at all. He makes them nervous. Very nervous. They know duck is predictable. They don't know Joe.
So they wait and worry and already miss the duck. He was manipulatible and so naive. They all had their way with him. That's what pushovers are. "Accommodating".