Beware of CLMs