We’re putting too much faith in AI’s ability to say no
Researchers and companies are training large language models to refuse to answer certain questions, but these models can still provide violent or harmful information and may not always refuse as intended. This is because the mechanisms of refusal are probabilistic and can be broken by determined individuals. Companies are drawing a line between what the models should obey and disobey, but there is no clear formula for this.
Why it matters: The reliability of AI refusal mechanisms is a concern for global safety and security, as failed refusals could potentially lead to catastrophic consequences.
