AI safety

Research and practices to ensure AI systems behave as intended, avoid catastrophic failures, and remain under human control as they become more capable.

Syntax

ai-ethics

alignment: model pursues intended goals
robustness: model behaves safely under distribution shift

Example

ai-ethics

// AI safety practices:

// 1. Red-teaming — adversarially test AI for failures:
//    - Jailbreaking attempts
//    - Prompt injection attacks
//    - Out-of-distribution inputs

// 2. Output monitoring — detect harmful outputs in production:
//    - Content classifiers
//    - Rate limiting suspicious patterns

// 3. Human oversight:
//    - Human-in-the-loop for high-stakes decisions
//    - Audit logs for all AI decisions
//    - Kill switches and rollback capability

// 4. Constitutional AI (Anthropic's approach):
//    Model critiques own outputs against a set of principles