Products That Use Generate AI as Safety-Critical Systems

In engineering, a safety-critical system is one whose failure could cost lives, cause serious property damage, or do severe environmental harm. Think of the software running an aircraft’s autopilot, or a hospital’s life support machine. Systems like these are held to the highest standards, and their development is subject to rigorous, independent oversight.

We think products that use generative AI deserve the same lens.

 

The Unseen Risks of Generative AI

At first glance, a text or image generator can look harmless. But as these products settle into daily life and professional workflows, their potential for harm grows fast.

A generative AI system is more than a creative tool. It’s a powerful engine that can:

  • Spread misinformation: it can produce convincing but entirely false news or medical advice that misleads a lot of people.
  • Generate harmful content: it can be used to create deepfakes, non-consensual imagery, and other malicious content that causes emotional, psychological, and reputational harm.
  • Open security holes: it can be prompted to write malicious code, assist phishing attacks, or build tools for cybercrime.
  • Reinforce bias and discrimination: trained on biased data, it can amplify existing prejudice, leading to unfair outcomes in areas like hiring and lending.

These failures aren’t abstract. They’re real, and they can be devastating.

 

The Call for a New Standard

Regulation for generative AI is still young, and it isn’t yet equipped for these risks. Companies are racing to ship, and without an independent view on safety, the exposure falls on all of us.

This is why AISF treats products that use generative AI as what they are: safety-critical systems. We work with organisations to think through that risk, we research where the real harms sit, and we produce guidance for building more responsibly.

We demand safety standards for our cars and our planes. The products shaping our future deserve the same seriousness.

If that matters to you, come and work with us.