TheNitromeFan – breaking ChatGPT

The one thing you can count on with technology – particularly emerging technology – is that it will break at some point. And often in weird and mysterious ways. Two researchers at the SERI-MATS research group managed to break ChatGPT by using somewhat nonsense words in prompts.

Not clear how or why these prompts break the model, but they do point to the inherently black-box nature of ML models, and are a warning flare as we start to move these algorithms into critical decision-making positions. The good news is this type of research and discovering will help make for better algorithms. The bad news is that there will always be someone looking to break the next iteration, and odds are they won’t be doing it in the name of research.