It's well known that testing can only prove the presence of bugs, not their absence.
One thing is clear about LLMs, is that they are a lot more capable if they can operate in an environment where they can get rapid feedback on their output. A compiler, a test suite. Or maybe they're trying to disprove a conjecture that just needs one counter-example. Without guard rails they just produce something that seems to fit with their prompt.
Some problems are asymmetrical. Computer security may be one of these problems. It looks like LLMs may be vastly more capable at breaking software than they are at making software secure. To break in, they just need to find one bug, one exploit that works. To prevent break-ins, they need to anticipate all possible exploits.
Like many problems software development, for example the ossification of development practices, this is not a new problem caused by LLMs, but an existing one made much greater by them.
Defending against LLMs (assuming they don't bring their own infrastructure to a halt) may involve poisoning the signals they use to determine success. Not just a container, but a house of mirrors that makes them think they've broken out when they haven't. If it's even possible to build such a thing.