By Dr. Charles Herder and Cory Ondrejka
An on-call IT worker receives an urgent call from the CEO requesting an MFA reset. The IT worker grants access – only to find out later the voice they heard was not the CEO. The fraudulent access was used to initiate a ransomware attack and tens of millions of dollars in damages.
An open-source contribution from a developer with three years of productive and helpful contributions includes an innocuous configuration script missed by other maintainers. The script installs a DLL that grants remote code access to developer machines and infects thousands of machines before discovery.
Today, these scenarios require dedicated and organized attackers who methodically pick and research targets, exploiting public information to credibly impersonate or prioritize potential victims. These types of focused attacks are far more effective than older, “spray and pray” phishing schemes, but are expensive in terms of personnel and time. High-cost, high-reward, non-scalable.
AI Is About to Change This
With the emergence of Large Language Models (LLMs) and Generative AI, tasks that previously required significant investments in human capital and training are about to become completely automatable and turnkey. The same script kiddies who helped scale botnets, DDos (distributed denial of service), and phishing attacks are about to gain access to Social Engineering as a Service.
Rather than a profit-motivated or state-sponsored hacking group picking one or two targets at a time, in a world of cheaply scalable, multi-channel attacks, this style of attack is going to radically expand. Instead of a single IT worker or executive assistant, attackers will now target all of them. Sophisticated attacks that become cheap will move “down-market,” to target multiple small and medium-sized businesses. For state actors pursuing strategic aims, why carefully infect one open-source project when bot contributors with synthetic identities can try to build trust in hundreds of thousands of them?
When every phishing attack at scale is targeted and multichannel, every human interaction that isn’t face-to-face becomes a threat surface. When the cost gets cheap enough, the attack surface broadens horizontally and vertically, exposing far more companies, institutions, and individuals to attacks. We are already starting to see the beginnings of this trend, with attackers using AI to automate identification of vulnerable companies and individuals.
Neither our institutions nor personal relationships are prepared for this. The inherent trust of human-to-human interaction over voice calls, text messages, video conferences, social media, etc., is so ingrained into our business processes today that it will take years if not decades for our institutions to meaningfully respond to such a threat.
As Terrifying as This Sounds, It Is in Fact Significantly Worse
The same AI that is being used today to generate fraudulent content and influence discussions on the Internet is also capable of generating synthetic accounts that are increasingly indistinguishable from real, human accounts. It is now becoming economical to completely automate the process of operating millions of accounts for years to emulate human behavior and build trust.
Therefore, in a few years, looking at an account’s activity will no longer be a valid signal of whether that account is human or machine. Even if we stop naively trusting voice or video calls, the follow up methods for filtering out bots or false identities we rely on today will also fail us.
This is a far more immediate and substantial threat than quantum computing. AI botnets capable of credible social engineering attacks at scale are a certainty within months or years. Quantum-resistant encryption is meaningless if people and organizations can’t distinguish staff or customers from AI attackers.
For the purpose of identity security and social engineering, LLMs and Generative AI have effectively passed the Turing test. Therefore, any security paradigm relying on user behavior to distinguish humans from bots will fail. AIs are too good at credibly imitating and replaying human interactions online. The only answer is to provide people with a stronger form of identity, specifically a cryptographic solution without stored secrets.
Avoiding stored secrets – eliminating the complexity and replay threats inherent to key management – is the only path forward to truly enable identity to enable authentication of you and not a proxy of you. It is inevitable that stored secrets will be compromised, so we must instead pick identity systems where the assertions of identity cannot be stolen or replayed.
AI is forcing our hand. Unless and until we can empower humans to securely possess their own cryptographic keys, we face a world where AI advantages the attackers far more than the rest of us.
–
Charles Herder, Ph.D.
Co-Founder, Badge Inc.
Dr. Charles Herder is a recognized and published thought leader in cryptography and embedded systems, and has been an invited speaker at the Global Cyber Innovation Summit, the De Vinci Innovation Center, the Advanced Cyber Security Center, TU Automotive, and MIT. An expert in cryptography, systems security, AI, and quantum computing, Dr. Herder has four degrees from MIT, including a Ph.D. in Electrical Engineering and Computer Science from the MIT Computer Science & Artificial Intelligence Laboratory. Dr. Herder has worked on embedded systems security at Texas Instruments, launched a cybersecurity company that was acquired by a public company leader in network assurance, and most recently is the Co-Founder of Badge Inc., the award-winning privacy company enabling Identity without Secrets™.
Cory Ondrejka
Creator of Second Life
Cory Ondrejka is a consumer tech executive who has held senior technical roles at some of the world’s largest companies. He is known for co-creating and founding Second Life, a virtual world described by the World Economic Forum as “the forerunner of the metaverse.” Cory previously led the mobile transition at Meta as VP Mobile. At Google, he served as VP of Product Management and Technical Advisor to the CEO, where he led the Core Experience team responsible for improving the product experience across all Google products. His teams delivered an entirely new user experience for Google products, multiple cutting-edge Generative AI experiences, and transformed foundational technology depended on by billions of people. He is the CTO for SmartNews, where he oversees delivery of quality news to millions of readers in Japan and the United States. Cory is also a Senior Fellow at the Burnes Center for Social Change at Northeastern University, a graduate of the United States Naval Academy, and a former Naval Officer.
