New York City Council holds landmark AI oversight hearing

The New York City Council held a landmark oversight hearing on artificial intelligence Monday. 

The city says it is the first in the nation, and world, to hold such a hearing. 

City Council Speaker Julie Menin said that New York City needs to take matters into its own hands because the federal government said AI companies can self-regulate. 

"Both New York State and the state of California have both talked about potentially a kill switch. That's actually in our legislation as well. And that's one of the questions we have today for the companies. How would a kill switch work? Would that be effective?" Menin said prior to the hearing. 

The testimony included whistleblowers like former Anthropic employee Jacob Coxon, who previously said AI could destroy humanity by 2030. 

Coxon had a warning for the City Council. 

"On the current path, I think it is more likely than not that humanity loses control to these AIs and it could end in human extinction," Coxon said. "From my experience, the companies are being extremely reckless given the stakes." 

Coxon also said AIs may soon be capable of "improving themselves." 

"We're approaching the point at which the AI systems will be capable of improving themselves, so, doing the research that humans were doing previously," Coxon said. 

Coxon pointed to the culture of the companies. 

"The companies run on a startup mindset. Move fast, break things, fix them later," Coxon said. 

"Even if they do manage to maintain control, I don't think we should trust them with that control. I think there's a very real chance they could use it to become dictators or oligarchs here in the United States," former OpenAI employee Daniel Koktajlo said. 

Menin took those concerns to executives who testified virtually from Anthropic, OpenAI, Google and Meta. Alongside them was a poster of more than a dozen AI security incidents this year from their companies. 

"Can you quantify what the risk is, or are you not able to do that?" Menin asked the executives. 

"Speaker, I don't wish to be imprecise in relation to the question of quantifying. I don't have our framework in front of me right now," Shane Cahill of Meta said. 

"Given the serious safety risks that have been raised here, to not be able to quantify what the risk is is troubling at best," Menin said. 

Menin asked about responsibility in the event an AI goes rogue. 

"If one of your frontier AI models goes rogue and causes serious financial harm, compromises sensitive data, causes physical injury or contributes to a death or deaths, do you each believe your company bears legal responsibility?" Menin asked. 

"I believe that OpenAI takes its responsibility for safety very seriously. And we are responsible for developing and evaluating our systems to ensure that they are safe," OpenAI's Morgan Dwyer said. 

"So, is that a yes? Or that is a no?" Menin said. 

One company not in attendance is SpaceX, which is owned by Elon Musk. Menin said after a subpoena was sent to the company, it responded with a letter saying they want to work with the city, but they won't be at the hearing. The city is now pursuing legal action. SpaceX did not return a request for comment. 

The City Council is considering 10 bills that will regulate AI in New York City, including prohibiting the unauthorized depiction of public officials by AI, third-party validation, and the ability for a human to shut down AI models, as well as chat box data privacy. 

"At Anthropic, privacy, data privacy, user security is core to our safety commitment," Logan Graham of Anthropic said. 

"Companies should need your affirmative consent before using your conversations to train models," City Councilmember Frank Morano said. 

There was also discussion about leveraging the benefits of the technology.

"AI can help scientists develop new cancer treatments, help entrepreneurs build and grow businesses," Dwyer said. 

"Google has investments in AI upskilling to address exactly this challenge of ensuring that current generations, and the next generation, are well-equipped to benefit from AI," Alice Friend of Google said. 

In a letter to the City Council ahead of Monday's hearing, an OpenAI executive wrote in part, "We test models for dangerous capabilities and concerning behavior, strengthen security, and monitor for failures of our safeguards."

Meta said, "At Meta, we are working to make personal superintelligence widely accessible so that it empowers people ... and safety is core to everything we do."