AI risk was not invented by AI CEOs to hype their companies
I hear that many people believe that the idea of advanced AI threatening human existence was invented by AI CEOs to hype their products. I’ve even been condescendingly informed of this, as if I am the one at risk of naively accepting AI companies’ preferred narratives.
If you are reading this, you are probably familiar enough with the decades-old AI safety community to know this isn’t true. But I don’t have a good direct way to reach the people who could use this information, and still I hate to leave such a falsehood uncontested. So if this is obvious, I hope the post is still perhaps useful to point more distant and confused people toward.
~
I personally know that AI risk was not invented by the tech CEOs because I have been near the middle of it since at least 2009—before any of the prominent AI companies existed, let alone had CEOs who might be trying to hype their products.
Here’s are some miscellaneous events over the years to give you a sense of the implausibility of this:
2008 - I attempt to contact Eliezer Yudkowsky to inform him that I am ‘trying to figure out the optimal way to use my life’ and would like to hear a better account of why his plan (of worrying about AI risk) is good. I have read about it online, but would like a clearer account. Traveling the world shortly after undergrad later, I meet a handful of people in person in the Bay Area who care about this, and one argues strongly that I should prioritize AI risk over my previously preferred causes e.g. climate change. I decide to think about this.
2009 - I am still not very convinced that AI is the most important thing to work on, but go to stay with the people who are worried about it for a few months. I argue about it a lot with a handful of them. There seem to be about twenty of them locally in the South Bay, though many more who comment on the relevant blogs. My photography collection from this era is quite sparse.
I go to The Singularity Summit for my first time (and its fourth), which is very lively and full of people who are thinking seriously about the future of AI.
2010 - Deepmind is founded. (I am back at school.)
2011 - I start a philosophy PhD at CMU, hoping to be eligible to work at somewhere like the Future of Humanity Institute one day, which is a happening hub of discussion about existential risk, AI and other important issues, that I like to visit.
2012 - I visit the Bay more and hang out with the growing AI risk community there. I visit the UK and do the same. I go to the AGI 2012 Winter Intelligence Conference.
2013 - I move to Berkeley and work at MIRI for a semester during grad school. I measure algorithmic progress over time across various computer science domains, as input to expectations for artificial intelligence in future. I visit the UK and attend the Center for Effective Altruism’s ‘weekend away’ where we have a debate on which cause is best, between global poverty, animal welfare and extinction risk. Extinction risk wins—the crowd leaves having changed their mind in that direction on net. The three advocates just before or after:
2014 - I join MIRI properly. I research The Asilomar Conference and Leó Szilárd as evidence about whether it is worth people trying to deal with risks early, because people around mostly believe that the risks from AI are at least a decade away, and there is disagreement about whether that makes it futile. I run an online reading group about Superintelligence, a new book about AI risk. I co-found AI Impacts, a project to answer questions about the future of AI, because AI risk seems at least fairly plausibly the most important thing to work on, and I want to investigate more and share my thinking with others.
2015 - I attend the first FLI conference—it seems that more people and more prominent people are interested in AI safety! OpenAI is founded.
2016 - I lead a team to run the first Expert Survey on Progress in AI. The median probability given to an outcome of advanced AI that is “Extremely Bad (e.g., human extinction)” is already 5%.
2017 - Some people around me are getting very worried, and saying AGI will happen within several years. My survey gets a shocking amount of media attention, becoming the ‘16th most discussed paper’ in 2017 according to Altmetric. Apparently there is interest in this topic..
2018 - I go to a big workshop for people working on AI risk in the English countryside, and a Chilean summit where I talk on TV and the radio about AI risk. It feels like interest is still picking up, and I feel optimistic about talking to the public.
2019 - GPT-2 comes out. Someone tries to get it to name our house. My favorite names include things like “World peace: tigers and humans” and “rooftop hillside: the highest place in the world”. It is hilarious and useless, but also magical and wild. The things we have worried about for years are feeling more tangible, and people’s ‘AI timelines’ are shrinking.
2020 - The world is reminded that really crazy things can happen. AI Impacts becomes remote. I spend the year with my household, who are almost all working on AI risk. We enjoy whiteboards a lot and run at least one good house conference in this period.
2021 - Anthropic is founded
Was it "invented" for that? Obviously not.
Is it being "used" for that? I can't immediately say no!
Really appreciate this write up. I think it’s obviously true that AI risk wasn’t invented by the CEOs of the labs. However, while this is not meant to be criticism of you since it’s in fact worth your time to correct a falsehood like this, I do think that it’s not crazy or unreasonable to worry that the CEOs are strategically emphasising this narrative because it implies that their product will be really capable and impressive, and they might be able to achieve super intelligence. I don’t mean that they would have picked this exact thing to amplify if they had their pic of things that could be prominent in the discourse, but given the discourse. They in fact are working with. It does seem like one of those things that is in fact in their interest to talk up to a point. This is also the fact that it’s obviously in their interest to talk up the possibility of future super intelligence, and once you do that, it’s kind of hard for them to completely avoid all talk of risk, even if they would have preferred the super intelligence narrative without any risk whatsoever. It’s notable that all the leaders of the three big labs appear to be at least a little into the AI risk discourse. All the part of that is probably selection effects because if you think AI will be a really big deal, you’re obviously more likely to found a company around it, but still it’s unusual for people to believe stuff that’s really inconvenient for their pocket book, so it does imply that it’s not as inconvenient as you might think at first glance. Of course, the AI safety talk going too far and resulting in a pause or something like it would be really bad for these people, but right now that looks like a remote possibility and also an idea gaining a life of its own, especially when it began with a life of its own does not disprove the possibility that some actors are strategically hyping up the idea because it’s convenient for them in the short term. Obviously, this doesn’t actually disprove anything and idea can be both true and strategically useful for the CEOs to talk up. I just think that AI risk is a really convenient topic for them to talk about in some ways because it doesn’t come across as obviously self-serving, but also necessarily implies that AI a will be really big thing well worth all the investment. It’s getting if it works out and likely more. I think if talking about AI is cause a pure negative for the industry, you would expect none of the CEOs or at least only a small fraction to talk about it and most of them to adopt a much more dismissive attitude. Although of course that could also be because they’re trying to attract talent that is, in fact concerned about AI safety or because they know enough about the technology that they are genuinely concerned to some extent and motivated reasoning isn’t strong enough to talk them out of it. None of this contradicts you post to be clear. It’s just an important caveat that I think is worth acknowledging, even though your post is in fact, quite excellent and necessary corrective for the more extreme formulations of this view.