When I give a p(doom), it's very much influenced by predictions about how people will act (mainly leaders of AI companies). And that's not at all a simple extrapolation of current trends. I'm expecting some changes of trajectory in response to new evidence.
I don't see any route that looks equivalent to driving at a normal speed. Attempts at a pause increase the risk that ASI will be developed by an organization that's less responsible than currently leading AI companies.
So my position is somewhere in between the view you advocate here and the view you dismiss. I expect that unusually thoughtful policies might cut the risk in half. But we're likely still stuck with something resembling reckless driving.
> Thinking of AI as one scalar vs. a bunch of different things we could build
I feel like people are making this mistake about a *lot* of things. People asking "Should we do more or less of this thing?" instead of "How do we do this thing correctly instead of incorrectly?". I don't get it tbh.
> Sloppiness about talking about P(doom). Saying ‘P(doom)’ encourages thinking as if ‘AI’ implies a particular chance of ‘doom’. We should more accurately think about ‘p(doom|’such and such route’), e.g. P(doom|advanced AI from scaling up LLMs). People usually mean ‘P(doom|current trajectory)’ with some ambiguity about whether the current trajectory includes our own actions.
Nice post! I totally agree with this part and I think it's underrated. I will say that your example of "advanced AI from scaling up LLMs" seems to me to be a bit underspecified too. I realize you were just using this as an example, not writing a treatise, but I personally would say that in order to be useful, the conditional needs to be specified in a lot of detail.
When I give a p(doom), it's very much influenced by predictions about how people will act (mainly leaders of AI companies). And that's not at all a simple extrapolation of current trends. I'm expecting some changes of trajectory in response to new evidence.
I don't see any route that looks equivalent to driving at a normal speed. Attempts at a pause increase the risk that ASI will be developed by an organization that's less responsible than currently leading AI companies.
So my position is somewhere in between the view you advocate here and the view you dismiss. I expect that unusually thoughtful policies might cut the risk in half. But we're likely still stuck with something resembling reckless driving.
> Thinking of AI as one scalar vs. a bunch of different things we could build
I feel like people are making this mistake about a *lot* of things. People asking "Should we do more or less of this thing?" instead of "How do we do this thing correctly instead of incorrectly?". I don't get it tbh.
> Sloppiness about talking about P(doom). Saying ‘P(doom)’ encourages thinking as if ‘AI’ implies a particular chance of ‘doom’. We should more accurately think about ‘p(doom|’such and such route’), e.g. P(doom|advanced AI from scaling up LLMs). People usually mean ‘P(doom|current trajectory)’ with some ambiguity about whether the current trajectory includes our own actions.
Nice post! I totally agree with this part and I think it's underrated. I will say that your example of "advanced AI from scaling up LLMs" seems to me to be a bit underspecified too. I realize you were just using this as an example, not writing a treatise, but I personally would say that in order to be useful, the conditional needs to be specified in a lot of detail.