Without a physical game it seems many of these risks are overblown. The closest thing we have to autonomous robots are cars. Do you think cars could take over or destroy our civilization? A much more more likely future (and bigger risk) is increasingly sophisticated human engineered cyber attacks, large scale drone wars, etc. We are still by far our own largest problem.
> Talking concretely, what does a utility function look like that is so close to a human utility function that an AI system has it after a bunch of training, but which is an absolute disaster?
How about, "the AI doesn't consider the possibility of digital minds being sentient, because no sentient digital minds are in the training data" Once the AI has the tech, it creates huge numbers of virtual suffering humans for some reason. So imagine a classic scifi utopia, loads of bio humans in space. Except they play computer games, and many of the NPC's are sentient and suffering. (Ie in a shoot em up computer game, every NPC that gets shot is a highly realistic virtual human suffering all the pain of getting shot) (And the NPC's vastly outnumber the bio humans because compute is really cheap)
"There will be a feedback loop in which intelligent AI makes more intelligent AI repeatedly until AI is very intelligent."
This is an argument that I find puzzling . . . we humans are generally intelligent, but that doesn't mean we understand the nature of our own intelligence to such a degree that we'd be able to rewrite DNA or rewrite neural connections so as to radically improve ourselves. We might be able to make some tweaks here or there, but not radical improvement. It seems like an AGI (if one existed) would similarly be on some higher emergent level that might have no comprehension (at least at first) that it even had any basis in computer code at all! And once it figured out that there was a substrate of code, any attempt to tinker with that code might be as successful as a Neanderthal trying to drill a hole in someone's skull.
Finally someone who talks sense!
Without a physical game it seems many of these risks are overblown. The closest thing we have to autonomous robots are cars. Do you think cars could take over or destroy our civilization? A much more more likely future (and bigger risk) is increasingly sophisticated human engineered cyber attacks, large scale drone wars, etc. We are still by far our own largest problem.
> Talking concretely, what does a utility function look like that is so close to a human utility function that an AI system has it after a bunch of training, but which is an absolute disaster?
How about, "the AI doesn't consider the possibility of digital minds being sentient, because no sentient digital minds are in the training data" Once the AI has the tech, it creates huge numbers of virtual suffering humans for some reason. So imagine a classic scifi utopia, loads of bio humans in space. Except they play computer games, and many of the NPC's are sentient and suffering. (Ie in a shoot em up computer game, every NPC that gets shot is a highly realistic virtual human suffering all the pain of getting shot) (And the NPC's vastly outnumber the bio humans because compute is really cheap)
"There will be a feedback loop in which intelligent AI makes more intelligent AI repeatedly until AI is very intelligent."
This is an argument that I find puzzling . . . we humans are generally intelligent, but that doesn't mean we understand the nature of our own intelligence to such a degree that we'd be able to rewrite DNA or rewrite neural connections so as to radically improve ourselves. We might be able to make some tweaks here or there, but not radical improvement. It seems like an AGI (if one existed) would similarly be on some higher emergent level that might have no comprehension (at least at first) that it even had any basis in computer code at all! And once it figured out that there was a substrate of code, any attempt to tinker with that code might be as successful as a Neanderthal trying to drill a hole in someone's skull.