EA - My highly personal skepticism braindump on existential risk from artificial intelligence. by NunoSempere

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My highly personal skepticism braindump on existential risk from artificial intelligence., published by NunoSempere on January 23, 2023 on The Effective Altruism Forum.SummaryThis document seeks to outline why I feel uneasy about high existential risk estimates from AGI (e.g., 80% doom by 2070). When I try to verbalize this, I view considerations likeselection effects at the level of which arguments are discovered and distributedcommunity epistemic problems, andincreased uncertainty due to chains of reasoning with imperfect conceptsas real and important.I still think that existential risk from AGI is important. But I don’t view it as certain or close to certain, and I think that something is going wrong when people see it as all but assured.Discussion of weaknessesI think that this document was important for me personally to write up. However, I also think that it has some significant weaknesses:There is some danger in verbalization leading to rationalization.It alternates controversial points with points that are dead obvious.It is to a large extent a reaction to my imperfectly digested understanding of a worldview pushed around the ESPR/CFAR/MIRI/LessWrong cluster from 2016-2019, which nobody might hold now.In response to these weaknesses:I want to keep in mind that do want to give weight to my gut feeling, and that I might want to updating on a feeling of uneasiness rather than on its accompanying reasonings or rationalizations.Readers might want to keep in mind that parts of this post may look like a bravery debate. But on the other hand, I've seen that the points which people consider obvious and uncontroversial vary from person to person, so I don’t get the impression that there is that much I can do on my end for the effort that I’m willing to spend.Readers might want to keep in mind that actual AI safety people and AI safety proponents may hold more nuanced views, and that to a large extent I am arguing against a “Nuño of the past” view.Despite these flaws, I think that this text was personally important for me to write up, and it might also have some utility to readers.Uneasiness about chains of reasoning with imperfect conceptsUneasiness about conjunctivenessIt’s not clear to me how conjunctive AI doom is. Proponents will argue that it is very disjunctive, that there are lot of ways that things could go wrong. I’m not so sure.In particular, when you see that a parsimonious decomposition (like Carlsmith’s) tends to generate lower estimates, you can conclude:That the method is producing a biased result, and trying to account for thatThat the topic under discussion is, in itself, conjunctive: that there are several steps that need to be satisfied. For example, “AI causing a big catastrophe” and “AI causing human exinction given that it has caused a large catastrophe” seem like they are two distinct steps that would need to be modelled separately,I feel uneasy about only doing 1.) and not doing 2.) I think that the principled answer might be to split some probability into each case. Overall, though, I’d tend to think that AI risk is more conjunctive than it is disjunctiveI also feel uneasy about the social pressure in my particular social bubble. I think that the social pressure is for me to just accept Nate Soares’ argument here that Carlsmith’s method is biased, rather than to probabilistically incorporate it into my calculations. As in “oh, yes, people know that conjunctive chains of reasoning have been debunked, Nate Soares addressed that in a blogpost saying that they are biased”.I don’t trust the conceptsMy understanding is that MIRI and others’ work started in the 2000s. As such, their understanding of the shape that an AI would take doesn’t particularly resemble current deep learning approaches.In particular, I think that man...