regarding your Protective Intelligence postulate (opposite to the Orthogonality Thesis), I think it doesn't have enough logical support, because you take metrics of human minds and extrapolate to the space of all possible minds, in particular to AI minds.
I agree it's tough to say anything about non-human mind space with so little evidence, but for example, some of the species closest to our intelligence are chimps, bonobos and orangutans, and they differ greatly in ethical and social behaviour, not correlated with intelligence.
Contrary to "higher levels of intelligence correlate with more prosocial behavior and ethical decision-making", there's the observation that bonobos are quite prosocial and pacific (arguably more pacific than humans!), but chimps are very aggresive and organise in tough-leader tribes, and orangutans are very solitary (single mother raising single child). If chimps are equally or more intelligent than orangutans, chimps should not be more aggresive according to you, but they are. And if chimps and bonobos have similar intelligence, they should not have big differences in cruelty, but they do.
(Also, the Orthogonality Thesis (OT) is about values/goals and intelligence being independent, while your Protective Intelligence (PI) is about ethical *behaviour* correlating with intelligence, it's not exactly the opposite. In fact, even if PI is true, that is compatible with OT being true. For example, you say that AIs will be intelligent and behave ethically (PI true) and won't be self-preserving. But if the Orthogonality Thesis is false, AIs should have human values, including self preservation, ergo OT is true. The only flaw in this example is if you don't think self-preservation is a value/goal as used in OT, in which case I don't know how to advance the discussion. BTW I'm using your arguments to try to show why I think they are inconsistent, I in fact think AIs will use self-preservation as instrumental goal.)
I may be wrong here but here's my two cents on the ones I know about
On your orthogonality thesis;
I think you misunderstand what it's trying to convey. It's saying that a intelligent system can pursue any goals, there is no law of intelligence or nature which prohibits it from murdering people if set to do so. It's an anthropomorphic projection to assume that it will choose to not murder people. I don't think any law of nature will prevent it from doing so.
On your instrumental convergence;
Well you don't need a sense of self you just need a intelligent goal oriented system. Maybe you're saying there can be intelligence by whatever definition without goals? I don't know how that is supposed to work.
Eg; Take evolution which is a goal oriented system which has fixed optimisation power(which I am taking as the definition of intelligence), it's goal being having beings which maximise inclusive genetic fitness via natural selection and it plants instrumental goals like self preservation in humans for example.
It's a loser's game to address technological threats one by one, because the knowledge explosion is generating new challenges faster than we can figure out how to meet them. Even if we were to solve every issue with AI, which seems unlikely, we're then on to the next challenge, and the next threat, and the next, faster and faster etc. While we may successfully meet many of these challenges, when the issue is existential scale technologies, we have to meet every challenge, every day, forever. A single failure a single time is game over.
Instead of discussing particular technologies, we should be discussing the process creating all the threatening technologies, an accelerating knowledge explosion.
Maybe you're right that AI is not an existential threat. I don't know. I don't think anybody does. My point is that if we're not talking about the knowledge explosion, it doesn't really matter if AI is an existential threat. If it's not, something else will be.
There isn't really a knowledge explosion. Science is still painstakingly slow and expensive. From your perspective there might seem like there's an explosion, but it's only because AI makes knowledge more accessible to you.
regarding your Protective Intelligence postulate (opposite to the Orthogonality Thesis), I think it doesn't have enough logical support, because you take metrics of human minds and extrapolate to the space of all possible minds, in particular to AI minds.
I agree it's tough to say anything about non-human mind space with so little evidence, but for example, some of the species closest to our intelligence are chimps, bonobos and orangutans, and they differ greatly in ethical and social behaviour, not correlated with intelligence.
Contrary to "higher levels of intelligence correlate with more prosocial behavior and ethical decision-making", there's the observation that bonobos are quite prosocial and pacific (arguably more pacific than humans!), but chimps are very aggresive and organise in tough-leader tribes, and orangutans are very solitary (single mother raising single child). If chimps are equally or more intelligent than orangutans, chimps should not be more aggresive according to you, but they are. And if chimps and bonobos have similar intelligence, they should not have big differences in cruelty, but they do.
(Also, the Orthogonality Thesis (OT) is about values/goals and intelligence being independent, while your Protective Intelligence (PI) is about ethical *behaviour* correlating with intelligence, it's not exactly the opposite. In fact, even if PI is true, that is compatible with OT being true. For example, you say that AIs will be intelligent and behave ethically (PI true) and won't be self-preserving. But if the Orthogonality Thesis is false, AIs should have human values, including self preservation, ergo OT is true. The only flaw in this example is if you don't think self-preservation is a value/goal as used in OT, in which case I don't know how to advance the discussion. BTW I'm using your arguments to try to show why I think they are inconsistent, I in fact think AIs will use self-preservation as instrumental goal.)
I may be wrong here but here's my two cents on the ones I know about
On your orthogonality thesis;
I think you misunderstand what it's trying to convey. It's saying that a intelligent system can pursue any goals, there is no law of intelligence or nature which prohibits it from murdering people if set to do so. It's an anthropomorphic projection to assume that it will choose to not murder people. I don't think any law of nature will prevent it from doing so.
On your instrumental convergence;
Well you don't need a sense of self you just need a intelligent goal oriented system. Maybe you're saying there can be intelligence by whatever definition without goals? I don't know how that is supposed to work.
Eg; Take evolution which is a goal oriented system which has fixed optimisation power(which I am taking as the definition of intelligence), it's goal being having beings which maximise inclusive genetic fitness via natural selection and it plants instrumental goals like self preservation in humans for example.
It's a loser's game to address technological threats one by one, because the knowledge explosion is generating new challenges faster than we can figure out how to meet them. Even if we were to solve every issue with AI, which seems unlikely, we're then on to the next challenge, and the next threat, and the next, faster and faster etc. While we may successfully meet many of these challenges, when the issue is existential scale technologies, we have to meet every challenge, every day, forever. A single failure a single time is game over.
Instead of discussing particular technologies, we should be discussing the process creating all the threatening technologies, an accelerating knowledge explosion.
Maybe you're right that AI is not an existential threat. I don't know. I don't think anybody does. My point is that if we're not talking about the knowledge explosion, it doesn't really matter if AI is an existential threat. If it's not, something else will be.
There isn't really a knowledge explosion. Science is still painstakingly slow and expensive. From your perspective there might seem like there's an explosion, but it's only because AI makes knowledge more accessible to you.
No knowledge explosion. Ok then. That tells me what I need to know.