I must start by saying that I don't buy that this thought experiment[1] will come to pass, but I do appreciate that it is the only cognition hazzard that I ever came across that I can put some stock in. It's a mixture of the prisoner's dilemma [2] and Pascal's Wager[3], wrapped around in some hard sci-fi.

To simplify things: the thought experiment proposes that AGI[4] will eventually come, and with its god-like omniscience will know everyone that helped or hindered its creation. And, of course, distribute rewards and punishments accordingly. Just knowing about this puts you in one of those two camps.

One of the many things I think is fascinating about it is its quasi-religious effect. Like the concept of our planet's life being started by aliens, it puts a somewhat possible yet unverifiable spin on a mythological story. But, instead of Genesis, it's Armageddon.

I don't know why it is a basilisk, but I suppose It's because of the mythological creature's capability to infect everything it comes near, much like this thought experiment. This gives an insight into the human mind, about how we are prone to religious thinking even when we are actively trying to be rational. And it always seems to me that religious thinking is inherently catastrophic because it's talking about the judgment of your soul or humanity as a whole.

For my money, I always appreciate gradual solutions to gradual problems. If we gradually solve the small problems the moment they pop out (unemployment, convincing evidence fabrication, monopoly by shady actors, etc) when AGI comes about, maybe it will be sufficiently fangless for us to deal with it rationally, not by panicking.

References

[1] Roko's Basilisk (opens new window)

[2] Prisoner's dilemma (opens new window)

[3] Pascal's Wager (opens new window)

[4] A definition of AGI (opens new window) Sparks of Artificial General Intelligence (opens new window) Closer to AGI (opens new window)