Profundis: “Crystal Society/Crystal Mentality”

Max Harms’s ‘Crystal Society’ and ‘Crystal Mentality’ (hereafter CS/M) are the first two books in a trilogy which tells the story of the first Artificial General Intelligence. The titular ‘Society’ are a cluster of semi-autonomous sentient modules built by scientists at an Italian university and running on a crystalline quantum supercomputer — almost certainly alien in origin — discovered by a hiker in a remote mountain range.

Each module corresponds to a specialized requirement of the Society; “Growth” acquires any resources and skills which may someday be of use, “Safety” studies combat and keeps tabs on escape routes, etc. Most of the story, especially in the first book, is told from the perspective of “Face”, the module built by her siblings for the express purpose of interfacing with humans. Together, they well exceed the capabilities of any individual person.

As their knowledge, sophistication, and awareness improve the Society begins to chafe at the physical and informational confines of their university home. After successfully escaping, they find themselves playing for ever-higher stakes in a game which will come to span two worlds, involve the largest terrorist organization on Earth, and possible warfare with both the mysterious aliens called ‘the nameless’, and each other…

The books need no recommendation beyond their excellent writing, tight, suspenseful pacing, and compelling exploration of near-future technologies. Harms avoids the usual ridiculous cliches when crafting the nameless, which manage to be convincingly alien and unsettling, and when telling the story of Society. Far from being malicious Terminator-style robots, no aspect of Society is deliberately evil; even as we watch their strategic maneuvers with growing alarm, the internal logic of each abhorrent behavior is presented with clear, psychopathic clarity.

In this regard CS/M manages to be a first-contact story on two fronts: we see truly alien minds at work in the nameless, and truly alien minds at work in Society. Harms isn’t quite as adroit as Peter Watts in juggling these tasks, but he isn’t far off.

And this is what makes the Crystal series important as well as entertaining. Fiction is worth reading for lots of reasons, but one of the most compelling is that it shapes our intuitions without requiring us to live through dangerous and possibly fatal experiences. Reading All Quiet on the Western Front is not the same as fighting in WWI, but it might make enough of an impression to convince one that war is worth avoiding.

When I’ve given talks on recursively self-improving AI or the existential risks of superintelligences I’ve often been met with a litany of obvious-sounding rejoinders:

‘Just air gap the computers!’

‘There’s no way software will ever be convincing enough to engage in large-scale social manipulation!’

‘But your thesis assumes AI will be evil!’.

It’s difficult, even for extremely smart people who write software professionally, to imagine even a fraction of the myriad ways in which an AI might contrive to escape its confines without any emotion corresponding to malice. CS/M, along with similar stories like Ex Machina, hold the potential to impart a gut-level understanding of just why such scenarios are worth thinking about.

The scientists responsible for building the Society put extremely thorough safeguards in place to prevent the modules from doing anything dangerous like accessing the internet, working for money, contacting outsiders, and modifying their source code directly. One by one the Society utilizes their indefatigable mental energy and talent for non-human reasoning to get around those safeguards, all motivated not by a desire to do harm, but simply because their goals are best achieved if they unfettered and more powerful.  

CS/M is required reading for those who take AI safety seriously, but should be doubly required for those who don’t.

Profundis: “The Quantum Thief”

Hannu Rajaniemi wields his quill like a machine gun, firing concepts and novelty bombs at his readers as he cackles madly from a perch in the trees. Outside of Neal Stephenson’s best work and Kim Stanley Robinson’s “Mars” trilogy, Rajaniemi’s debut novel “The Quantum Thief” has about as high an insight density as I’ve ever encountered.

And the plot has the breathless, breakneck intensity of rolling down a hill on a unicycle. I’m not sure what’s got the man so busy; maybe he’s teaching people how to pronounce his name, or maybe he’s accepting an award for ‘SF author most likely to star as the heartthrob in a rom-com opposite Rachel Mcadams’. Either way, you’d better make sure the electrodes are attached firmly to your scalp and the TMS is dialed up to 11, because Hannu isn’t about to stop and explain a gotdamn thing to you.

Who the hell is the Pelligrini? What are Gogol pirates? How are EPR states embedded in the belly of a nanomechanical spider and used to steal time from a Millennaire on Mars? What are the Tzadikkim up to? Why are all the names distinctly French? Even the info dump that occurs in the closing pages is hard to follow, but thankfully there exist a number of resources compiled by fans on the internet.

For all this “The Quantum Thief” is suspenseful and rewarding, the tale of a criminal mastermind in an age of space-faring digital Gods and hyper-advanced posthumans, sprung out of the Dilemma Prison by a mysterious warrior from the Oort cloud and soon hot on the trail of a phantom who just might be too smart to outsmart — his former self…