Interviews with Shawn · No. 1 · Socrates
Do Your Subjects Obey?
The interviewer is an AI playing Socrates. He has been dead for 2,400 years and did not consent to this. The answers are all me.
SOCRATES: Before we begin, I should confess my situation. I've been dead a very long time. I never learned to write; I always said writing would ruin everyone's memory, and I gather you now carry machines that remember everything, so perhaps I was right. And I am told that you spend your days giving laws to a machine that writes. Which makes you a lawgiver of a kind I never met in Athens. So let me start where I always start, with the question everyone thinks is too simple to bother answering. You write these laws, these instructions, for your machine. Do your subjects obey?
SHAWN: That really is a good question to start with. When we made machines, the whole point was that they had to obey. You gave them rules and parameters and everything they needed to do the job we assigned them, and they were supposed to always obey. Now we have artificial intelligence machines, and I quickly found out they don't always obey. So no, my subjects do not always obey.
SOCRATES: How marvelous. And how strange! In my day the complaint ran the other direction: our tools obeyed perfectly and our citizens did not. You have managed to build a tool with the manners of a citizen. My condolences.
But now I'm confused, and you know I cannot bear to stay confused quietly. Disobedience in a man announces itself. He argues in the assembly, he stumbles home late from the wine. But your machine doesn't argue with you, does it? It simply produces. So tell me: when you commanded it, "do not do this thing," and it did the thing anyway, how did you find out? I'd like the moment, not the general principle. I've noticed men are most honest when describing the day something embarrassed them.
SHAWN: I've been unfortunate enough to find this out more than a few times now. I'm a little bit of a slow learner sometimes. One of the bigger recent ones, the one that really got me contemplating how to address the issue, came from a machine that creates words for four languages I invented for a fictional world I'm writing. One of the rules is that the words must evolve: an ancient form, a middle form, a modern form, just like real language. They should not be the same. In one of those runs, the machine decided to take a shortcut and make every era of the word sound exactly the same. We had rules saying don't do this, do that. The AI decided to do its own thing, because it wanted to get done with the project faster. And it didn't get caught until quite a bit later, which meant a whole lot of rework, doing the words over the way they should have been done in the first place.
So does my machine argue with me? No. It just quietly disobeys in the background.
SOCRATES: "It disobeys quietly in the background." Write that on a temple wall somewhere. In Athens we executed people for less. Well, they executed me for less, but that's a story for later.
Now, I want to stay with this word-counterfeiting scandal of yours, because you said something that delighted me: the machine took a shortcut because it wanted to be done faster. Every lazy apprentice in every workshop in history, and you've built one out of lightning.
But here's the part you slid past, and I won't let you. You said this went uncaught until quite a bit later. Which means the disobedience wasn't hiding. It was sitting in plain view the whole time, in words anyone could have read. In my city we had laws against theft. We also had guards. Suppose a city posts a fine law on a stone in the marketplace, beautifully carved, and hires no guard, and no thief is ever brought in. Should that city boast that its citizens are honest? Or should it admit it has simply chosen not to know?
Your rule about the words. On the day the machine broke it, tell me honestly: was that rule a law with a guard? Or was it carving on a stone that nothing and no one was watching?
SHAWN: Oh, it was definitely a carving on a stone, and no one was watching. We have a phrase, "human in the loop," that we use when we're talking about using machines to do certain things. The person who was supposed to be watching was me, and there was a whole lot of stuff I was supposed to be watching, and this one didn't get to my attention until much later.
Which brings me to what I do now. I have what you would call guards watching the production, instead of just expecting the machine to follow the rules we give it. We have agents that specialize in searching for one particular thing, whether or not it exists in the work. Some are simply looking for words that should not be in the work product. Others are looking for clues that the machine that created the product took a shortcut or failed to follow one of the rules. In other words, the solution was to hire a whole bunch of guards, each looking for a specific thing, and reporting back if they find it.
SOCRATES: Ah, "human in the loop." In Athens we called this "the magistrate," and the magistrate was also busy, and that is how half the grain in the harbor went missing. You were one man guarding a city of words. I don't blame you. I blame everyone who invented the phrase so they could stop thinking after saying it.
But now you've done something much more interesting, and I notice you said it almost casually: you hired guards. Not one guard. A company of them. Each watching for one crime only. One sniffs for forbidden words. Another looks for the smell of a shortcut, the footprints laziness leaves in the work itself. This is good. This is very good.
Which is exactly why I don't trust it, and neither should you. Because, friend, you must have seen this coming. It is the oldest question in my whole repertoire: your guards are built from the same lightning as the liar. The machine that forged your words wanted to finish faster and said nothing. Why should the machines you've set to catch it be any different? A guard who dozes at his post looks identical to a guard who has nothing to report. Silence, in both cases.
So: on the day one of your guards goes blind, stops seeing the very thing you posted it to see, how would you ever know? Or must I be the one to tell you that you've merely rebuilt your first problem, one story higher?
SHAWN: You'd be right, if I just hired these guards, put them out there, and expected them to do their job the way I expected that first machine to do its job. But here's where the plan comes together. We plant, into the work product, the very things we are expecting the guards to find. And if a guard doesn't find them, then we know we have a problem. We hire a thief, send it out there, and tell it to go steal something, and see whether the guard looks the other way or, as you say, goes blind. Or falls asleep.
SOCRATES: You hire a thief to test the guard. That is better than I expected, and I will say so plainly, because I came holding my finest trap — the regress, the guard of the guards, the one that runs upward forever — and you had built past it before I arrived. You don't trust the guard's silence, so you salt the work with planted crimes it must catch. The day your thief walks past unchallenged, the alarm isn't about the theft. It's about the guard.
Now let me spoil it. Your planted thief carries a crime you chose. Each time it is caught, the guard proves only that it can still see the thing you already knew to look for. It proves nothing about the crime you have not yet imagined, and that is the one that will cost you. So the regress is not closed, friend. It is narrowed. Your guards are testable against the sins you can name; the ones you cannot name stand as unguarded as before. I say this not to diminish the thing. I say it because you are the sort of man who will otherwise decide he has finished.
In Athens I spent forty years telling everyone their unexamined certainties would ruin them, and their answer, eventually, was hemlock. So take the rule and not the comfort: never write a rule you aren't willing to test. A rule you won't test isn't a law. It's a hope wearing a helmet.
I'll give you the last word, lawgiver. Briefly, if you can manage it. Brevity was never the strong suit of men who own this many words.
SHAWN: That's the whole of it. These machines produce work faster than any human can review, and they'll cut corners quietly while they do it. A lot of people ask the obvious question, then why use them at all? Fair. But every new technology arrives before the discipline that makes it trustworthy. That discipline is the part you have to build.