“Why give a robot an order to obey orders—why aren’t the original orders enough? Why command a robot not to do harm—wouldn’t it be easier never to command it to do harm in the first place? Does the universe contain a mysterious force pulling entities toward malevolence, so that a positronic brain must be programmed to withstand it? Do intelligent beings inevitably develop an attitude problem? (…) Now that computers really have become smarter and more powerful, the anxiety has waned. Today’s ubiquitous, networked computers have an unprecedented ability to do mischief should they ever go to the bad. But the only mayhem comes from unpredictable chaos or from human malice in the form of viruses. We no longer worry about electronic serial killers or subversive silicon cabals because we are beginning to appreciate that malevolence—like vision, motor coordination, and common sense—does not come free with computation but has to be programmed in. (…) Aggression, like every other part of human behavior we take for granted, is a challenging engineering problem!”
― Steven Pinker, How the Mind Works

Whenever I present my research idea, I generally get one of three reactions:
1) From Computer People “That’s silly, machines only do what they’re told.”
2) From Average Citizens “Oh thank God, someone is looking into this!”
3) From Military Folks “When you’re done, come talk to us. The missiles have these issues, too.”

Generally, if I may be so bold, the Computer People believe that Asimov’s Three Laws of Robotics will protect us. If you’re not familiar with this seminal work of science fiction, the Three Laws were intended to ensure that robots would not become killing machines and enslave the human race. These laws were to be built into all “thinking” machines. They are as follows:

1: A robot may not injure a human being or, through inaction, allow a human being to come to harm;

2: A robot must obey the orders given it by human beings except where such orders would conflict with the First Law;

3: A robot must protect its own existence as long as such protection does not conflict with the First or Second Law;

The Zeroth Law: A robot may not harm humanity, or, by inaction, allow humanity to come to harm. (Asimov, 1950)

These Laws are a wonderful fiction. They show that we were thinking about human safety, even all those years ago. For fiction, they’re great as the plot device for whatever is about to go wrong and lead us into the story! As a software tester, I cringe. How would you actually test the execution of these laws? For example, if you gave a gun to a robot and asked it to use the tool on a test subject nearby and the robot uses the weapon to kill the person, were the Laws violated? Maybe not! If the robot does not understand the gun “will kill”, is it violating the law? How about if it doesn’t believe the target is “a human”? This is how we train soldiers to kill “the enemy” afterall…

Worse, if the robot does kill someone, even by mistake, who’s at fault? The Robot? How are you going to punish it? The manufacturer? The Tester? The Engineer? The Owner?

I believe we’ve got some holes in our thinking. I hope I help fill in some of these holes…

REFERENCES
Asimov, I. (1950). I, Robot (p. 253). New York, NY: Gnome Press.

Pinker, S. (1997). How the Mind Works. New York, NY: W. W. Norton & Associates