Comparison of Modeling Techniques to Generate Requirements for Autonomous Vehicles
The automotive industry is heavily regulated and has many years of experience defining what a vehicle is and what it should be capable of doing. For the new breed of autonomous vehicles coming to market, requirements gathering efforts center on prototype behaviors such as how a vehicle would behave if a human were operating it, perhaps slowing it down for rough terrain, or stopping to avoid hitting a pedestrian. These examples are typical “acceptance testing” examples where we can certify that the product was intended to do something and it did, therefor the product is “good.”
Prototyping is a well known tool for gathering requirements in both hardware and software development. However, there are other tools available which may allow us to build more robust understandings of the final product. Tools like use cases, a variety of diagraming tools, and a variety of testing techniques allow us to more concretely define what the system shall do and when it shall do it. It is my belief that we need a clearer understanding of how the autonomous vehicle should behave in a variety of situations including worst-case scenarios where damage will be done to the fragile and litigious human beings the vehicles will be operating around.
Overview of Topic
The automotive industry has a great deal of prototype awareness. They have been building vehicles and know how they operate in a variety of human-controlled situations. A human-in-the-loop would be able to infer problems and correct for them, often learned from bad decision making, either by themselves or through the experience of others.
The seminal document for autonomous vehicles comes from the 1939 World’s Fair. General Motors defined the vision for an autonomous vehicle which would self-navigate from your home to place of business or shopping and back again. The vehicle (GM, 1939) would allow a typical family to ride in comfort, unconcerned by the world outside their vehicle. Today, these machines are finally approaching reality with a variety of companies like GM and Google creating and testing “driverless vehicles.”
Companies overseas are taking a different tact, attempting to keep humans “in the loop.” “Our idea is not that a car should totally replace the driver, but that it will give the driver freedom,” Yu Kai, Baidu’s top AI researcher, told the South China Morning Post. (Wan, A. and Nam, W, 4/20/2015) “So the car is intelligent enough to operate by itself, like a horse, and make decisions depending on different road situations.”
So far, these vehicles have mostly operated on test tracks with controlled environments, even at the DARPA level. So far, vehicle tests have mainly be conducted to show that a group of vehicles can operate “together” or can avoid simple obstacles. So far, less than half of the states in the US have contemplated laws for allowing such a vehicle on limited roads. As these vehicles approach the public, we will need to certify them as road-worthy and safe.
Importance to the field (Level 2 Heading)
Since these vehicles are intended to some day operate on public streets around fragile and litigious humans, they must be tested fully. To do that, the testers must have a set of requirements to work with which handles the myriad capabilities of the vehicle so that the tests can be exhaustively run. Prototyped requirements often leave out details of execution which could be fatal in this case, pun intended.
Is a Vehicle a Robot?
In the literature, we dream of robots that will clean the house (The Jetsons), drive us to work (I, Robot), run our errands (Red Dwarf), wage war (Terminator), and be our faithful companions (Star Wars). In real life, there are machines that do all these tasks, at least a prototype level. Companies like iRobot, Boston Robotics, and many others are working on robots to fill these needs and many others. But, the term “robot” really covers a variety of machine types and functions. Are all mechanical “beings” robots?
For the purposes of this discussion, let us define robots as a state between a mechanical construct like a 1980 Ford Pinto that requires full human interaction to perform and a presumably self-realized construct like R2D2 in Star Wars. A robot should also be able to make some decisions on its own but be governed by programming logic and rules.
This rather vague opening allows us to further discuss the parameters that define these machines. To be useful, the robot must do something. Let us further define that the robot must move and be able to do so under its own power and direction. Baichtal (2015) suggests that the robot must interact with its environment though sensors, programmed instructions, or human interaction.
At the moment, constructs like industrial manufacturing machines, the “sandwich robot” (JRK, 2011), and the next generation of automobiles which operate without direct human intervention fit this general viewpoint while machines like smart phones and vending machines do not.
To be useful, the robot must do something that benefits either a human or a process. This purpose generally defines the kinds of robots that exist. There are animatronic robots who entertain us, cleaning robots like the Roomba, assembly robots that build our cars and perform other industrial needs, combat robots which aid our soldiers, Drones and ROVs which allow a human operator to remain safe behind the lines while directing its mission, food and beverage robots who make, inspect, and assemble our foods, and of course, robots that are designed to interact as companions to people.
A robot is typically made up of a body placed over a chassis that contains mounting equipment for sensors, control systems, motors, manipulators, and a power supply. The body can be rigid or flexible and acts as a cover to protect the moving and/or fragile components of the robot. This “skin” layer can act as the barrier between the robot and the environment. If the body is considered the skin, the chassis could be considered the skeleton supporting the other components that make up the robot. Sensors work to being stimulation from the environment to the robot and may include optical, tactile, or aural capabilities. The sensors feed into the control systems that determine how the robot will behave based on the environmental inputs. The behaviors may be to move the robot, manipulate something in the environment, interact with a human or other input source, or any of a myriad of other tasks.
What is Testing?
Though testing was once the culmination of a successful development career, Juristo, Moreno, and Strigel (2006) lamented that the state of software testing is less advanced than other software techniques, which may be due in part to the psychological desire to gain satisfaction for creating a new product as opposed to testing something that already exists. They described how it is a common practice in software companies to relegate software testing tasks to junior personnel. Juristo, Morneo, Vegas, and Shull (2009) documented 25-years worth of software engineering testing techniques. Software testing techniques were grouped into five categories: randomly-generated test cases, where test administrators guess at potential errors and design test cases to cover them; functional testing, also known as Black Box Testing, in which testers examine possible input values to determine sets for which a behaving system should produce those sets; control-flow testing, which is one variety of White Box Testing, where test designers exploit knowledge of the internal order in which the code executes; data-flow testing, which is another variety of White Box Testing, where test cases are created to explore different executable paths, variable definitions, and the like; and mutation testing, in which versions of the code are generated to include deliberately injected faults—if the test cases detect the known faults in the “mutants” then the same tests should detect natural faults as well. Their testing of testing techniques revealed that none of the five techniques was significantly more effective than any of the others.
What I hope to learn/ Problem Statement or Description (Level 1 Heading)
Since we are using prototyped experience for defining what vehicles should do, how do we ensure we have gathered all the requirements? There are a variety of tools for requirements gathering. Taken together, tools like use cases, design diagrams, and questions poised by testing techniques allow us to have a more complete image of the end result we hope to achieve.
Use Cases
Each one of these sections can contain multiple paragraphs that obviously belong under this heading. Blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah….
Blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah….
Diagramming Tools
Each one of these sections can contain multiple paragraphs that obviously belong under this heading. Blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah….
Testing Techniques
Juristo, Morneo, Vegas, and Shull (2009) documented 25-years worth of software engineering testing techniques. Software testing techniques were grouped into five categories: randomly-generated test cases, where test administrators guess at potential errors and design test cases to cover them; functional testing, also known as Black Box Testing, in which testers examine possible input values to determine sets for which a behaving system should produce those sets; control-flow testing, which is one variety of White Box Testing, where test designers exploit knowledge of the internal order in which the code executes; data-flow testing, which is another variety of White Box Testing, where test cases are created to explore different executable paths, variable definitions, and the like; and mutation testing, in which versions of the code are generated to include deliberately injected faults—if the test cases detect the known faults in the “mutants” then the same tests should detect natural faults as well. Their testing of testing techniques revealed that none of the five techniques was significantly more effective than any of the others.
James Bach (2003) introduced a new concept into the Testing lexicon by defining the “Exploratory School” of testing. Using it, he believes testers should be free to define ad hoc tests based on results seen and find answers to “I wonder what would happen if…” kinds of questions that seasoned testers may ask. His method is to be used with the other tools, not instead of the other tools.
AI algorithms have been developed for test case generation, execution, and verification by Alsmadi (2002). He explained that the test cases are selected to ensure test adequacy with the least amount of test cases possible, and the user interface components are serialized to an XML file. This model transformation allows utilizing the user interface as a substitute for the requirements, which are often missing or unclear, especially for legacy software (Alsmadi, 2002).
Khoshgoftaar, Seliya, and Sundaresh (2006) noted that business resources allocated for software quality assurance and improvement have not kept pace with the complexity of software systems and the growing need for software quality. They claimed that a targeted software quality inspection can detect faulty modules and reduce the number of faults that occur during operation. They presented a software fault prediction modeling approach using case-based reasoning (CBR) by quantifying the expected number of faults based on similar modules previously developed for new modules under development.
References
Alsmadi, I. (2002). Using AI in Software Test Automation [Electronic version]. Department of Computer Science and Information Technology, Yarmouk
University, Jordan.
Bach, J. (4/16/2003). Exploratory Testing Explained. Retrieved from http://www.satisfice.com/articles/et-article.pdf
Baichtal, J. (2015). Robot Builder: The Beginner’s Guide to Building Robots. Que: Indianapolis, IN
GM (1939). New Horizons. Retrieved from http://www.1939nyworldsfair.com/worlds_fair/wf_tour/zone-6/general_motors.htm
JRK (September 29, 2011). PR2 getting a sandwich. Retrieved from https://www.youtube.com/watch?v=RIYRQC2iBp0
Juristo, N., Moreno, A., Vegas, S. & Shull, F. (2009). A Look at 25 Years of Data. IEEE Software, 26(1), \5-\7.
Khoshgoftaar, T., Seliya, N. & Sundaresh, N. (2006). An Empirical Study of Predicting Software Faults with Case-Based Reasoning. Software Quality Journal, 14, 85-110.
Wan, A. and Nan, W. (4/20/2015). Baidu’s Yu Kai talks autonomous cars, artificial intelligence and the future of search. South China Morning Post. Retrieved from http://www.scmp.com/lifestyle/article/1767799/baidus-yu-kai-talks-autonomous-cars-artificial-intelligence-and-future