Robot Learning from Human Teachers. Sonia Chernova

planners for demonstration.

      Figure 3.2: The correspondence problem arises due to the differences in the sensing abilities and physical embodiment between the human and robot, making it more challenging to accurately map between their respective state and action representations [49].

      For LfD to be successful, the states and actions in the learning dataset must be usable by the learner. In the most straightforward setup, the states and actions recorded during the demonstrations map directly to the sensing and movement capabilities of the robot. In other cases, however, a direct mapping does not exist between the teacher and learner due to differences in sensing ability, body structure or mechanics. For example, a robot learner’s camera will not detect state changes in the same manner as a human teacher’s eyes, nor will its gripper apply force in the same manner as a human hand. The challenges which arise from these differences are referred to broadly as the correspondence problem [186]. Specifically, the issue of correspondence deals with the identification of a mapping between the teacher and the learner that allows the transfer of information from one to the other.

      The correspondence problem lies at the heart of Learning from Demonstration, and is intertwined in the choice of both the human-robot interaction method and computational technique used for learning. Using a direct demonstration technique that does not require correspondence simplifies the learning process significantly as it removes one source of possible error—the mapping function that translates human capabilities to those of the robot. As discussed below, several demonstration techniques directly map between the actions of the teacher and those of the student, the primary examples of which are teleoperation of the robot through kinesthetic teaching [51] or a controller such as a joystick or computer interface [1, 237]. However, not all systems are amenable to teleoperation. For example, low-level motion demonstrations are difficult on systems with complex motor control, such as high degree of freedom humanoids. Furthermore, physically controlling the robot may not be natural, or even possible, in a given situation. Instead, the teacher may find it more effective to perform the task with their own body while the robot watches. Enabling the robot to learn from observations of the teacher requires a solution for the correspondence problem, the states/actions of the teacher during the execution must be to be inferred and mapped onto the abilities of the robot. Learning in such settings depends heavily upon the accuracy of this mapping. Finally, the teacher may not demonstrate the task at all, and instead observe the robot and provide critique or corrections to the current behavior. In the following sections we discuss techniques for enabling the robot to learn from its own experiences, observation of the teacher and the teacher’s critiques. We conclude the chapter with a discussion of the tradeoffs and implications that the choice of interaction mode has on the design of the overall robot learning system.

      Figure 3.3: (a) Kinesthetic teaching with the iCub robot [13]. (b) User controlling the full-body motions of an Aldebaran Nao robot using the Xsens MVN inertial motion capture suit [141].

3.2 LEARNING BY DOING

      Teleoperation provides the most direct method for information transfer within demonstration learning. During teleoperation, the robot is operated by the teacher while recording from its own sensors. Demonstrations recorded through human teleoperation via a joystick have been used in a variety of applications, including flying a robotic helicopter [1], soccer kicking motions [40], robotic arm assembly tasks [64], and obstacle avoidance and navigation [118, 237]. Teleoperation has also been applied to a wide variety of simulated domains, such as mazes [70, 214], driving [3, 66], and soccer [7], and many other applications. Teleoperation interfaces vary in complexity from hand-held controllers to teleoperation suits [159]. Hand-written controllers have also been used to teleoperate the robot in the place of a human teacher [11, 102, 221, 237].

      Kinesthetic teaching offers another variant for teleoperation. In this method, the robot is not actively controlled, but rather its passive joints are moved through the desired motions while the robot records the trajectory [51]. Figure 3.3(a) shows a person teaching a humanoid robot to manipulate an object. This technique has been extensively used in motion trajectory learning, and many complementary computational methods are discussed in Chapter 4. A key benefit of teaching through this method of interaction is that it ensures that the demonstrations are constrained to actions that are within the robot’s abilities, and the correspondence problem is largely eliminated. Additionally, the user is able to directly experience the limitation of the robot’s movements, and thus gain greater understanding about the robot’s abilities.

      Another alternative to direct teleoperation is shadowing, in which the robot mimics the teacher’s demonstrated motions while recording from its own sensors. In comparison to teleoperation, shadowing requires an extra algorithmic component which enables the robot to track and actively shadow (rather than be passively moved by) the teacher. Body sensors are often used to track the teacher’s movement with a high degree of accuracy. Figure 3.3(b) shows an example setup used by [141], in which the Xsens MVN inertial motion capture suit worn by the user is used to control the robot’s pose. This example demonstrates tightly coupled interaction between the user and the robot, since almost every teacher movement is detected by the sensors.

      Shadowing also allows for loosely coupled interactions, and has even been applied to robotic teachers. Hayes and Demiris [109] perform shadowing with a robot teacher whose platform is identical to the robot learner; the learner follows behind the teacher as it navigates through a maze. Nehmzow et al. [187] present an algorithm for robot motion control in which the robot first records the human teacher’s execution of the desired navigation trajectory, and then shadows this execution. While repeating the teacher’s trajectory, the robot records data about its environment using its onboard sensors. The action and sensor data are then combined into a feedback controller that is used to reproduce future instances of the demonstrated task.

      Trajectory information collected through teleoperation, kinesthetic teaching or shadowing can be combined with other input modalities, such as speech. Nicolescu and Mataric [190] present an approach in which a robot learns by shadowing a robotic or human teacher. In addition to trajectory information, their technique enables the teacher to use simple voice cues to frame the learning (“here,” “take,” “drop,” “stop”), to provide informational cues about the relevance or irrelevance of observation inputs and indications of the desired behavioral output. In Rybski et al. [225], demonstration of the desired task is also performed through shadowing combined with dialog in which the robot is told specifically what actions to execute in various states. Meriçli et al. [175] present a similarly motivated approach which additionally supports repetitions (cycles) in the task representation and enables the user to modify and correct an existing task. Breazeal et al. [36] also explore this form of demonstration, enabling a robot to learn a symbolic high-level task within a social dialog.

      Finally, some learning methods pay attention only to the state sequences, without recording any actions. This makes it possible to communicate the task objective function to the learner without traditional action demonstrations. For example, by drawing a path through a 2-D representation of the physical world, Ratliff et al. provide high-level path planning demonstrations to a rugged outdoor robot [215] and a small quadruped robot [143, 216]. Human-controlled teleoperation demonstrations are also utilized with the same outdoor robot for lower-level obstacle avoidance [216]. Since actions are not provided in the demonstration data, at run time a learned state-action mapping does not exist to provide guidance for action selection. Instead, actions are selected by employing low level motion planners and controllers [215, 216], and provided transition models [143].

      Figure