The user interface for interactive robotic workstations

John L Dallaway and Robin D Jackson
Department of Engineering
University of Cambridge, UK

Richard M Mahoney
Applied Science and Engineering Laboratories
Alfred I duPont Institute, USA

Abstract

The importance of the user interface within interactive robotic workstations is examined and design criteria are discussed. The command of an interactive robotic device is considered as a direct manipulation system and the potential benefits of graphical user interfaces for such systems are outlined. Aspects of current work in user interface design are introduced in the context of the Cambridge University Robot Language (CURL). Interface development tools and evaluation techniques are described. Finally, the future of robot user interfaces is examined with respect to recent advances in computing technology.

Background

The user interface requirements of interactive and non-interactive robotic devices are fundamentally different. Industrial robots have been traditionally programmed using a teach pendant to guide the manipulator through a sequence of positions within a highly structured environment. This sequence is either repeated continuously or invoked on demand by a supervisor or sensor-based signal. In many cases, the robot performs a single sequence of movements only. More recently, sensor-guided controllers have enabled manipulators to perform increasingly complex operations. However, the common goal in most industrial applications of robots is to limit human participation to a supervisory level only.

Interactive robots, by definition, function on the basis of frequent bi-directional communication with the operator. The user interface of such devices must allow the operator to perform a variety of tasks, it must also be sufficiently flexible to cater for unanticipated circumstances within a semi-structured environment. These requirements suggest a radically different approach to user interfacing for interactive robotic systems.

Rehabilitation robotics

The authors' interest in interactive robotics lies in the area of rehabilitation. In this context, the usefulness of robotic devices is largely dependent on the degree of independence which they provide to their operators [1]. One measure of the power of a rehabilitation robot is in the mean duration between necessary interventions by carers or other personnel. The highly interactive nature of a typical robotic aid serves to emphasise the requirement for a high quality user interface. Shortcomings in the user interface can act as major deterrents to the widespread use of robotic systems in human service [2].

Within the EC TIDE programme, the authors have been involved with development of the RAID workstation [3]. Using this workstation (figure 1), severely physically disabled individuals are able to manipulate books and paper documents within a conventional office environment. Design criteria adopted for this project typify the user interface requirements for interactive robotic systems in general:

Figure 1. The RAID concept (robot simplified for clarity)

Direct manipulation systems

The command of an interactive robotic device falls into the category of a direct manipulation system for the purposes of user interface design. Direct manipulation systems may be considered in terms of the Syntactic-Semantic model of Objects and Actions (SSOA) [4]. SSOA underlines the desire to represent the command of the manipulator in terms of objects and actions in the task domain while minimising the operator's need for computer-related knowledge. Substantial research in the area of generic direct manipulation systems has demonstrated the following beneficial concepts: The advantages of these concepts are multiple and complex. However, it is evident that users of such interfaces are better able to learn and remember basic functionality and experience fewer errors in operation. Direct manipulation systems do not necessarily offer any speed advantage over command line approaches to robot command for experienced operators.

Graphical user interfaces

Graphical User Interfaces (GUIs) provide a rich environment in which to implement direct manipulation systems. Within the field of rehabilitation robotics, the command and interface aspects of service workstations have normally been implemented in personal computers. A widely used graphical environment for personal computers is Microsoft Windows. Windows applications offer some advantages over character-based software packages to people with motor impairments, but the applications generally require the use of software-based access solutions appropriate to the individual. An important factor is the consistency with which Windows applications access the computer hardware. This consistency facilitates the development of software-based access solutions for non-keyboard/mouse users which are suitable for all applications.

When considering the design of a GUI as an interface to an interactive robotic system for rehabilitation, it is axiomatic that a user's need for an assistive robot presupposes the existence of a motor impairment and makes it necessary to provide appropriate access to the interface. As a result, the fundamental benefits of direct manipulation may not always be available.

Human upper (and lower) limb movement may be described as making use of either unconstrained or constrained movements. The movement of the hand to pick up an object is considered to be an unconstrained movement. It is this human ability that interactive robotic systems are intended to replicate. In GUIs, information input using a mouse also requires a form of unconstrained movement. Instead of manipulating physical objects in the environment, the mouse user manipulates virtual screen objects in order to access the functionality of the computer application. Constrained movements are those in which the response of the physical system is limited so that only one or two dimensions of the system come under human control. Scanning is a common example of constrained manipulation that is used extensively as part of adaptive interfaces for people with motor impairment.

In terms of manipulation performance, the typical effect of motor impairment is to limit an individual's capacity to carry out unconstrained movements. In adaptive GUI interfaces, the need for unconstrained movements for direct manipulation may be overcome through two general approaches. One approach is to harness some other suitable unconstrained movement using an alternative input method making use of, say, eye movement or head movement. A second approach is to introduce a software application that makes use of a scanning technique to allow switch inputs to be transformed to selections. Examples of such software include the Windows Visual Keyboard (WiViK) [5] and Switch Access to Windows (SAW) [6]. The productivity of an individual using access software does not generally match that of a keyboard/mouse user. However, the strategic use of macro recording and playback facilities can compensate to some degree, even where a user is restricted to input via a single switch.

More research is required to establish the efficacy of alternative unconstrained input solutions for people with disabilities. In terms of interaction with a robot, the emphasis for such research depends on the control approach that is most appropriate for a particular application. If a set of pre-programmed routines can provide sufficient functionality, the research should focus on improving the method of selection within the GUI. If, however, the application requires extensive use of direct control over the motion of the manipulator, research must focus on improving the flow of information between the user and the movement of the manipulator. Evaluation of the demands of vision and the development of methods for reducing the GUI reliance on vision will also be useful in improving the efficiency with which an individual interacts with a robot.

The Cambridge University Robot Language

The Cambridge University Robot Language (CURL) is a Windows-based robot-independent programming environment designed and developed over several years specifically for applications in rehabilitation [7]. It provides an intuitive method of performing pick and place operations through the definition of objects and goals within a world model. Complex tasks may be realised by creating CURL procedures using commands which employ a natural language syntax. Direct robot control functions are also provided to complement the task-level facilities.

CURL has been employed in several robotic installations where it has been used as a command system for document manipulation [8], education [9] and manufacturing [10] tasks. The Interactive Robotic Quantitative Assessment Test (IRQAT) employs CURL to assess the ability of disabled individuals to use robotic equipment [11].

A number of user input devices have been used in conjunction with CURL. These include mice, switches, keyboards, trackballs, joysticks, touch screens and voice recognition systems. The Microsoft Windows environment allows new input devices to be implemented without modification of application software such as CURL.

CURL has recently been used as a tool with which to explore the power of the Microsoft Windows graphical environment for interactive robot command. The current version provides command of a robotic device at three levels, direct control, command line and procedure selection. Each level of interaction is provided by a separate overlapping window within the user interface. Operators are therefore able to rapidly switch between the various forms of command. The direct control window provides telemanipulation facilities in a number of robot co-ordinate frames. The operator selects a frame/axis combination and initiates motion by clicking on a button. The motion continues until the button is released. The command line allows experienced operators to enter CURL commands for immediate interpretation and execution. CURL commands use a natural language syntax to accelerate the learning process. Finally, the procedure selection window displays a scrolling list of previously defined CURL procedures. A procedure may be rapidly invoked by double clicking on the appropriate entry in the list. For safety purposes, a cancellation button appears while the robot is executing CURL commands. It should always be possible to halt the robot by a single action using any input device.

To provide greater versatility, the CURL user interface (figure 2) has been made configurable. Those aspects of the interface which are not required within a particular installation may be minimised to iconic form. The size and position of the windows which are required may be adjusted and saved. In this way, the operator is not confused by irrelevant interface elements.

Figure 2. The CURL user interface

Windows also enables multiple applications to co-exist on the desktop. This facility allows a complete workstation, such as the RAID system, to be implemented using a single computer, reducing cost, complexity and overall system size.

An interactive robot command interface

The current version of CURL has demonstrated many of the benefits inherent in the use of GUIs as robot user interfaces. Operators of CURL-based workstations have indicated their desire for a more intuitive approach to robot command. In this respect, alternative interfaces are currently under evaluation. One method for interactive, task-level robot command involves the representation of objects within the robot's working envelope as icons. The spatial relationships between the objects are conveyed by the relative positions of the icons on a display. A possible arrangement of icons for the RAID workstation is shown in figure 3. The representation of objects in this way facilitates the operator's perception of the user interface as a model of the physical environment. Using this metaphor, robotic tasks are invoked by a drag-and-drop technique. For example, a robot command procedure to retrieve a book from a shelf and place it on a reader board might be invoked by dragging a book icon to lie above a reader board icon. Following such a command, the arrangement of icons is adjusted to reflect the revised state of the physical environment.

Figure 3. A user interface for interactive robotic workstations

Interface development

The development and enhancement of user interfaces is generally iterative in nature. Users provide feedback to the development team who, in turn, provide improved interfaces for evaluation. This approach is now taken very seriously by several office software vendors who have established 'usability' laboratories in which users are closely monitored to determine the way in which they comprehend and operate unfamiliar software.

The iterative approach to user interface design requires that the developers are able to rapidly modify the interface components and their behaviours. The use of CASE tools in this context is increasingly popular. Within the Microsoft Windows environment, developers are also employing programming tools such as Microsoft Visual Basic. This product facilitates the rapid prototyping of user interfaces. More computationally intensive algorithms are created using other development tools and may be linked to the user interface at run-time. The use of CASE tools during user evaluation of robot user interfaces is proposed. Such tools would ensure a rapid response to feedback from the operators.

The future

Interprocess communication

Within the next few years, object-oriented computing technologies will substantially change the way users interact with computers. A major change will be the move away from application-centred computing, where a data file is created and manipulated within a specific software package. In the future, data files will be more generic in nature and will be central to the computing procedure. Application software will act as tools to manipulate parts of the data as appropriate. One such technology is Microsoft Object Linking and Embedding (OLE). OLE 2.0 provides document-centred facilities within the Microsoft Windows environment.

A particularly significant aspect of OLE 2.0 allows an application to 'export' any number of its functions to other applications. OLE Automation enables developers to create application containers which combine the features and functions of several applications. These containers can be tailored to provide the precise functionality required for a specific task. In the context of rehabilitation robotics, it is possible to employ the facilities of OLE 2.0, and in particular OLE Automation, to provide task-centred robot command. For example, a disabled office worker involved in the preparation of company accounts, could, by invoking the accounts data file, load an application around the existing data to facilitate editing. The application's tool bar could include buttons to retrieve the invoice folder from the shelf using a robot. Other applications incorporating different robot command buttons could be associated with each major task in which the worker is involved. The robot command software would run as a background task and the worker need not be aware of its presence.

Telepresence systems

In contrast with the distributed robot user interface concept described above, the authors are currently considering the use of real-time video within the user interface for certain robotic workstations. Recently introduced standards for PC-based video hardware and software have created the possibility of user interfaces in which the operator interacts with a real-time video image using a pointing device. A selected object could be highlighted and manipulated within the image by superimposing computer generated images on the display.

Another possibility for future user interface design lies in the application of virtual reality technologies. A demonstrator system, built at the Swedish Institute of Computer Science, Kista, allows an operator to control the end effector by directly manipulating an on-screen representation of the environment [12]. However, the hardware costs of such a system are high and the resulting user interface is highly specific to a particular application. Price sensitivity within the assistive technology market may restrict the application of telepresence systems until hardware costs fall significantly.

References

  1. Jackson RD (1993) Robotics and its role in helping disabled people. IEE Engineering Science and Education Journal. 2. 267-272
  2. Leifer L (1992) RUI: factoring the robot user interface. RESNA 92 - Proceedings. 580-583
  3. Dallaway JL, Jackson RD (1992) RAID - A vocational robotic workstation. ICORR 92 - Conference Papers.
  4. Shneiderman B (1992) Designing the user interface. Addison-Wesley.
  5. Shein F, Hamann G, Brownlow J, Treviranus J, Milner M, Parnes P (1991) WiViK: A visual keyboard for Windows 3.0. RESNA 91 - Proceedings. 160-162
  6. Head P, Poon P, Morton C, Colven D, Lysley A (1993) Switch Access to Windows 3 (SAW) - A new concept in emulation techniques. ECART 2 - Proceedings. 22.2
  7. Dallaway JL, Mahoney RM, Jackson RD, Gosine RG (1993) An interactive robot control environment for rehabilitation applications. Robotica. 11. 541-551
  8. Holmberg L (1993) The installation of a robotized workstation at Samhall-HADAR. ECART 2 - Proceedings. 29.2
  9. Beitler MT, Stanger CA, Howell RD (1994) The design of an integrated interface to an educational robotic system. RESNA 94 - Proceedings. 448-450
  10. Dallaway JL, Jackson RD, Mahoney RM (1994) The application of Rehabilitation Robotics within manufacturing industry. ICORR 94 - Proceedings. 145-149
  11. Dargie G (1994) Handbook for the Interactive Robot Quantitative Assessment Test (IRQAT). Cambridge University Engineering Department Technical Report CUED/F - INFENG/TR 162.
  12. Andersson M, Fahlén LE, Söderlund T (1993) A virtual environment user interface for a robotic assistive device. ECART 2 - Proceedings. 29.3
This paper may be referenced as follows:

Dallaway JL, Jackson RD, Mahoney RM (1994) The user interface for interactive robotic workstations. Proceedings of the 1994 IEEE/RSJ International Conference on Intelligent Robots and Systems. 1682-1686