Please note that this page has been moved to a new location: http://hri.cogs.indiana.edu/~pscherme/HRI.html. You should be redirected there in a few seconds.
AAAI 2007 Human-Robot Interaction Challenge
Chairs:
Matthias Scheutz
Paul Schermerhorn
Artificial Intelligence and
Robotics
Laboratory
University of Notre Dame
mscheutz [at] nd [dot] edu
pscherm1 [at] nd [dot] edu
Building on the success of the Open Interaction Challenge in 2005 and last
year's Human-Robot Interaction challenge, the goal of the AAAI 2007 Human-Robot Interaction
is to demonstrate engaging interactions between people and robots.
This year's challenge will again provide a structured framework for
competition that will both allow teams to compete directly in seven
pre-defined categories and allow judges to better evaluate the employed AI
techniques and their level of sophistication. Critically, all
categories will be aimed at human-robot interaction and involve activities
that intrinsically integrate perception and action. In addition, all
categories will also involve one or more higher-level AI techniques (e.g.,
natural language understanding, reasoning, learning):
- Recognition of and
reaction to human
motions and/or gestures (e.g.,
mimicking human motion; naming a human's action; following a hand
motion command; determining the referent of a pointing finger; etc.)
- Emotion recognition and
appropriate emotion expression (e.g.,
noticing surprise in a human face and expressing surprise in response;
determining frustration in the human's voice and expressing regret for
being slow in understanding; noticing somebody said or did something
funny and laughing at the right moment; etc.)
- Natural language
understanding
and action execution (e.g.,
following requests from humans to move in particular ways; getting a
requested item from some other person; understanding descriptions of
directions and applying them to lead other people to specific places;
etc.)
- Perceptual learning
through
human teaching and subsequent
recognition and categorization of people, objects, locations or actions
(e.g., remembering the face of a person; learning of a location in the
environment and being able to remember and recognize it; learning
what it means to "turn around" and being able to repeat it; learning
what an object like a Coke can
looks like and recognizing it among other different objects; etc.)
- Perception, reasoning,
and action
(e.g., noticing that a person
moved behind an obstacle and did not reappear, concluding that the
person must still be behind the obstacle, and announcing where the
person is; determining that an object that the robot cannot move is
blocking its way, noticing that a person is moving a similar object in
a different location, and asking a person close-by to help with moving
the object; etc.)
- Shared attention,
common
workspace, intent
detection
(e.g., remembering referents from previous sentences and being able to
disambiguate "this" and "that"; following human eye gaze to
determine objects of interest in the environment and using shared
attention in constructing referents in sentences or picking topics of
conversation -- "did you see that? the door just closed";
deriving human intent from multimodal information including gestures,
body language, facial expressions, head movements, prosodic
information, and linguistic expression -- "I see you are pressed for
time, maybe we can chat later." derived from watching the person after
the person nervously turned the head left and right and looked at their
watch, or "Would you like me to get you a drink?" after observing that
the person had noticed that everybody in their group had a drink; etc.)
- Integration challenge:
demonstration of an extended, multimodal interaction that combines at
least
three (!) of the above six categories. As a specific instance,
with clearly specified conditions, the previous AAAI Robot Challenge would
fall under
this category (i.e., (1) starting at the entrance to
the
conference center and finding the registration desk, (2) registering
for
the conference, (3) performing volunteer duties as required, (4)
interacting
with conference attendees, and (5) finally reporting at a prescribed
time
to a conference hall to give a talk). However, other integration
projects are also welcome. Teams who are planning to participate
in category 7 should submit a short description of their "integration
challenge" to the chair for approval.
All entrants may compete in as many categories as they like, but must
compete in at least one. Beyond meeting the basic requirements
for a category, we are looking for systems that are interesting and fun
to interact with.
Information for Participants:
Scheduling:
During the exhibition, each team will be given at least two specific time
slots during which their entry will be featured. One of the time
slots will be used for judging. Teams may also practice or
demonstrate their entries at other times during the exhibition as long as
they do not conflict with the featured team. Each team will need to discuss
the needs of their entry (area required, etc.) so that we can best
coordinate.
Judging:
As in last year's competition, a combination of ratings from audience
members, other teams, and a judge panel will be used to determine the
winners. Winners will be determined for each individual category and
certificates will be awarded for outstanding or creative examples of
different types of AI and social interaction. A separate award will
be given to any team meeting the AAAI Robot Challenge in category 7 (that
team may or may not be the winner of category 7). Moreover, an
overall winner of the Human-Robot Interaction challenge will be determined
based on the individual category results. The technical evaluation
will be performed independently for all 7 categories. Each team will have
up to 15 min. to demonstrate categories 1 through 6 and an additional 10
min. for category 7 (the integration challenge). Additional judging
policies will be discussed prior to the challenge as needed.
The following criteria will be used for the technical evaluation:
- appropriateness for category
- robot performance
- effectiveness of interaction
- ease of interaction
- complexity of architecture
- robustness of architecture
- novelty of architecture
- level of integration of different AI components
We will assign numbers to each criterion and use the sum of all
criteria scores to determine a category score. For each category we
will determine the winner based on the category score. The team with
the most category wins will win the technical competition. To account
for the difference in difficulty of the 7 categories, categories will
be in addition weighted (by 1 for categories 1,2,3,4; by 2 for
category 5,6, and 3 for category 7).
Audience Evaluation:
The audience interaction evaluation will require people from the
audience to interact with your robot and to fill out an evaluation
form subsequently (which they can drop off in a marked area).
Different from the technical evaluation, the audience evaluation is
focused on interaction aspects (rather than including architectural
aspects):
- appropriateness of robot for task (did not fit, fit partly, fit well)
- robot performance (did not work, worked partly, worked well)
- effectiveness of interaction (uneffective, partly effective,
effective)
- ease of interaction (difficult, fair, easy)
- complexity of interaction (low, medium, high)
- robustness of interaction (brittle, fairly robust, highly robust)
- level of entertainment (tedious, fair, fun)
- level of novelty of interaction (none, some novel aspects, many
novel aspects)
Numeric scores for each criterion will be computed as in the technical
evaluation, but different from the technical evaluation the average scores
of each criterion across categories will be used to determine the winner
(i.e., no category winners will be determined). It is up to you which
task/category you would like people to perform, just make sure that you get
at least 10 people so that we have enough data points for computing the
averages.
Environment:
The regular audience at AAAI is becoming increasingly habituated to robots
wandering around, and tends to not pay them much attention any more. You
will get individuals coming up to your robots and "kicking the tires" a bit
(hopefully figuratively, but sometimes literally). You will want to make
sure your robot can grab attention. Visitors to the conference will tend
to crowd the robot in groups as they come through (particularly during
breaks in the conference talks), so your robot ought to be able to handle a
press of people and deal with the situation robustly. For mobile and
wandering robots, try to keep some distance so that it doesn't look like
you are shepherding or controlling the robot. Do make sure to have somebody
on hand to talk to the audience and answer questions (and to step in if
anything goes wrong!), but it is important that your entry be able to stand
on its own without need for explanation.
Sponsors:
To be determined...
AAAI Robot
Competition Home Page
Last updated 04/23/07