Course Description

This advanced course brings together methods from Machine Learning, Computer Vision, Robotics, and Human-Computer Interaction to enable interactive machines to perceive and act in a variety of environments. Part of the course will examine approaches for perception with different sensing devices and algorithms; the other part will focus on methods for decision making and applied machine learning for control. The course is a combination of lectures, state-of-the-art reading, presentations and discussions, programming assignments, and a final team project.

Instructor: Marynel Vázquez (marynel.vazquez at
T.F.: Debasmita Ghose (debasmita.ghose at, Sydney Thompson (sydney.thompson at

Class Hours: Mondays & Wednesdays, 1:00pm - 2:15pm

Office Hours:
Tuesdays, 3:00pm-4:00pm ET (Sydney)
Thursdays, 11:00pm-12:00am ET (Debasmita)

Canvas Link:

Learning Objectives

At the end of this course, students will have gained an understanding of:

  • the challenges involved in building autonomous, interactive systems like robots;
  • the limitations and advantages of various sensing techniques; and
  • well-established frameworks for sequential decision making.

The assignments will teach students about the Robot Operating System (ROS). Students will also be able to demonstrate their ability to work in a team and to communicate scientific content to a peer audience.


CPSC 201, CPSC 202, and CPSC 470 or 570.

Understanding of probability, differential calculus, and linear algebra is expected for this course. Programming assignments require proficiency in Python and high-level familiarity with C++. Students who do not fit this profile may be allowed to enroll with the permission of the instructor.


The following topics will be covered in the course:

  • Perception
    • Sensor Design Choices for Interactive Systems
    • Projective Geometry for Computer Vision
    • Bayesian Filtering (e.g., for Tracking)
    • Deep Learning for Function Approximation of Complex Physical and Social Phenomena
  • Decision Making
    • Markov Decision Processes
    • Imitation Learning
    • (Inverse) Reinforcement Learning

See the Schedule for more details.


The course grade will be based on:

  • Student Presentations (10%). Students will get the opportunity to present recent papers in class (those enrolled in CPSC 559 will present more times than those in 459). Presentations will be graded based on clarity, how well students answer questions from the rest of the class, and how well they relate papers to other course material.

  • Programming Assignments (40%). There will be 5 individual programming assignments (with extra questions for students enrolled in CPSC 559). Students will have 3 late days to be used as needed to extend the deadline of an assignment for 1, 2, and up to 3 days after the original deadline. Not all late days need to be used on the same assignment.

  • Final Project (40%). Students will work in groups on a final project creating an interactive system. Project grading will be based on 4 deliverables: project proposal (5%), project milestone (10%), project presentation (10%), and final report and supplementary material (10% + 5%).

  • Video quizzes (5%). For some classes, students will be assigned to watch pre-recorded lectures by the instructor and to complete quizzes about the material.

  • Participation (5%). Being engaged and asking questions will be rewarded.

Note on CPSC 559 vs. CPSC 459: This course is double-numbered, which means that it can be taken as either 559 or 459. Graduate students must take it as 559; undergraduates must take it as 559 if they are enrolled in the combined MS-BS program. Those taking it as 559 must present extra papers in class and complete extra questions in the programming assignments.

Policy on late days: The late days are only valid for the programming assignments. After the 3 late days, the assignments will be penalized with -50% of the grade for every 24h after the deadline. No other late assignments will be permitted without a Dean’s excuse.


The class sessions will be divided into lectures and student presentations. Recent technical papers will be used as the main text reference, along with a few chapters of the books:

Past Editions of this Course