Course Description

This advanced course brings together methods from Machine Learning, Computer Vision, Robotics, and Human-Computer Interaction to enable interactive machines to perceive and act in a variety of environments. Part of the course will examine approaches for perception with different sensing devices and algorithms; the other part will focus on methods for decision making and applied machine learning for control. The course is a combination of lectures, state-of-the-art reading, presentations and discussions, programming assignments, and a final team project.

Instructor: Marynel Vázquez (marynel.vazquez at
T.F.: Tim Adamson (tim.adamson at

Class Hours: Mondays & Wednesdays, 1:00pm - 2:15pm
Class Location: WTS A51

Office Hours:

  • Mondays, 2:30pm - 3:30pm (Marynel, AKW 402)
  • Tuesdays, 9:00am - 10:00am (Tim, AKW 411)
  • Friday, 3:00pm - 4:00pm (Tim, AKW 411)

Canvas Link:

Learning Objectives

At the end of this course, students will have gained an understanding of:

  • the challenges involved in building autonomous, interactive systems like robots;
  • the limitations and advantages of various sensing techniques; and
  • well-established frameworks for sequential decision making.

The assignments will teach students about the Robot Operating System (ROS), and provide practical experience with a robotic platform. Students will also be able to demonstrate their ability to work in a team and to communicate scientific content to a peer audience.


CPSC 201, CPSC 202, and CPSC 470 or 570.

Understanding of probability, differential calculus, and linear algebra is expected for this course. Programming assignments require proficiency in Python and high-level familiarity with C++. Students who do not fit this profile may be allowed to enroll with the permission of the instructor.


The following topics will be covered in the course:

  • Perception
    • Sensor Design Choices for Interactive Systems
    • Projective Geometry for Computer Vision
    • Bayesian Filtering (e.g., for Tracking)
    • Visual Pose Estimation
    • Device Interaction Detection
    • Deep Learning for Function Approximation of Complex Physical and Social Phenomena
  • Decision Making
    • Markov Decision Processes
    • Imitation Learning
    • (Inverse) Reinforcement Learning

See the Schedule for more details.


The course grade will be based on:

  • Student Presentations (15%). Students will get the opportunity to present recent papers in class (those enrolled in CPSC 559 will present more times than those in 459). Presentations will be graded based on clarity, how well students answer questions from the rest of the class, and how well they relate papers to other course material.

  • Programming Assignments (40%). There will be 5 individual programming assignments (with extra questions for students enrolled in CPSC 559). Students will have 3 late days to be used as needed to extend the deadline of an assignment for 1, 2, and up to 3 days after the original deadline. Not all late days need to be used on the same assignment.

  • Final Project (40%). Students will work in groups on a final project creating an interactive system. Project grading will be based on 4 deliverables: project proposal (5%), project milestone (10%), project presentation (10%), and final report and supplementary material (10% + 5%).

  • Participation (5%). Being engaged and asking questions will be rewarded.

Note on CPSC 559 vs. CPSC 459: This course is double-numbered, which means that it can be taken as either 559 or 459. Graduate students must take it as 559; undergraduates must take it as 559 if they are enrolled in the combined MS-BS program. Those taking it as 559 must present extra papers in class and complete extra questions in the programming assignments.

Policy on late days: The late days are only valid for the programming assignments. No other late assignments will be permitted without a Dean’s excuse.

Students need to email the course instructor (marynel.vazquez at to request using late days for a programming assignment before it is due. Otherwise, they will not be able to submit the assignment past the deadline. After the 3 late days, the assignments will be penalized with -50% of the grade for every 24h after the deadline.


The class sessions will be divided into lectures and student presentations. Recent technical papers will be used as the main text reference, along with a few chapters of the books:


This course is made possible through the generous support of Yale’s Provost Office, IT at Yale, and Google Cloud.