6.S063 Engineering Interactive Technologies (fall 2017)

PSet1: Build a Multi-Touch Pad

step 1/2: due Friday (September 22, 2017) at 11:59pm, upload here
step 3/4: due Friday (October 13, 2017) at 11:59pm, upload here

  • Sept. 22: upload to Gradebook
    • a short video showing your laser cut / bend casing (step 1)
    • a short video showing how you turn on/off LEDs with the switch (step 2)
    • use your mobile phone to capture the video, no fancy editing please
  • Oct. 13: upload to Gradebook:
    • a short video showing results from finger tracking (step 3)
    • a short video showing your application and the gestures you implemented (step 4)
    • the code for step 3 + step 4
  • (TBD): give a live demo of your touch pad to your TA, sign up for a demo slot at (doodle, to come)
  • grading criteria are mentioned below after each step

  • post on piazza
  • go to a TA office hour for building/coding oriented questions (tbd)
  • go to an instructor office hour for logistical questions and grading (mondays, 4-5pm, 32-211)
  • do not send us emails

Build a multi-touch pad using the principles of FTIR (Frustrated Total Internal Reflection).
The multi-touch pad has all the components required later for your group project: custom laser-cut parts, custom electronics, and custom code.
We hope by doing this problem set and getting experience with end-to-end prototyping, you will be able to better scope your group project.
We will provide you with all the materials you need, you will get from us:
  • a camera
  • an acrylic sheet
  • USB connector + cover
  • a switch
  • LEDs
  • resistors

In addition we give you the following components, which help you prototype your touchpad but are not part of the final prototype:

  • microcontroller
  • breadboard
  • prototyping cables
  • alligator clips

To build the multi-touch pad you need access to the following equipment:
  • a laser cutter
  • a heat gun or strip heater for bending the sheet
  • a soldering iron
  • Step 1: Laser Cutting (25 points total)
    • Finished all steps: 9 points
    • Clean bend with no heat bubbles: 4 points
    • Enough space for camera attachment below LEDs: 4 points
    • Edges have a clean cut: 2 points
    • No burn marks on pad: 2 points
    • Snug hole for LEDs: 2 points
    • Touchpad stands stable and does not wiggle: 2 points
  • Step 2: Electronics (25 points total)
    • Finished all steps: 8 points
    • LEDs turn on: 4 points
    • Correct resistor used: 3 points
    • Switch works: 3 points
    • USB cable works: 3 points
    • Circuit is correct: 2 points
    • USB cable is twisted: 1 point
    • Soldered connections look good: 1 point
  • Step 3: Computer vision (25 points total)
    • (TBD)
  • Step 4: Gesture recognition (25 points total)
    • (TBD)

Step 1: Laser cut and bend the casing (25%, due Sept. 22)
  • create the .svg file for the acrylic sheet in a 2D drawing program
    • outline of touch pad:
      • height: 18cm
      • tracking area on top: 14cm fold to fold x 12cm wide
      • add thin engraved lines at the locations where you will bend your pad
      • do not change any of these dimensions since camera image will not be in focus anymore and you will have tracking problems later on
    • horizontal line with electrical components:
      • leave 6cm of space from the bottom (the camera will be glued underneath the LEDs)
      • 1 hole for the switch
      • 1 hole for the resistor
      • 7 holes for the LEDs
      • 2 holes for wires near the resistor
    • customize your touchpad on the opposite side:
      • on the side that does not hold the LEDs, add a custom cut-out or engraving
      • this can be anything you like, e.g. your name, an image, the MIT logo, be creative
  • use strip heater or heat gun to bend the acrylic sheet
    • warm up equally from both sides to make sure the sheet is thouroughly heated
    • use an alignment aid to make sure you get a 90 degree bend
Grading of step 1
If you finished this part completely, you will get full points except when one of these is the case:
  • measurement errors: holes for components are too large / small, wrong height, not enough space for camera etc.
  • laser cutter problems: edges are not clear cut or lots of burning marks on the pad etc.
  • bending problems: touchpad does not stand stable (bends are not exactly 90 degree), visible heat bubbles at the corners due to overheating during bending etc.
  • see rubric (above).

Step 2: Getting the electronics to work (25%, due Sept. 22)
  • Add the 7 LEDs, the right resistor, and the switch to your prototype and solder them
    • prototype the LED setup first using the breadboard and the microcontroller
    • test 1 LED with resistor on breadboard first, lights up? good.
    • extend your circuit to 7 LEDs, adjust resistor, all lights up? good
    • remove LEDs from breadboard and plug them into your acrylic sheet (all long legs on one side)
    • cross LED legs to connect them to each other
    • test with the Microcontroller's 3V and GND if they still all light up
    • if yes, solder them
    • add switch to the setup, solder the switch to the LEDs
    • test with Microcontroller's 3V and GND, does the switch work?
  • create your USB cable
    • create a twisted cable using two cables and a drill
    • solder the two cable ends to the outer two pins of the USB connector. See this image for which pins to solder to. In the image, the red pin is positive, and black pin is negative.
    • assemble the USB connector and the casing around the cables
    • solder the end of the cables to your touch pad
    • connect the USB cable to your laptop, do the LEDs light up (make sure the switch is on ;))?
Grading of step 2
If you finished this part completely, you will get full points except when one of these is the case:
  • LEDs: the LEDs don't turn on, the contact is shaky, or the LEDs are very dim, which means you used the wrong resistor
  • soldering: you used an unsual high amount of solder to get your connections done
  • switch: the switch doesn't work
  • USB cable: your USB cable does not work, doesn't have the casing on top, or is not twisted
  • see rubric (above).

Step 3: Computer vision finger tracking (25%, due Oct. 13)
  • tape the camera onto your touchpad below the LEDs
  • final goal:
    • display the live camera image showing the coordinates of the finger tips drawn into the image as (x,y) where the finger tips are
    • your interface also needs to have sliders to threshold the image for HSV min/max
  • tips how to get there:
    • check if the camera works: display the raw camera image
    • convert the raw camera image to HSV color space
    • threshold for your LED color (outcome is a black/white image: white is where you color is, black everything else)
      • it helps to create some sliders for hmin, hmax, smin, smax, vmin, vmax so you can try around until you find good values since light conditions change
    • find contours in threshold image (use openCV draw functions to draw them into the image for debugging)
    • only keep the contours that are the size of a finger tip (discard too small or too large contours, again having sliders for this helps, i.e. minArea and maxArea)
    • find contour center to have touch coordinates
      • fit a circle around contour (fitElipse ())
      • get center of circle for touch coordinate
    • draw (x,y) coordinates of each finger tip onto the image
Grading of step 3
If you finished this part completely, you will get full points except when one of these is the case:
  • finger tips: not recognized or recognition shaky
  • code quality: your code is a mess
  • threshold sliders: your interface has no sliders for thresholding but instead only uses hard coded values
  • coordinates: the coordinates are not drawn at the finger tips but in a fixed location
  • see rubric (above).

Step 4: Custom application with gesture recognition (25%, due Oct. 13)
  • write your own application that uses the touchpad
  • you can come up with any application you like and we are excited to see what you come up with!
  • a boring one, but one that's easy to do is a drawing application:
    • e.g. finger tip size = thickness of brush
    • e.g. finger number + short tip = color selection (1 finger yellow, 2 finger blue)
    • e.g. scaling the drawn image = dragging with two fingers
Grading of step 4
  • you lose points if your application has less than 3 recognized gestures (pinch with two fingers, tap with one finger, use finger width as input etc.) and less than 3 different functions (e.g. select color, adjust line thickness, scale drawing)
  • we give up to 5 bonus points that can make up for some of the other lost points in step 1-3 for especially creative and comprehensive solution
  • see rubric (above).