Tactile Human Interface Gesture Control System (THIG Control System)
Materials List:
Ipac2 Board (you could also mod an old keyboard)
Xbox 360 controller (with at least one working analogue control)
Webcam (got mine from eBay and I lucked out as the model I got made the job a lot easier)
Pre-Exposed Film (any size it’s going to be cut down to around 1x1cm)
A pair of gloves (heavy duty is best as it gives a stronger platform to work from)
Soldering Iron and De-soldering pump
Solder/Flux
Clips or a vice (this is as helpful as having a third hand)
Rhodium Reed Switches
Small magnets
Wire (lots of wire and a few different colours)
Tape (Electrical/ Duct/masking any really)
Infrared LEDs (best get 10 or so)
CR2032 Button cell batteries and the holders/ Clips/ or a 9V and the right resistors for your LEDs
A small breadboard (the circuit development kind)
Plastic tube (big enough to house your wire, no thicker than your finger)
A decent knife or cutting device
Wire clippers
Torx 8 security screwdriver (for Xbox controller)
standard screw drivers
Spare material of decent strength
Super glue and a needle and thread
Computer (Windows or Linex)
A copy of ‘Webcam Whiteboard’ Software from either http://www.webcam-whiteboard.com/ or http://www.brothersoft.com/webcam-whiteboard-352167.html
Overveiw
With the basics of this design it’s possible to make many variations, indeed coupled with a motion control system, (such as Kinects) and some coding ability it wouldn’t take much to make this viable for consoles as well as PC.
The systems design has evolved from a game design, I was wanting to show how as we add move further and further from controller based HCI systems we lose various amounts of control, I was also looking at the use of Augmented Reality gaming as well as how a device and be used for game play (my first idea used the iPhone and its features to inspire game play rather than relying on the touch screen for control.) This idea then became a hardware design, as I was challenged to create a way of showing my ideas at my Colleges degree show, I first looked re-mapping and modding a keyboard so I could use pressure pads to control things on screen, from this I bought an iPac2 board this is a device that is used in arcade machines, and when plugged into a PC or Mac via USB it acts like a keyboard, it also overrides the USB block on the triggering of more than 5 button presses on a peripheral.
For a long time my idea was a music based system I was going to have a series of platforms with pressure pads that would trigger loops and samples, this idea then became a two player system
where one player explored a maze, and a second would score with music their journey, using the music to warn of impending danger. This idea stuck for a long time, and then whilst looking into gesture control, I decided I would abandon this idea and concentrate entirely on the problem with motion tracking/ gesture control games systems.
This system is aimed at gamers of the first person kind, or really any game that uses the mouse/keyboard combination to control an on screen avatar, what the system does is to take the keyboard and mouse away and give the player functionality within their own hands, it also will allow them to remap controls to different movements, and fingers depending on how natural it feels to them.
The software that allows for the mouse control to become gesture control is the ‘Webcam Whiteboard’ software this software tracks Infrared light and allows for users to use any peripheral with IR LEDs and a slightly modified webcam, the software then tracks a single IR marker, a second will act as the left click and three would act as the right click.
As I said before there is a large amount of room for adaption of design, I tried to source all my IR LEDs from remote controls that were no longer being used, this presented a problem when the controller was broken or out of battery so I created a breadboard circuit to test this with an IR photo-resistor. The Rhodium Reed switches could be substituted for any method of switch you care to use, and in fact in a couple of places I am looking to recycle the Micro-switches from an Xbox 360 controller.
As a games system this could have around 24 button press combinations allowing the player to remap almost any shortcut to a single movement.
It would be a challenge but not imposable to forgo the game control side of the system and instead create a keyboard interface using one or both hands, by dividing up the keyboard in the same way it is divided when we learn to type, assigning rows to fingers, this would of course mean learning a new way to type as well as working out a system of keyboardless typing that was intuitive, this may well have to incorporate the gesture control system as well.
I see the outcome of the project as the foundations this is why I have chosen to release instructions on its construction as an Open source hardware project, I hope by releasing its designs to get feedback on the construction as well as see inventive ways in which my designs are adapted, I hope to then use this to influence my own ideas in future.
Ultimately my goal would be to take this idea to a hardware R and D company as a part of my future employment and see where my ideas can be implemented in the future of control on gaming and HCI.
This project also requires the modding or hacking of at least one piece of hardware, and a large amount of soldering and a good knowledge of circuits, it also requires a large amount of patience and time, I had a few months to draw up my designs and in the end I still struggled for time.
Part 1
Right the first and most important step is to create a simple circuit with 3 IR LEDs (its worth pointing out at this point that it is possible to use coloured LED’s (I built one system that used red LEDs as its primary, this works well in environments where you have no control over the lighting) try and get all the LEDs facing in the same direction and also set them so you can move them around, at first set them all as close together as possible.
Second, take your webcam and open it up (mine was made easier as the lens was screwed in and this allowed me to simply unscrew it, for whatever model you have look over the internet for instructions, but the basic idea it to remove all the lens part from the main body. You will then see a small piece of glass in the back, this needs to be removed, mine was held in with small pieces of plastic that cut away easily, once removed, reassemble and test the webcam (just to make sure it still works as id you broken it somehow there is no point going on with this stage.) Disassemble the webcam again and cut a piece of you exposed film to fit the space, if like mine the filter you removed was just painted with a red(ish) colour that can be scratched off then you can clean it off with white spirit and then place the exposed film in front of this and then re-stick the filter in place.
Now install the ‘Webcam Whiteboard’ software and set it up as directed, using your temporary circuit take/make note, of the distances between the LEDs if the left and right click don’t work right away, then move the LEDs apart until they work correctly (this shouldn’t be an issue at all its just best to check this parameter before you proceed.)
With this section finished we move onto Part 2a.
In this part we will test the iPac2/ Keyboard part of the design if you are using a keyboard (this is the cheaper but less elegant (for lack of a better term) solution) you need to look at a keyboard modification tutorial, simply put each of the contacts have to be rewired and you also need to make note of each key using a key-logger (or an open Word Doc), this is where tape (masking ideally) is the best option you also need to assign a colour to one set of keys that will become the ‘negative’ they don’t have a charge but it makes the wiring diagrams easier to understand later.
If your using an iPac board this job becomes a lot easier simply attach a longish piece of wire to the GND terminals (one on each side, “don’t cross the streams” use the A GND with the A terminals and the same for B) then go along and test each terminal against the table online, you can also download the program to re-assign the keys to each terminal this is useful later so it’s worth getting now.
Part 2b. (optional)
This is the time to deconstruct your Xbox controller and extract the analogue controls again there are plenty of tutorials on the deconstruction of a controller and the extraction of each part, try and keep all the parts that you can (I lost the rotational controllers from the triggers due to brute force and large amount of ignorance) but the mico-switches are useful so savage as much as you can. The other thing to be aware of is to check each component as you remove it from the console on your keyboard or iPac I created a temporary circuit on a breadboard for testing. Just to make sure you burn out any of the components. Once the harvest is complete set aside all the components labelling any terminals and important information as you go along.
Once you have the mouse and keyboard parts sorted (and controller deconstructed), it’s time to get down the bigger parts of the design in:
Part 3.
I started with the mouse as this is ultimately the hardest part of the design (sort of) take a look at the wiring diagrams for the IR LED interface and assemble them as directed tried to keep as much of the circuit on the glove as I could it’s also worth drawing up detailed dimensions of your hand, both open palm and closed fist (this allows for more precise design and this should untimely make the design more ergonomic)
The first thing I did was to make the IR LED tubes and connect those to the series circuits, in the end I opted for a series circuit as obviously it allowed me to use one battery, it took me some time to work out how to keep one LED on all the time and have the other two worked by other switches, over all there are the 3 switches and they do work the same in the end, but the first Rhodium Reed Switch and its magnet should be set very close to the surface so that the magnetic field is strong enough to make it hard to break the connection, (not so hard that its impossible to break the connection, but as this is the main tracked LED its important that the connection isn’t lost accidently) the other switches need to be set under the fabric and maybe even under a second layer so that the second and third switch connections can be made and broken easily.
The diagram should show that the magnet goes on the end of the IR LED tube and the Rhodium Reed Switch (RRS) goes close to the knuckle. This design was born out of necessity, as the gloves I bought have the thumb and first two fingers missing, I needed to attach a switch that allowed for that (I intend to cover those fingers with the tips from a second glove at some point but for now that’s a part of the design as it allows the user to pick things up and manipulate objects like a glass without the need to remove the glasses. I may use rubber tips at a later date.
So once these 3 tubes are attached test it with the ‘Webcam Whiteboard’ software, one light should be the primary track and add the second and that should left click and three should give you a right click, if this doesn’t work re-check the LEDs with the IR LED checking circuit, if they are all working then it’s a software set up issue.
One other thing to think about at this point the right hand glove (assuming you have stuck with the ‘traditional’ set up of right hand mouse control) now has one spare finger with no use, and also the under side of each finger can also be used as we will see in:
Part 4.
For the most basic version of the left hand glove you can wire up 4 RRS’s on each finger allowing for 16 moves, the magnet goes in the thumb, each RRS (3 on the under side I will get to the top in moment this will account for 12 of the possible 16 contacts) goes as close to the middle of each finger segment as is possible, but first do make sure that while wearing the gloves you are dexterous enough to touch each finger segment with the tip of your thumb, this will also allow you to mark better where to place the RSS’s for your own system. For the top of your fingers, reuse the IR LED tube design, just remove all of the circuitry and keep the magnet. So you now have 4 independent connections and 12 that require you to make single connections, with the iPac you can have a shift line, and this is what allows you to make 24 connections from 13, for instance, I have it set up so if you extend your index finger it will engage the Shift level and the 12 connection on the under side of you finger all have a 2nd function, allowing for 24. You can also set up another finger for the Ctrl button allowing you to map your own hot keys, this will then require the user to make two movements to trigger either a second level connection or make three movements to make a second level shortcut or macro, example using the system to control Word for instance you might map the C button to the index finger on the bottom (closest to the knuckle) segment and V to the segment above, then have Ctrl on the extended little finger so to cop and paste you would extend you little finger, then touch the bottom segment of the index finger to copy some text, then the segment above to paste. But say you wanted to print (Ctrl+ P) and P is on the shift row on the iPac, and on the tip of the middle finger, to make the move to print, you would extend both your index and little finger and then with your thumb touch the tip of your middle finger.
I also added a row of magnets on the palm of my glove at two points where I can touch them with the tips of each finger where I have one or more of each finger extended, this is quite a hand exercise, and some of this might be hard for some people, I am lucky in that my left hand is extremely dexterous as I have been playing guitar for around 10 years. And this is again a part of the reason for why I decided to write this as instructional step to step because it gives the freedom to create a system that is exacting to you own specifications.
This is the basic set up for simple system, however I have gone one step further, I have created a Velcro ‘Wii’ style nun-chuck from the skeleton of an Xbox 360 controller, this was inspired by some tutorials I saw on making a second thumb/analogue stick for a Song PSP this idea was to rewire the D-Pad by re-soldering the contacts to an analogue control. While researching first hand the internal components of an Xbox 360 controller, I found a treasure trove of components, so using a plastic rod and then using Blu-tack and fabric made an ergonomic nun-chuck fit for my hand (although I used Blu-tack to allow for other users to be able to slightly reshape the nun-chuck).
The basic design of this is simple; the analogue stick sits on the top on the nun-chuck and the micro-switches are set where each finger sits on the nun-chuck, the wires then run down the rod to the bottom and has a fairly long chord because if I have time I intend to make a sleeve to house the iPac2 board and then have a space for the nun-chuck but that more of a would like than a need. Giving the option of duel functionality’s with one system.
The complete setup
I made a mount so the webcam could be mounted with a bird’s eye view, my webcam also have a little blue light that interfered with the set up, and this was beaten with the application of electrical tape.
Plugging in the left hand Control, open the ipac2 setting and assign the Keystrokes you require.
Open the Webcam Whiteboard software, set up for Colour and Finger tracking, adjust all settings until they are good, calibrate, and start.
