Testing time!

Its been a bit of a scary week in all honest in regard to this project. Plenty of realisations, compromise but in hand with this came a lot of learning.

I noticed my pulse sensor was working a bit oddly when max reported values of only 50, 54, 60, 66, 75, and 85 (some more onwards) seemingly the gaps were becoming further apart. I tried power supplys etc, trying to change almost every component, the arduino itself got replaced etc etc. No luck.

After a LOT of research and a lot of thinking i starting looking at other options including the Polar Heart Rate Monitor Interface (HRMI) board by Dan Julio. I got in contact with him and we talked about how the HRMI transmits data through ASCII code to other programming environments. We talked about other solutions to getting a stable heart rate into an Arduino and he also mentioned an inductance device that also picks up the the transmissions for a Polar strap sensor and outputs as a clean 3V signal.

He went on to mention how serial connections between the Arduino and environments (such as PD and MAX) experience lag and bad averaging. This seems to be what is happening here. Because of lag times the board cannot physically sample many more then around 10 samples per second. This makes sense as at 60bpm i was experiencing 10 different values but at 85bpm only around 8. Unfortunately the higher the heart rate, the higher the error which of course makes things difficult.

SO...

I am going to get a HRMI and try and get it working serially at first then attempt to interface it with Max/MSP via either logic level serial or I2C protocol. If Max will not accept such messages then i will have to make do with some averaging functions and use the work i already have. If this is the case i don't see a huge problem as the heart beat will still meander around 60-75 bpm until electricution events where it will jump up very quickly. The issue may be that the reported values at the higher heart rates will jump quite erratically between 90-120bpm. If this is the case i may change the way i report the visual, particually in regard to colour, where it works on a three tear scale, blue for resting (50-65bpm), yellow for active (70-85bpm), and red for stimulated (90-120bpm)

So im going to have a bit of a think i reckon before spending £50 on the HRMI and £15 on the polar strap. Up side is the devices will be much more stable in terms of reporting the heart rate (no need to signal condition etc like i'm having to do at the moment), and it is better attached to the performer. The current finger attached device has a tendency to react badly to lighting level changes.

Modul8 and Midi control

This week i've been experimenting with Modul8 and how triggering video transformation effects can add value to the visuals tied into the sensors.

The effects within the program are fairly easy to use and would work well alongside the respiratory sensor for example, which could pull apart the video feed (grid and distort in Modul8) in respect to the expansion of the chest through breathing.

I have found the best way to do this is to use Midi to control the relevant parameters in Modul8. This is easy when using a midi keyboard as the program maps physical midi signals automatically. It finds it difficult however to recognise virtual midi ports such as those set up by Max.

Scouring forums and the like it seems it is either possible through Pure data using Midipipes but i don't really want to start using another programming interface as this adds more potential problems. There are hints that Abelton live may help by sending midi through API. I can then use Max for live to open my patches and hopefully establish midi control that way.

If not i may have to use keyboard triggers instead, but although these can be automated in max, they cannot be assigned to faders and sliders in Modul8, only momentary switches

Sensor calibration

The weakest link of my system so far is the translation of data from the BVP sensor. I have found that there is only a small margin of error to capture a true pulse signal above thermal noise levels and nominal blood flow within the finger.

Left uncalibrated Max will either miss 'pulse' events or perceive the occasional burst of noise as a trigger. As the visual works on an average bpm (using a tap tempo), changing its colour in respect to a rising or falling average. If max recognises the data as two very close pulse events then this will be interpreted as a rapid rise in bpm and therefore colour.

I can see two ways of solving this. Firstly is to use the max window to observe directly what the data is doing. Within this data it is much easier to see what values to set thresholds at to provide the most accurate translation. This worked well and meant that there were only minimal instances where beats were missed (there tended not to be any unwanted events in close succession).

I now need to make some kind of averaging stage where if a beat is missed it will not affect the bpm. At the moment the only way i can think of doing this would be to compare the last bpm result with the present one and only accepting changes which are within a particular threshold. It is unlikely your heart will slow by half within a single heartbeat so this should work. To accommodate for higher pulse rates its important that the threshold is a proportion of the previous bpm value and not a constant number.

Functions could be similar to below where x is the previous bpm and y is the current value:

Accept y if - y >= (x - 1/3x) so at 60bpm y would be accepted at 40bpm or above

Reject y if - y < (x - 1/3x) so at 60bpm x would remain at 60bpm if y is at 39bpm or under

This allows higher heart-rates to accept more sudden drops, which is likely after sudden startling etc.

i.e. 120bpm can safely drop to 80bpm without being rejected. Just have to figure out how this would work within Max.

Max interface

Kind of works!

Had to do a lot of scaling within Max to get this to work with any degree of stability. Now with a little calibration and what a 'settling in period' (where the sensor sits on the finger for a minute to stabilise and detect) it works quite well.

The video above displays a very simple visual that represents the wearers heart-rate. The sensor seems to provide minimal lag.

To further improve its performance in terms of delivery of data i tried adding gain to the signal through an op amp but i experienced strange results. It may be that i need to use better quality components. However, as mentioned in previous posts i am keep to keep signal processing stages to a minimum as i am not using medical grade equipment and there is a good chance i will introduce even more noise to the system. For now it is stable enough to work with until another solution arises.

Mock up testing of Sensors

Several things bugged me.

1. Do the sensors work if powered without the base device

2. Will my extension connectors work i.e. soldered correctly, no significant leaks

3. What signals will return from the +sig and -sig outputs

4. What scale are the return voltages

I used a battery voltage supply of 12V as it was what i had lying around and guessed a swing of 6V with respect to ground wasn't too much more then 5V.... I then powered the sensors and just used a multimeter to read what returned from sensors. The respiratory sensor uses stretch sensors to change a resistance, therefore it works with relative voltage changes when the band is expanded. As it requires no calibration it works pretty well.

The BVP device looks as if it needs a little more calibration and care with what supply voltages it is given. The sensor returned voltages that did move between 120mV and 160mV in a pattern that did look as if it followed a typical heart rate. This needs clarification by using faster signal sampling. This would be provided by the Arduino. It was encouraging however to see that if you removed the sensor off the finger it stopped returning voltages out its signal channels.

Scale

Is a relative change of 20-40mV from the BVP enough to work with bearing in mind the Arduino's input pin resolution. The maths (5/1023) suggests a 4mV change can be detected however when you begin working within Max small changes like this are unstable and scaling them doesn't tend to yield great results.

It would be favourable to remove the need for any gain stages after signals are returned from the sensor however this may be inevitable. This would open up a whole new chapter of signal conditioning and require low noise op amps and a cleaner power supply. We will see!

Sensor Follow up

Extending the sensor connectors

Finally solved using some pretty tight soldering, insulation tape, jumper wires and several clamps.

Powering the Nexus Sensors

My last post now makes me laugh quite a bit with some of the solutions i was offering! After a little more research into potential difference (PD) i tried a few things.

Firstly i tried an op amp to try and invert 0v to 5v to give a -5v to 0 output. This doesn't work of course because you have to power the op amp with a PD enough thats high enough to swing between -5v and 5 volts which is of course 10V. I tried to do this using a few batteries. I then found out that i am not using a common earth by using two different voltage supplies. So i needed to use the Arduino to power everything. Luckily i later found out that the microcontroller has the facility to use an external power source. Using this knowledge i managed to provide a 0V, 5V, and a 14V supply from the device. Using a potential divider and i can pull the 14V level to 10V which give me a 10V PD. I then will use 5V as a virtual earth, 0V then translates to -5V, and 10V to 5V. This gives me the power and reference levels i need to power the Nexus sensors whilst providing a common earth.

However, when i get return signals from the BVP device (in particular because it requires calibration) it will be in respect to 5V as a virtual ground. As the Arduino only samples analog inputs from 0 to 5V at a resolution of 1023, i will have to again have to employ another signal conditioning stage to work within these levels. Of course with small voltage work it is important to try to avoid as many potential noise producing phases as possible. Hopefully though, if i am only using potential dividers the only real noise problems (providing good component placement and decent interconnect) will be thermal noise from resistors.

Next hurdle - The sensor

I'm happy with the visualisation i have now (will upload an example soon) and now its time to fully tackle the sensor connectivity problem.

The first stage, which i've already overcome, regarded the way i can use the sensor and manipulate it without destroying a £300 piece of hardware. After a little digging around i found the type of connector the nexus used for its auxiliary input, a LEMO 0B 5-pin. I found the tech sheets on it and after A LOT of looking around found a company which makes a female equivalent.

Next few problems -

Firstly the female connector needs extending so that i can work with the signals and power it correctly. At the moment the adaptor is extremely narrow. This maybe solvable using some jumper wires and some very delicate soldering.

Secondly the sensor needs powering. For the nexus tech sheets it shows that the auxiliary input to the amplification pack (power etc) provides +5, -5v, and 0v to the sensor. I am presuming here that the sensors require powering and that they are not passive. The BVP sensor (heart-rate monitor) works with Infrared so i guess they need powering. The issue is of course how much current to provide it with. From the tech sheets this suggests 10mA - clearly i don't want to damage the components. I need then to be able to provide a stable power supply. The Arduino only provides 5v and 0v so this means i may have to use an inverting unity gain amplifier to change the polarity of the signal. After this i should be able to pull the current levels down fairly easily if too high.

The third issue of course concerns the way we are able to detect if the sensor is actually doing anything. I have a digital multimeter but this will not be precise enough to figure out exactly what is going on. The final two pins of the LEMO connector provide a positive and negative signal return. Do i need to use both these? Does the base unit augment the signal using a difference amplifier using -sig and +sig as inputs? this would make sense however its a presumption.

Essentially i need an oscilloscope, a bit of solder, a couple of simple op amps for the power supply, and some patience.

EEG, Biodata performance examples

I thought it best to list a few examples of similar work to help describe the direction my current project is going in.

INsideOut - Claudia Robles - Germany 2009

The performer, who is surrounded by sound and images, interacts with them using an EEG (electroencephalogram) interface, which measures the performer’s brain activity. Those sounds and images -already stored in the computer- are modified consequently by the brain data via MAX/MSP-Jitter. Hence, the performer determines how those combinations will be revealed to the audience. Images are projected to a screen and also onto the performer, while sounds are projected in surround.

Empathizer - RobotLab - 2000

Linking human brain and robot

The installation 'empathizer' gave visitors the opportunity to experience an extreme interconnection between man and machine. The experimental environment consists of two industrial robots and a brain interface by which the visitors can connect with the machines.

Brain voltages are measured and analyzed. Then the data is forwarded to the robot control units. The image of the brain signals is projected on the robots. They give a representation of the data through their movements and always perform new and individual configurations - interaction without physical action.

Terrain01 - Ulrike Gabriel - 1993

The interactive AL-installation consists of cybernetic vehicles. the movements of the active 'life' depends entirely on the intensity of the light being projected onto the colony of robots. A brainwave sensor, placed on the head of the interactant, measures his or her brain activity, which is then sent to the system and controls, in turn, governs the intensity of the projected light. The more intense or erratic the viewer's brain activity, the less light strikes the robots and the more apathetic the behavior of the colony; or the weaker the brain impulses (the more relaxed the viewer), the more chaotic the movements of the robot colony become ...

Thought Conductor - Bruce Gilchrist et al. 1997-2001 various versions

Thought Conductor was inspired by the notion: what could have occupied the mind of the musician David Tudor when he performed John Cage's silent composition 4'33"? In reality he would have been preoccupied with keeping track of time, but this led to the idea of a musical performance where musicians are responding not to an arranged score, but instead to a direct manifestation of a conductor's thought processes on stage. For the Thought Conductorperformances, passages of musical notation remixed from the contents of a database were called up in the real time of the performance. "...score and sound aquire an immediacy which is characteristic of neuroscientific imaging in general, but which in this specific context lends new meaning to the notion of a 'live' performance. It is tempting therefore to suggest that Thought Conductor is a techno-scientific portrait that captures the inner kinetic melody of the individual who sits at its centre..."

Mariam Fraser, excerpt from abstract for Making Music Matter.

For the OX1 Festival the database archive made from the Oslo based composers was supplemented with The EEG files and notations from six Oxford based composers. The same procedures were followed. For the performance at the Hollywell Music Rooms several members of the festival audience participated. Again they were invited on stage, connected to the biomonitor and asked to engage in creative thought: an origami enthusiast recollected making a paper lotus flower; an accountant thought about a financial projection for a board of directors; a journalist recalled her last piece of arts criticism; an academic contemplated his tie...

Brainwaves and Plants - Miya Masaoka - 2002

This one is particularly odd

Presented as part of Lincoln Center Out of Doors, Homemade Instrument Day in New York, Pieces for Plants is an interactive sound installation for laptop, synthesizer, and the American semi-tropical climbing Philodendron. Versions of the piece have also been presented in a musical setting in which the plant participates as a member and soloist within an instrumental ensemble.

In the piece, a plant’s real-time responses to its physical environment are translated to sound. Highly sensitive electrodes are attached to the leaves of the plant. Scored movements by a human “plant player” stimulate physiological responses in the plant that are monitored via the electrodes and biofeedback wave analysis. The “plant player’s ” proximity, touch and interactions with the plant are then expressed in sound via midi and synthesizer. During the piece, the plant is brought to a range of physical/psychological states, from calm to agitation.