Project Description

Phantom Record is a music studio with recording and playback functions for two instruments. This project allows a user to remotely record (in or out of sync) two different instruments: a keyboard and a noise pad set. The recordings can then be remotely played back and locally visualized.

Personal contributions:

• Arduino programming
• Processing programming
• Filming and Pictures

Command Center

The Command Center for this project acted as the recording station for the keyboard and the noise pads. The user who is using the Command Center may use it to coordinate and save music tracks with their friends at the instruments. The user may record each instrument individually, or the user may simultaneously record the instruments. The same is true with individual/dual playback. When playing back a track, the Command Center sends back the recorded notes to the instruments, and the instruments will then play those sent notes. Only one track may be recorded per instrument.

The hardware components used:

• Arduino Uno (1)
• xBee Transmitter/Reciever (with xBee shield) (1)
• Push buttons (6)
• Red LED (3)
• Green LED (3)
• Wooden, laser engraved enclosure

Instrument 1: Keyboard

The "keyboard" is a "keyless" musical keyboard that plays notes based on hand distance from a proximity sensor. Users can play this remote instrument, one note at a time, by simply striking the air or waving their hand in front of a sensor. The intended user is anyone who is interested in playing music with their hands. Notes played can be recorded and sent to the command center as an array of data. While the user is playing the instrument, the sound of the instrument will be played back in real time.

The primary sensor is an ultrasonic distance sensor, also known as a proximity sensor. The primary actuator is a 3 inch speaker. Secondary actuators include 1) LED light feedback per ‘key’ and 2) LCD message of the note played. The components used were:

• Arduino Uno (1)
• Ultrasonic distance sensor (1)
• 3 inch speaker (1)
• LEDs (1 per ‘key’) (13)
• 16 x 2 LCD display (1)
• xBee Transmitter/Reciever (with xBee shield) (1)
• Wooden, laser engraved enclosure

When a hand is at certain distance from the distance sensor, the speaker will play certain note. When a note is played, the corresponding red LED lights up on the specific key. The specific note is etched onto the key of the keyboard while the note and octave are displayed on the LCD screen.

Instrument 2: Launchpad

The Launchpad allows users to generate audio by pressing a set of four force-sensitive pads. The sound generated is a sustained tone of a frequency that increases the harder a user presses their finger against a pad. The set of noise pads rests in a compact enclosure to maximize portability and minimize storage space. The unit is intended to function as part of  a larger band, but also functions as a solo instrument. Upon successful wireless connection with the command center, users are also able to record and playback audio. A user of our 4 product is someone who enjoys casual social music creation, but lacks the storage space required for a larger instrument. The components used were

• Force Sensitive Resistors (FSR) (4)
• 4 Ω, 3 Watt Speaker (1)
• Red LEDs (4)
• 10 kΩ resistors (4)
• Arduino Uno (1)
• xBee Transmitter/Reciever (with xBee shield) (1)
• Wooden, laser engraved enclosure

Implementation

Command Center

There are six total buttons on the Command Center -Record Keyboard, Play Keyboard, Record Noise Pads, Play Noise Pads, Record Both Instruments, Play Both Instruments - in addition to a red LED and a green LED.

When a user presses a record button, the red LED lights up; the Arduino begins two new strings - notes and time - for the selected instrument (or for each instrument if the Record Both buttons are pressed). Each instrument constantly - regardless of whether the Command Center is recording - outputs a code base on which note is pressed, or if no note is being pressed. The Keyboard sends a 0 if no note is being pressed, otherwise it sends a 1 to a 89 depending what note is pressed; the noise pad set sends a 99 for no note, otherwise it sends a 100 to 150. Based on the selected instrument(s), the Arduino begins picking up the corresponding signals through the xBee and stores the notes in the "notes" string and stores the delays (in milliseconds) between the notes in the "time" string. During the recording , the Arduino sends the received notes as a Serial Monitor message to the Processing program, where it lights up the received note on a visual computer display, that appears as a piano keyboard and noise pad squares. When the user presses same record button again, the red LED turns off; the Arduino ceases the recording process, and overwrites any previously stored track.

When a user presses a play button, the green LED lights up; the arduino sends a code(s) (90 for keyboard, 151 for noise pads) that is picked up by the corresponding instrument. When the instrument receives the code, it disables its direct user input in preparation for playback. The Arduino begins broadcasting the code/notes in the corresponding music track string; broadcasting duration is determined locally, using the proper "time" string. During the playback , the Arduino sends the broadcast notes as a Serial Monitor message to the Processing program, where it lights up the note on a visual computer display. When all notes have been broadcast, or the user presses the same play button in the middle of playback, the Arduino sends a code to (91 for keyboard, 152 for noise pads) that allows the corresponding instrument to allow receiving direct user input again.

Keyboard

We designed for each key to be 3 inches for a total span of 39 inches across 13 keys and mapped this distance (in centimeters) for 13 intervals. Each interval corresponds to a specific key and depending on where the distance sensor detects a hand or object, it will play that specific key by mapping the index. The different note frequencies are pre-programmed in 7 different octaves starting from lower C to upper C. For example, if a hand is placed 0-3 inches away from the sensor, it will play the lower c note from that respective octave. The default octave is octave 4 out of the 7 octaves, which is exactly in the middle.

When the ultrasonic distance  sensor detects an object interrupting the high-frequency sound pulse it sends out, it calculates the distance and plays the respective note from the speaker, lights up the LED on the key, and displays the note played on the LCD screen (e.g. "Note: b"). The user is able to adjust the octaves by pressing the "up" button, which increases the octave by one or the "down" button which decreases the octave by 1. The current octave is displayed on the LCD under the note (e.g. "Octave: 4").

Launchpad

As a user presses upon one of the force sensitive resistor pads, our software measures the resistance from the Arduino's analog inputs.  The resistance is initially interpreted as an integer from 0 to 1024, before being translated to a smaller range that can be read by an XBee radio.  Each pad corresponds to a unique range of values to differentiate the sound they produce. A small multiplier is applied to the input value to produce numbers that correspond to frequencies in an audible range. These are given to the tone() function that generates the sound on the speaker. 

When given the signal from the command center to record, each value received from the Arduino's analog inputs is translated and sent to the command center via the XBee radio. In playback mode, the analog inputs are disabled, and our XBee receives a series of integers from the command center, that are translated to the appropriate audible range and played back from the speaker.

Software

Command Center

For the Command Center, there are two sets of code: the Arduino code and the processing code.

Arduino Code

Due to the fact that all the Arduino code has to run in a single-logic loop, there were many tricky workarounds implemented in order for proper recording and playback. But to begin, on start the Arduino initializes all the ports and sets the note/time tracks for the instruments to a default sequence; strings were chosen to store tracks as they are easily expandable and explodable. In order for playback and recording functions to not interfere with each other, the program creates smaller internal loops that will ignore all but the appropriate button input when a function is activated. More simply, when recording the keyboard, the user is prevented from doing anything other than stop the recording, or prevented from anything but stop the playing during playback, etc. This allows the program to narrow down its processing time so it can efficiently inscribe received notes, as well as being user friendly as the user cannot overlap conflicting functions. The codes for notes and instruments were agreed upon earlier to minimize sent information.

Processing Code

The processing code takes "note" and "duration" on the serial monitor and visualizes them in a form of a Piano keyboard and a drum pad. The graphical element consists two still images files of a Piano keyboard and a drum pad. When a note is played, the processing code will then draw an rectangle on a certain area to highlight the key/pad being played. Since each note has a distinct number assigned to it, the processing code can tell what the note is and highlight the area accordingly. To make sure the highlighted area matches the image files, each individual key and pad is measured in pixels. The measurement is stored in the pde file as comments. Since processing highlights any note when Arduino is sending, there is no need to deal with the "time" parameter. When Arduino sends the signal for "mute", processing simply displays noting.

Keyboard

There are pre-programmed 7 octaves based on its frequency, totaling 85 keys. Each octave consists of 13 keys from lower c to upper C. The program keeps track of the current octave it is on to be able to play the correct pitch. All 13 keys are mapped based on distance to 13 different indexes. The distance range is 39 inches, starting at 2cm and going to 101.06 cm. The method "readDistance" reads the distance from the ultrasonic distance sensor to determine what the distance is and this distance is then corresponds to an index in the range of 0 to 12. Based on the index, the corresponding note will play by calling the method "playNote" and providing the given index as a parameter. This method plays the note from the speaker, lights the correct LED, displays the note on the LCD screen. Also, the note will be sent through the xBee to Mission Control based on a number value between 1 to 86; the value 0 represents silence. It knows the note because the notes are stored by value and frequency in separate arrays of notes and frequencies. Specifically, the notes are stored in a 13-element string array from "c" to "C" and the note frequencies are stored in a 7 x 12, 2-dimensional array, holding frequencies for all of the possible keys.

For the LED's, Charlieplexing was used for 12 of the 13 keys. For 12 LED's, it required using 4 pins of the Arduino which are initialized at the top of the source code. Each key has it's own index (0 to 12) and based on the key that is played, the LED will light up by calling the "turnOn" method and providing the index of the LED as a parameter. Another method is called "changeOctave" which handles the change in state of octaves. The state will changed based on the button "up" and "down" button states. If either of these buttons are clicked, the method will be called to increase or decrease the current octave, respectively. This change will be reflected on the LCD screen, as it displays the current octave and the notes played after will reflect a different octave.

Lastly, the keyboard is connected to Mission Control through an Xbee. Aside from sending notes to another Arduino, the system is able to receive notes to play back. It works by determining if the other xBee is available. If it is available, we read in the note value and convert that note value to correspond to how we interpret notes: by index and by octave. Then, this note is played through the speaker and the respective LED lights up as it is playing back.

Launchpad

All sounds are generated using the included Tone Library in the Arduino IDE. The software implementation was the same as the keyboard except the digital input from the distance sensor was changed to force sensitive sensors.

Gallery

Reflection

Successes

• The Command Center was able to record and playback music tracks reliably.
• The Processing station was able to correctly highlight the played back notes, as well somewhat display the recorded notes (there was some lag from the xBee connection).
• The workaround with recording times between notes worked surprising well, considering the Arduino has no built in process for co-routines and the recording logic had to be contained in a continuous looping function.
• The laser cut enclosure visually well designed and fit our project components

Challenges

• The initial plan was to have the Command Center locally play the tracks with its own speakers. However, due to how the Arduino's internal tone timing function works, it was impossible to simultaneously play both tracks, which is why the playback is sent back to the instruments in the final design. If we wished to play two music tracks at the same time from one Arduino, a customized square/sine wave function would have to be developed, and time constraints did not allow this.
• When testing out the recording feature in its final steps, it seemed to "fizzle out" for no particular reason. It was later discovered that, because of the (inefficient) way the recording data is stored, the Arduino would get easily overloaded from trying to store the data. If we wished to store longer tracks, we would have to offload it somewhere else, and possibly implement a load/save function.
• Attaching the buttons on the enclosure faced several problems, as the hardware was in the way and there was no simple way to attach wood to the pushbuttons.
• Buttons for changing octaves were low within the enclosure so they were difficult to reach and made it difficult to troubleshoot the Arduino because it required removing the toggles for the buttons
• The ultrasonic distance sensor ineffective when many people are around

Reflection and Comments

Overall, the project was a bit too ambitious from the beginning, meaning that numerous changes and cut corners needed to be applied. Due to hardware and software limitations, the progress turned out to be more difficult than expected. One unfortunate aspect of the project was that the simultaneous record/playback functions were never actually applied, as while each instrument did function they coincidentally were never functioning at the same time. But although there were numerous issues, the end project still functioned in most of its aspects.

Music should be as easy to record as languages

Other Contributors:

Kenji Goodson, Bonnie Tran, Emily Wong, Tariku Allen and Eric Eckert