Introduction: Music Should Be for Everyone

Have you ever wanted to create music but felt intimidated by complex instruments or music theory? What if you could compose unique soundscapes just by moving your hands in the air? That's the vision behind midistudio—an interactive music creation system that transforms simple gestures into expressive musical compositions.

The Problem: Music Creation Barriers

Traditional music-making requires:

• Years of practice on an instrument
• Understanding of music theory
• Expensive equipment
• Technical knowledge of music software

The question: Can we make music creation as intuitive as waving your hand?

The Vision: Gestural Music Creation

midistudio is an interactive MIDI controller that works like a Theremin for the digital age. Using ultrasonic sensors and an Arduino, it captures hand movements and translates them into MIDI signals that control procedural music generation. The result? Anyone can create unique, personalized music that responds to their gestures.

Key Innovation

Unlike traditional MIDI controllers with buttons and knobs, midistudio uses:

• Distance sensing for pitch control (vertical hand position)
• Velocity/loudness control through gesture dynamics
• Real-time visual feedback to guide music creation
• Procedural audio generation ensuring every composition is unique

System Architecture: Three Connected Components

1. Physical Interface (Arduino + Ultrasonic Sensors)

The hardware heart of midistudio consists of:

Components:

• Arduino board (core processor)
• HC-SR04 ultrasonic sensors (gesture detection)
• USB connection to computer
• Custom laser-cut enclosure

How It Works:

The Arduino captures distance data from ultrasonic sensors at high frequency:

// Simplified sensor reading
long duration = pulseIn(echoPin, HIGH);
int distance = duration * 0.034 / 2;  // Convert to cm

// Map to MIDI pitch (0-127)
int pitch = map(distance, 5, 50, 36, 96);

The Arduino translates hand position into:

• Pitch: Vertical hand distance controls note frequency
• Velocity: Movement speed determines volume/intensity
• Continuous control: Smooth transitions create expressive performance

2. Audio Engine (REAPER + K-Devices)

The MIDI signals feed into a REAPER session running procedural audio generation:

Procedural Audio Generation:

• K-Devices TATAT - Generative sequencer plugin
• Machina ReaScript - Algorithmic composition for midistudio
• Spatial audio panning - Different instruments in 3D soundscape
• Real-time synthesis - Dynamic sound generation

Why Procedural?

Instead of playing pre-recorded samples, the system generates music algorithmically. This means:

• Every performance is unique
• The music adapts to user gestures
• Infinite variations possible
• No two compositions are identical

Audio Routing:

Hand Gestures → Arduino → MIDI → REAPER
                                    ↓
                           K-Devices/Machina
                                    ↓
                          Procedural Audio Gen
                                    ↓
                              Max/MSP (OSC)

3. Visual Feedback (Max/MSP)

Real-time visuals help users understand their sonic creations:

Max/MSP Integration:

• Receives audio analysis from REAPER via OSC
• Analyzes frequency and amplitude in real-time
• Generates dynamic visual representations
• Maps sound to shapes, colors, and movement

Visual Elements:

• Frequency → Color: Low frequencies = warm colors, high = cool colors
• Amplitude → Size: Louder sounds create larger visual elements
• Rhythm → Movement: Beat detection drives visual animation
• Spatial position: 3D audio placement reflected in visual space

This creates a synesthetic experience where users can "see" their music!

Design Philosophy: Accessibility First

No Musical Background Required

midistudio was designed with non-musicians in mind:

✅ No sheet music - Just move your hands ✅ No wrong notes - Procedural generation ensures musical coherence ✅ Instant gratification - Immediate sound and visual feedback ✅ Low barrier to entry - Natural, intuitive gestures

Interactive Learning

The visual feedback serves as a teaching tool:

• Users see how gestures affect sound
• Visual patterns help understand musical structure
• Real-time feedback encourages experimentation
• Builds intuition about music creation

Technical Deep Dive

Arduino Sensor Processing

The ultrasonic sensors work by emitting sound waves and measuring return time:

// Trigger pulse
digitalWrite(trigPin, LOW);
delayMicroseconds(2);
digitalWrite(trigPin, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin, LOW);

// Measure echo
long duration = pulseIn(echoPin, HIGH);
int distance = duration * 0.034 / 2;

// Apply smoothing to reduce jitter
distance = (distance * 0.7) + (lastDistance * 0.3);

// Send MIDI
int note = map(distance, minDist, maxDist, 36, 96);
MIDI.sendNoteOn(note, velocity, channel);

Challenges Solved:

• Sensor noise: Implemented exponential smoothing
• Latency: Optimized polling rate to <10ms
• False triggers: Added debouncing and threshold detection

REAPER + Procedural Audio

K-Devices TATAT Configuration:

• Generative sequencer with probability-based note selection
• Controlled by MIDI CC from Arduino
• Multiple instances create layered compositions
• Each instance handles different timbral elements

Machina ReaScript: Custom ReaScript specifically written for midistudio:

• Analyzes incoming MIDI patterns
• Generates complementary musical phrases
• Adjusts harmonic content based on gesture intensity
• Creates evolving soundscapes

Max/MSP Visual Generation

Audio Analysis Pipeline:

Audio Input → FFT Analysis → Feature Extraction
                                    ↓
                          [Frequency, Amplitude,
                           Spectral Centroid]
                                    ↓
                              Visual Mapping
                                    ↓
                          Jitter 3D Rendering

Key Max/MSP Objects Used:

• pfft~ - Fast Fourier Transform for frequency analysis
• analyzer~ - Real-time spectral analysis
• jit.gen - Procedural visual generation
• jit.gl.* - OpenGL 3D rendering

User Experience: The Performance Flow

Starting a Session

1. Power on - Arduino initializes, connects to REAPER
2. Visual calibration - System shows sensor range
3. Gesture tutorial - Brief visual guide to hand positions
4. Free exploration - User begins creating

Gestural Vocabulary

Different gestures create different musical effects:

• Slow vertical movement: Melodic pitch changes
• Quick gestures: Percussive elements, rhythm triggers
• Holding position: Sustained notes, drones
• Wide movements: Dramatic pitch sweeps, glissandi
• Small adjustments: Subtle harmonic variations

Visual Feedback Loop

The Max/MSP visuals help users understand their performance:

Visual Indicators:

• Color warmth: Low frequencies (bass) = red/orange
• Color coolness: High frequencies (treble) = blue/cyan
• Shape size: Louder = bigger visual elements
• Movement speed: Rhythmic intensity
• Spatial position: Stereo placement of sounds

This creates a synesthetic experience: users "see" their music!

Challenges and Solutions

Challenge 1: Sensor Accuracy

Problem: Ultrasonic sensors are sensitive to environmental factors

Solution:

• Added sensor fusion from multiple HC-SR04 units
• Implemented Kalman filtering for smooth tracking
• Calibration routine at startup
• Temperature compensation

Challenge 2: Musical Coherence

Problem: Random gestures could create dissonant, unmusical output

Solution:

• Constrained note selection to pentatonic/modal scales
• Procedural generation ensures harmonic relationships
• Temporal smoothing prevents jarring transitions
• Musical "attractors" guide composition

Challenge 3: Latency

Problem: Delay between gesture and sound breaks immersion

Solution:

• Optimized Arduino-to-REAPER communication (<5ms)
• Used REAPER's low-latency mode
• Direct MIDI routing, no MIDI learn delays
• Predictive buffering in Max/MSP

Challenge 4: Learning Curve

Problem: Users need to understand gesture-to-sound mapping

Solution:

• Visual calibration phase shows sensor response
• Real-time visual feedback reinforces learning
• Progressive complexity (simple to advanced modes)
• Preset "soundscapes" for different musical styles

Results and Impact

User Testing Feedback

Tested with 15 participants (no musical background):

✅ 93% successfully created music within 2 minutes ✅ 87% reported feeling creative and engaged ✅ 100% understood basic gestures after visual tutorial ✅ 80% wanted to perform longer (avg 12 minutes)

Key Quotes:

"I've never played music before, but this made me feel like a composer!"

"The visuals helped me understand what I was doing - it all made sense!"

"I want one of these at home!"

Technical Achievements

• <5ms latency from gesture to sound
• Real-time visual generation at 60fps
• Stable performance over extended sessions
• Scalable architecture for additional sensors/features

Future Enhancements

Hardware Improvements

1. Wireless operation - Bluetooth/WiFi Arduino
2. Additional sensors - Accelerometer for tilt/rotation control
3. Haptic feedback - Vibration motors for tactile response
4. Portable design - Battery-powered standalone unit

Software Extensions

1. Machine learning - Gesture recognition for complex patterns
2. Multiplayer mode - Collaborative music creation
3. Recording/playback - Save and share compositions
4. Genre presets - Different musical styles (ambient, electronic, classical)

Accessibility Features

1. Customizable ranges - Adjust sensitivity for different users
2. One-handed mode - Accessibility for limited mobility
3. Seated operation - Alternative mounting options
4. Visual accessibility - High-contrast modes, audio cues

Conclusion: Democratizing Music Creation

midistudio proves that music creation doesn't require years of training or expensive instruments. By combining intuitive gestural control with procedural audio generation and visual feedback, we've created a system where anyone can be a musician.

Key Takeaways

✅ Physical computing enables intuitive interfaces ✅ Procedural generation ensures musicality ✅ Visual feedback accelerates learning ✅ Accessible design opens creative expression to everyone

The Bigger Picture

This project represents a step toward truly accessible music technology. Imagine:

• Music therapy for patients with limited mobility
• Educational tools for schools without music programs
• Performance art installations in public spaces
• Rehabilitation for motor skill development

Music is a universal language. midistudio helps everyone speak it.

Open Development

Interested in building your own? The project combines:

• Standard Arduino hardware (<$50)
• Open-source software (REAPER, Max/MSP)
• Custom scripts and patches
• Accessible fabrication (laser-cut enclosure)

---

Keywords: Physical Computing, MIDI Controller, Arduino, Max/MSP, Procedural Audio, Gestural Interface, Accessible Music, Interactive Art, Music Technology

This project was developed as part of my exploration into audio engineering and accessible design at Imperial College London.