Wearemin - A Bluetooth Wearable Theremin-like-instrument

My entry for the 2023 Musical Instruments contest. I added the code and onshape design files here https://github.com/StyxyDog/wearemin 

What is Wearemin?

Using an ESP32, a couple of cheap laser range finders and a few inexpensive components you can make a this portable Theremin-like instrument that connects to a Bluetooth speaker or headphones.  Wearemin (Wearable Theremin). 

How to play.

Connect to a nearby Bluetooth Speaker. Just put a nearby speaker in paring mode. I've found some devices reluctant to pair, but most work fine. 

Raising and lowering one handheld controller causes the pitch of the note to change over several octaves. If you press the thumb control, there is a step change between notes - like running your finger across piano keys. 

The other handheld controller adjusts the volume of the note - or press the thumb control button to instantly mute the sound.

The laser is eye-safe as it is low power and unfocused. However, you should check you are comfortable by doing your own research.

Huge apologies to anyone who decides to watch my video - the production and sound quality is truly awful! 🤣

Want to play standing up, against a wall or at a table?

The central control box has a latching button. Press this to enter programming mode. Whilst in programming mode hold down the thumb control and wave your hand between the near and far points of the range you want to play. When you let go of the thumb button that range is stored for that controller - repeat for the other handheld controller and then press the latching button again to exit programming mode and resume play mode.

The Project.

I had the idea for this instrument right at the start of the contest. It has been a very steep learning curve for me in terms of music theory, programming and electronics. I hope that either myself or others will improve my code and electronics in time. I was careful not to research the theremin or other devices too deeply so I can't say if this device is unique, but I believe it is - this is all my own work. Its taken me the full two months to get to this point, not helped with a period of ill health and delays in getting parts and supplies shipped out to me. What a great challenge it has been! Most of the code is built from example code that is supplied to use with the VL53L0X time of flight module. Also the amazing (and to me almost incomprehensible) work of Phil Schatzmann who seems to have provided code and examples of almost every sort of ESP32 to Bluetooth audio connection.

Below you will find the code and instructions to build this fun instrument. Maybe you can even make it play a tune! There is a lot of potential.

The Hardware.

I have built the device to be very easy to assemble, basic soldering is required, but I have tried to use breakout boards and DuPont connectors where possible.

Printed parts.

Quickly making an ergonomic handheld device would be a difficult job by almost any other process, but 3D printing makes this whole project possible. This is a fairly simple print, You need to print 2 each of the handheld controllers (top, base and cable clip), and 1 each of the central control box base and lid. My Ender 3 struggled with some of the finer details, wish I had a Pruser 😅. 20% infill and a 0.2 layer height is fine. 

Bill of Materials.

Please see pictures to confirm you are using similar parts.

For the central controller:

• an ESP32 development board;
• an ESP32 breakout board;
• 2 x RJ45 Ethernet network LAN 5 wall end plug module adapter jack;
• USB C breakout board;
• a couple of meters of Ethernet patch lead with connectors, ideally look for a stranded one for a little more flexibility. Solid core is fine too though;
• a 10mm latching button.

For the handheld controllers:

• 2 x 16mm round momentary push buttons;
• 2 x VL53L0X Time of flight laser range finders (these sound expensive, but are very cheap). Be careful to get the right format. You can also try the VL53L1X for a longer range wearemin. They will fit with the lens in the controller, but I didn't have time to fully make these longer range modules work reliably.

Other hardware:

• 14 x M2 screws (approx. 5 mm) to hold the VL53L0X and USB-C & ESP32 breakout boards in place. Also these will hold the central controller case closed;
• 8 x M3 screws (approx. 10 mm) to hold the cable clips and hand controllers together;
• Some spare bits of wire for connecting everything up - I just used some of the Ethernet cable cores;
• Tools for soldering and assembly. DuPont connectors and crimps are nice to keep things simple and neat, as are some short lengths of shrink tube;
• USB power source, battery pack for true wireless; 
• Bluetooth speaker or headphones.

Assembly Instructions.

The handheld controllers.

You need an Ethernet/patch lead with only one plug head on, so either cut a longer one in half or cut the ends off two shorter cables. Strip back about 10 cm of the plastic outer sheathing.

I used an angled header on the VL53L0X to save space. I connected via DuPont, but 

soldering connections would be good too. Connections as follows:

• Orange to  VL53L0X VIN pin;
• Brown to VL53L0X GND pin;
• Blue to VL53L0X SCL pin;
• Blue/White to VL53L0X SDA pin;
• Orange/White to VL53L0X XSHUT pin;
• Green to the thumb button (either terminal);
• Green/White to the thumb button (either terminal);
• Brown/White is unused. It could enable another button or connect the interrupt pin on the VL53L0X.

As long as the right pins connect to the right pins on the controller you should be able to use whatever colour scheme or cable you like. 

Install the 16 mm button and screw the VL53L0X  (ensure the laser is pointing out of the hole) and secure the cable with the cable clip (invert the clip if you have a lower diameter cable) there should be plenty of space to took any extra cores from cable in without trapping. 

The central controller.

Push the ESP32 into the breakout board, mine was too narrow by a mm or so, but it went in despite that. Solder wires on to the latching button and push into place and install the locking nut if needed. The RJ45 sockets are a loose fit and are only secured when the lid is attached. If you followed my colour instructions above and you followed the type B wiring guide on the socket (orange/white first on the cable) then you can use the following to wire up the control box. Solder leads to the USB C Breakout board - I used Orange for VCC and Brown for GND.

Your ESP32 may have pins in different positions so check before powering up:

• 3 x Orange from  both VL53L0X VIN pins and USB-C VCC to 5V on ESP32 Breakout;
• 3 x Brown from  both VL53L0X GND pins and USB-C GND to a GND on ESP32 Breakout;
• 2 x Blue VL53L0X SCL lines should connect to SCL pin 22 on ESP32 Breakout;
• 2 x Blue/White VL53L0X SDA lines should connect to SDA pin 21 on ESP32 Breakout;
• 2 x Orange/White VL53L0X SHUT lines should connect to pins 16 and 17 on ESP32 Breakout;
• 2 x Green from the thumb buttons on the handheld controllers can connect to any GND on ESP32 Breakout;
• 2 x Green/White from the thumb buttons on the handheld controllers can connect to pins 19 and 23 on ESP32 Breakout;
• Make sure you connect the right button to the right  pin, so connections to pins 23 and 17 should come from the same handheld controller. If you get this wrong you'll find the thumb button is on the wrong handheld controller;
• The momentary switch on the central controller should connect to GND and P18 on ESP32 Breakout.

Don't put the lid on as you have to program the board first.

The code.

I used Arduino IDE to program my board, instructions for that process are widely available. If this is your first time using and ESP32, I recommend you start with a simpler/better documented project! I hope to improve this code as my knowledge and ability improves!

The program does a few key things. It creates the Bluetooth connection and generates a sine wave. Measurements are taken from each handheld controller to adjust the frequency and amplitude of the wave. I run this process as a separate task on the ESP32's other core to try and speed up processing - this also mitigates my basic programming abilities to some degree.

The program also monitors the status of the mute button and the ‘notes mode’ button as well as checking if the momentary switch has been activated for programming mode.

One of my proudest moments in this project:

frequencyTarget = round(frequencyMin * pow(2,((controller1Range - controller1Min) / (controller1Max-controller1Min) * log2(frequencyMax/frequencyMin))));

That was when I realised that musical notes are not linear (I have no musical theory knowledge - Ha! I do now)! I was getting all my tones stacked up towards the top end of my movement. It took a long time and a lot of paper to come up with the formula above to change this logarithmic scale into a linear one.

The full code is at the very end of this article.

Limitations and Improvements.

1. Well the sound production is a little laggy or maybe noisy, is this because of Bluetooth or my poor code? I don't know but I have gone to my limit without investing more time in educating myself better. The sound in the video is much worse than in reality as I had no reliable way to record the output.
2. I wanted to use the longer range VL53L1X modules - good for up to 4 m. However I could not get these to work reliably and in the end decided to abandon the idea and aim to complete the project. The VL53L0X is good to 2 m although I think about 90 cm is more realistic for playing the instrument. That is good for playing against the floor or a desk whilst standing or sitting.
3. I don't think it sounds much like a theremin, which is a shame as they are great. Still it is fun to play around with and I think with the right level of thought and attention this idea could be a fun instrument. It's more of a weird sound generator at the moment. Mind you if I played violin of any other musical instrument it would probably sound like a weird sound generator too!!
4. Some status indicator LED's would be great to show when you are in programming mode or when the device is connecting to Bluetooth.
5. Time, I need more time to make this really great, but I'm really happy I got to some sort of end to the project in the 60 days allowed.
6. The VL53L0X TOF sensor is a great little device and I'm thinking of loads of applications already.
7. I'm making everything fully open source so that anyone can take my idea forwards - keep in touch and share your tunes!
8. If you decide to print and build this, I hope you have fun and maybe improve on my design. 

The full code…

#include "Adafruit_VL53L0X.h"
#include <RunningMedian.h>
#include <math.h>

////////////////
#include "AudioTools.h"
#include "AudioLibs/AudioA2DP.h"
///////////////

// address we will assign if dual sensor is present
#define CONTROLLER1_ADDRESS 0x30
#define CONTROLLER2_ADDRESS 0x31

// set the pins to shutdown
#define SHT_CONTROLLER1 17
#define SHT_CONTROLLER2 16
#define CONFIG_SWITCH 18
#define CONTROLLER1_SWITCH 23
#define CONTROLLER2_SWITCH 19
#define numberOfnotes 37
//( 48  =49 notes )double notes[numberOfnotes] = {130.81,138.59,146.83,155.56,164.81,174.61,185,196,207.65,220,233.08,246.94,261.63,277.18,293.66,311.13,329.63,349.23,369.99,392,415.3,440,466.16,493.88,523.25,554.37,587.33,622.25,659.26,698.46,739.99,783.99,830.61,880,932.33,987.77,1046.5,1108.73,1174.66,1244.51,1318.51,1396.91,1479.98,1567.98,1661.22,1760,1864.66,1975.53,10000};
double notes[numberOfnotes] = { 130.81, 138.59, 146.83, 155.56, 164.81,
                                174.61, 185,    196,    207.65, 220,
                                233.08, 246.94, 261.63, 277.18, 293.66,
                                311.13, 329.63, 349.23, 369.99, 392,
                                415.3,  440,    466.16, 493.88, 523.25,
                                554.37, 587.33, 622.25, 659.26, 698.46,
                                739.99, 783.99, 830.61, 880,    932.33,
                                987.77, 10000};
TaskHandle_t Task1;
RunningMedian controller1Samples = RunningMedian(3);
RunningMedian controller2Samples = RunningMedian(3);

/////////////////////

const char* name = "";//"Uproar Wireless";
SineFromTable<int16_t> sineWave(32000);                // subclass of SoundGenerator with max amplitude of 32000
GeneratedSoundStream<int16_t> sound(sineWave);             // Stream generated from sine wave
BluetoothA2DPSource a2dp_source;


int32_t get_sound_data(uint8_t * data, int32_t len) {
  return sound.readBytes((uint8_t*)data, len);
}


double frequencyTarget, frequency = 500;
int amplitudeTarget, amplitude = 1000;
double frequencyMin = 130.81;
double frequencyMax = 987.77; //1975.53;
int volume = 90;
//////////////////////////




double controller1Range, controller2Range;
int configSwitchState, controller1SwitchState, controller2SwitchState;
double controller1Min = 50, controller1Max = 800; //default range settings
double controller2Min = 50, controller2Max = 800; 

// objects for the vl53l0x
Adafruit_VL53L0X controller1 = Adafruit_VL53L0X();
Adafruit_VL53L0X controller2 = Adafruit_VL53L0X();

// this holds the measurement
VL53L0X_RangingMeasurementData_t measure1;
VL53L0X_RangingMeasurementData_t measure2;

void setup_LOX() {
  //setup the controllers
  pinMode(SHT_CONTROLLER1, OUTPUT);
  pinMode(SHT_CONTROLLER2, OUTPUT);
  pinMode(CONFIG_SWITCH, INPUT_PULLUP);
  pinMode(CONTROLLER1_SWITCH, INPUT_PULLUP); 
  pinMode(CONTROLLER2_SWITCH, INPUT_PULLUP);
  
  delay(500); 
  Serial.println("Starting mesurment control task");
  Serial.println(F("Shutdown pins inited..."));
  digitalWrite(SHT_CONTROLLER1, LOW);
  digitalWrite(SHT_CONTROLLER2, LOW);
  Serial.println(F("Both LOX in reset mode...(pins are low)"));
  Serial.println(F("Starting LOX..."));
    digitalWrite(SHT_CONTROLLER1, LOW);    
  digitalWrite(SHT_CONTROLLER2, LOW);
  delay(10);
  // all unreset
  digitalWrite(SHT_CONTROLLER1, HIGH);
  digitalWrite(SHT_CONTROLLER2, HIGH);
  delay(10);

  // activating CONTROLLER1 and resetting CONTROLLER2
  digitalWrite(SHT_CONTROLLER1, HIGH);
  digitalWrite(SHT_CONTROLLER2, LOW);

  // initing CONTROLLER1
  if(!controller1.begin(CONTROLLER1_ADDRESS)) {
    Serial.println(F("Failed to boot first VL53L0X"));
    while(1);
  }
  delay(10);

  // activating CONTROLLER2
  digitalWrite(SHT_CONTROLLER2, HIGH);
  delay(10);

  //initing CONTROLLER2
  if(!controller2.begin(CONTROLLER2_ADDRESS)) {
    Serial.println(F("Failed to boot second VL53L0X"));
    while(1);
  }
  Serial.println("TOF addresses set.");  
}

void setup_BT_wave() {
  auto cfg = sound.defaultConfig();
  cfg.bits_per_sample = 16; cfg.channels = 2; cfg.sample_rate = 44100;
  sound.begin(cfg);
  
  sineWave.begin(cfg, frequency);
  
  // start the bluetooth
  Serial.println("starting A2DP...");
  a2dp_source.set_auto_reconnect(true);
  a2dp_source.start_raw(name, get_sound_data);
  a2dp_source.set_volume(30);



  Serial.println("A2DP is connected now...");
}

void setup() {
  Serial.begin(115200);
  setup_LOX();
  setup_BT_wave();
  
  Serial.println("Calling mesurment control task");
  xTaskCreatePinnedToCore(measureControl, "Task1", 10000, NULL, 1, &Task1, 0); // create a task to run on core 0
}

void measureControl( void * pvParameters ){

  for(;;){
    controller1.rangingTest(&measure1, false);
    controller2.rangingTest(&measure2, false);
    controller1Samples.add(measure1.RangeMilliMeter);
    controller2Samples.add(measure2.RangeMilliMeter);
    controller1Range = controller1Samples.getMedian();
    controller2Range = controller2Samples.getMedian();


    //programming loop
    bool controller1ProgramingModeEntry = true, controller2ProgramingModeEntry = true;
    configSwitchState = digitalRead(CONFIG_SWITCH);
    while (configSwitchState == LOW) { //enter range setup mode
      Serial.print(".");
      if (controller1ProgramingModeEntry == true) {
        controller1Min=8192;
        controller1Max=0;
        controller1ProgramingModeEntry = false;
        Serial.print("Set Default Values");
      }
      if (controller2ProgramingModeEntry == true) {
        controller2Min=8192;
        controller2Max=0;
        controller2ProgramingModeEntry = false;
        Serial.print("Set Default Values");
      }

      controller1SwitchState = digitalRead(CONTROLLER1_SWITCH);
      if (controller1SwitchState == LOW) { //program controller 1
        controller1.rangingTest(&measure1, false);
        controller1Samples.add(measure1.RangeMilliMeter);
        controller1Range = controller1Samples.getMedian();
        Serial.print("Controller 1 : Programming Mode Range : "); Serial.print(controller1Range); Serial.print(" -:- Min : "); Serial.print(controller1Min); Serial.print(" -:- Max :"); Serial.println(controller1Max);
        if (controller1Range > controller1Max && controller1Range < 2047) { controller1Max = controller1Range; Serial.print("SetMax"); }
        if (controller1Range < controller1Min && controller1Range > 0   ) { controller1Min = controller1Range; Serial.print("SetMin"); }   
      }

      controller2SwitchState = digitalRead(CONTROLLER2_SWITCH);
      if (controller2SwitchState == LOW) { //program controller 2
        controller2.rangingTest(&measure2, false); 
        controller2Samples.add(measure2.RangeMilliMeter);
        controller2Range = controller2Samples.getMedian();
        Serial.print("Controller 2 : Programming Mode Range : "); Serial.print(controller2Range); Serial.print(" -:- Min : "); Serial.print(controller2Min); Serial.print(" -:- Max :"); Serial.println(controller2Max);
        if (controller2Range > controller2Max && controller2Range < 2047) { controller2Max = controller2Range; Serial.print("SetMax"); }
        if (controller2Range < controller2Min && controller2Range > 0   ) { controller2Min = controller2Range; Serial.print("SetMin"); }   
      }
      
      delay(10);
      configSwitchState = digitalRead(CONFIG_SWITCH);
    }
    
    controller2SwitchState = digitalRead(CONTROLLER2_SWITCH);
    if (controller2SwitchState == LOW) { 
      volume = 0;
    } else {
      volume = 100;
      controller1Range = constrain(controller1Range, controller1Min, controller1Max);  
      controller2Range = constrain(controller2Range, controller2Min, controller2Max);  
      amplitudeTarget = map(controller2Range,controller2Min,controller2Max, 0,32000);
      amplitude = amplitudeTarget; // no smoothing needed but it would go here
    }
    frequencyTarget = round(frequencyMin * pow(2,((controller1Range - controller1Min) / (controller1Max-controller1Min) * log2(frequencyMax/frequencyMin))));

    controller1SwitchState = digitalRead(CONTROLLER1_SWITCH);
    if (controller1SwitchState == LOW) {
      for (int i = 0; i<numberOfnotes-1 ; i ++) {
        if (frequencyTarget > notes[i] && frequencyTarget < notes[i+1]) { frequency = notes[i]; break; }
      }
    } else { frequency = frequency+((frequencyTarget-frequency)/3); //smooth the progression }
  } 
}

void loop() {

  /* uncomment for troubleshooting!
  Serial.print("frequency%:");
  Serial.println(float(frequency)/(frequencyMax-frequencyMin)*100);
  Serial.print("amplitude%:");
  Serial.println((float(amplitude)/32000)*100);//*/
  a2dp_source.set_volume(volume);
  sineWave.setFrequency(frequency);
  sineWave.setAmplitude(amplitude);
}