PushVFX is an interactive visualization tool that reacts to music. PushVFX uses an Ableton Push to control the effects of a music visualization system. The music drives the effects, and PushVFX allows for real time enabling/disabling/tuning of the visualizer’s effects.

The Ableton Push is a Midi interface with a grid of buttons, knobs, and a control screen. In PushVFX, each column on the grid corresponds to a visualization effect. For example, the left most grid column is for enabling/disabling the Cube effect. The green button means that it’s on, and the only parameter that is adjustable for the Cube effect is Scale, which is currently set to 14.5. Turning the knob above the display for that column will change the value of the currently selected parameter.

PushVFX uses a Fast Fourier Transform on the music in order to separate the sound frequency being played into a set of sample ranges each with a value for the strength of that part of the frequency range. Here’s an example frequency spectrum created by the FFT of a frame of music separated out into many samples:

The strength of each sample indicates how loud that part was for that sample. In practice, PushVFX uses a lower resolution FFT sample size that separates the sound out into 16 or 32 sample buckets. This is fairly efficient and provides enough information to drive the various effects in the system. See AudioSource GetSpectrumData for more information on Unity’s FFT processing.

Here’s an example with 32 samples graphed to show how each frequency range changes over time:

Each sample range is then associated with a particular effect parameter, which allows the music visualization to change with the music. For example, the low frequency samples that correspond to the drum beat might control how the cube’s scale pulses in response to the music. The song being played by PushVFX in the GIF below has a lot going on, which results in lots of good variety in the sample buckets for each frame.

Now that PushVFX is driving the visualization via the music, the Push interface allows for the effects to be toggled on/off, and also allows for each effect’s parameters to be further tweaked while playing live. This example below shows the Push controlling the character size of the ASCII effect.

MIDI Communications between Unity and Push device was challenging. Keijiro Takahashi wrote a plugin that allows unity to receive messages from a MIDI device named MidiJack. However, it only allows for reading incoming messages, and the PushVFX interface needs to send outgoing messages to the Push to update the text for the display, turn button lights on/off, etc. To allow for outgoing messages, MidiJack’s PluginEntry.cpp needed to be updated to allow for setting the outgoing Midi port, and to allow sending Status/Note/Velocity information and Byte arrays to the Midi device.

// Send Data
extern "C" void MidiJackSendOutgoingData(uint8_t channelStatus, uint8_t note, uint8_t velocity)
{
    if (!ResetIfRequired()) return;

    // only sending to active source
    MIDIEndpointRef dest = MIDIGetDestination(activeSourceID);

    MIDIPacket midiPacket = MIDIPacket();
    midiPacket.timeStamp = 0;
    midiPacket.length = 3;
    midiPacket.data[0] = channelStatus; // 0x90 + 0; // Note On event channel 1  (0x80 is off)
    midiPacket.data[1] = note; // 0x3C; // Note C3
    midiPacket.data[2] = velocity; //100;  // Velocity

    MIDIPacketList packetList = MIDIPacketList();
    packetList.numPackets = 1;
    packetList.packet[0] = midiPacket;

    MIDISend(midi_port_out, dest, &packetList);
}

Also, in MidiDriver.cs, need to make a few additions to support sending messages too:

// small message
public void SendData(byte channelStatus, byte note, byte velocity){

	byte finalStatus = (byte)(0x90 + channelStatus); //0xb0;
	SendOutgoingData(finalStatus, note, velocity);
}

and further down:

[DllImport("MidiJackPlugin", EntryPoint="MidiJackSendOutgoingData")]
public static extern void SendOutgoingData(byte channelStatus, byte note, byte velocity); // 8 bits each

Now that Unity can receive and send messages to the Push, it was just a matter of figuring out the keycodes and values in order to set text, turn lights on/off, etc. This guide was very helpful in figuring it out: Ableton Push MIDI Display Interface.

In Unity, the Push Controller game object is responsible for communicating with the Push device, and each slot is setup with a game object that controls the shader and a corresponding color:

For example, the last purple column corresponds to Custom Shaders. The named Game Object has a shader interface that receives messages and updates the shader. All of the shaders are using the V2 Post Processing Effect stack. Each parameter for the shader is configured with a range and step value.

Other custom image effects are interfaced the same way as a post processing effect shader. In this example, the Push is being used to turn a bool on/off that enables/disables the effect.

There are still a bit left to do on the PushVFX project, however it is very functional in it’s current state. In the mean time, ping me on twitter if you are interested in hearing more about this project.

Jesse from Smash/Riot