Standards for Control Data

A central concept of Sonosthesia cross-modal control flow between components. Sonosthesia uses a form of control data heavily influenced by MIDI and OSC, but also drawing from protocols used for multi-agent systems such as ACL.

Drawing from MIDI, OSC and ACL

MIDI has survived as a protocol for sound control despite its limitations (7 bit depth for channel, pitch, small number of channels, limited control and after-touch etc...) shows the power of enforcing standard semantics. Electronic musicians can visualise musical ideas in MIDI because musical ideas are buit into the protocol. In order to overcome the limitations of MIDI, audio-visual digital art instalations often use the open-ended OSC protocol, which is very flexible and generic but lacks the standardised semantics of MIDI which makes colaboration between projects and software more complicated as different agents must agree on some overlaying structure to exchange data. OSC hasn't replaced MIDI, it has complemented it, often as a low-level controller (for colors, textures, frequencies, amplitudes, filter parameters...).

The protocol used Sonosthesia also draws from a small subset of ideas used in the field of multi-agent systems (such as component ontology expressed by ACL). This allows interactions between components to be more expressive, dynamic and flexible, by enabling them to publish their input and output channels with associated control paramaters. The central hub application which routes control messages from different client component applications uses this to offer a generic mapping interface which reflects the currently available channels.

JSON and transport abstraction

JSON has been chosen for data representation as it combines flexibility and structure with an explicit, self-documenting human readable format. It is also transport-independent and its capacity for handling a wide variety of messaging patterns is well tested. There might be some over-head in terms of size but with modern hardware, the effect will be insignificant unless in the case of very high message trafic.

Declaring Components

Components can declare themselves, the channels that they support, for each declaring a set of control parameters to the central hub. An example DAW component with synth and sound analysis channels:

    {
        "component": "daw1",
        "channels": [
            // a typical DAW synth control component declaration
            // recieving MIDI like control data
            {
                "identifier": "synth1"
                "description": "lead synth", // optional
                "flow": "receiver",
                // paramaters can be declared as a list of identifiers
                // in which case they are [0-1] with default 0
                "parameters": ["pitch", "velocity", "bend", "shine"]
            },
            // a typical DAW sound description channel declaration, emitting sound descriptors
            {
                "identifier": "analysis1"
                "flow": "emitter",
                // paramaters can be declared as a list of objects
                // with specific dimensions, min, max and default
                "parameters" : [
                    {
                        "identifier" : "pitch",
                        "dimensions": 1,
                        "min": 0,
                        "max": 10,
                        "default": 0
                    },
                    {
                        "identifier" : "spectral_bins"
                        "dimensions": 10, // multi-dimensional, energy in spectral bins
                        "min": 0,
                        "max": 10,
                        "default": 0
                    }
                ]
            }
        ]
    }

An example game engine component with contact descrition and force (wind) channels:

    // Example game engine component with contact descrition and force (wind) channels
    {
        "component": "vr1",
        "channels": [
            // a typical game engine contact descrition channel declaration
            {
                "identifier": "sphere1",
                "flow": "emitter",
                "parameters": ["incidence", "velocity", "wood", "metal", "hardness"]
            },
            // a typical game engine force control channel declaration
            {
                "identifier": "wind2",
                "flow": "receiver",
                "parameters" : [
                    // note direction could be a single float describing direction angle
                    {
                        "identifier" : "direction",
                        "dimensions": 3, // vector 3, [0,1] range
                        "default": [0,0,1]
                    },
                    {
                        "identifier" : "force" // default 1 dimensionality
                        "default": 0.5
                    }
                    {
                        "identifier" : "swirl" // default 1 dimensionality
                    }
                ]
            }
        ]
    }

Controling Static Channel Parameters

Channel parameter control messages only the message contains a channel field instead of an object field to identify the target. Although they can be used as object factories, channels can also be used as static control mechanisms (controlling the tempo of a musical rendering for example). If the channel has created objects, the parameter or property changes will be propagated to them.

{
    "type": "control",
    "component": "component1"
    "channel": "channel1",
    "parameters": {
        "parameter1": 0.32352,
        "parameter2": 0.87467
    }
}

Creating Dynamic Instances

An instance is created on a given channel, with a given (machine-generated) unique identifier. Initial values can be given for its parameters.

    {
        "type": "create",
        "component": "component1",
        "channel": "channel1",
        "instance": "123e4567-e89b-12d3-a456-426655440000",
        "parameters": {
            "parameter1": 0.3245543,
            "parameter2": 0.5433435
        }
    }

Destroying Instances

An instance is can be destroyed using its unique identifier. Final property and parameter values can be given to guide it's destruction process (fade outs, explosions, etc...).

    {
        "type": "destroy",
        "component": "component1",
        "channel": "channel1",
        "instance": "123e4567-e89b-12d3-a456-426655440000",
        "parameters": {
            "parameter1": 0.3245543,
            "parameter2": 0.5433435
        }
    }

Modifying Instance Parameters

Once an instance is created, any its parameters can be changed by using its unique object identifier.

    {
        "type": "control",
        "component": "component1",
        "channel": "channel1",
        "instance": "123e4567-e89b-12d3-a456-426655440000",
        "parameters": {
            "parameter1": 0.3039989,
            "parameter2": 0.5386749
        }
    }

A parameter message can be applied to multiple instances by specifying an array of unique instance identifiers.

    {
        "type": "control",
        "component": "component1",
        "channel": "channel1",
        "instance": ["123e4567-e89b-12d3-a456-426655440000", "42665544-e89b-12d3-a456-123e45670000"],
        "parameters": {
            "parameter2": 0.3039989
        }
    }

Multi-dimensional parameters are allowed but think carefully about whether you need them as they make mappings more complex. Just pass an array of floats instead of a float. Note that the demensionality should be specified in the component declaration.

    {
        "type": "control",
        "component": "component1",
        "channel": "channel1",
        "instance": "123e4567-e89b-12d3-a456-426655440000",
        "parameters": {
            "parameter2": [0.3039989, 0.3425266]
        }
    }

Receiver and Emitter Channels

This messaging pattern can be used for both controlling and describing things. That is a channel can emit messages in order to describe an ongoing process, or a channel can receive messages and apply them to control an ongoing process. In this sense channels are emitters or receivers.

Controlling Sound Synthesis

Concrete examples of how these concepts can be applied are useful. For sound synthesis, we can use the patterns described above to elegantly emulate and augment MIDI capabilities. A note on message translate to

    {
        "type": "create",
        "component": "daw1",
        "channel": "pad2",
        "instance": "123e4567-e89b-12d3-a456-426655440000",
        "parameters": {
            "pitch": 440.0,
            "volume": 0.5433435,
            "shine": 0.232838,
            "attack": 0.642466
        }
    }

There are obvious advantages over MIDI here. Channels can have meaningful names and their number is not limited by the standard. Volume has float precision, pitch is a float which allows it to address non-western or micro-tonal pitching or raw frequencies, and additional parameters can be added to alter the synthesiser parameters for each individual note. MIDI control messages fit just as nicely.

{
    "type": "control",
    "component": "daw1",
    "channel": "pad2",
    "instance": "123e4567-e89b-12d3-a456-426655440000",
    "parameters": {
        "pitch": 440.0
    }
}

A MIDI pitch bend, for example, can be obtained by sending control messages with a changing pitch parameter, or a bend parameter to specify a pitch offset. A MIDI note off is an instance destruction, for example

{
    "type": "destroy",
    "component": "daw1",
    "channel": "pad2",
    "instance": "123e4567-e89b-12d3-a456-426655440000",
    "parameters": {
        "release": 0.342342
    }
}

In comparison to MIDI, the additional flexibility in defining a synthesizers attributes and control mechanism comes at a cost of complexity for the synthesizer itself. But it is a price worth paying, the growing capability of synthesisers to explore exotic soundscapes make the shortcomings of MIDI ever more apparent. It is up to the synthesizers to document the parameters which they support (as described in the previous section), and up to the controller to adapt.

Describing Sound

A channel can be used to transport sound analysis parameters for a given track in a DAW (or any audio application). Note the absance of an instance attribute means the parameters are applied for the static channel as opposed to a dynamic instance.

{
    "type": "control",
    "component": "daw1",
    "channel": "bass",
    "parameters": {
        "pitch": 440.0,
        "spectral-bins": [0.265434, 0.634633, 0.453434, 0.328053],
        "spectral-diff": [0.084524, 0.134308, 0.334342, 0.024233],
        "spectral-crest": 0.325342
    }
}

Describing Virtual Object Manipulation

Virtual object manipulation is of central importance to the Sonosthesia project, it is a complex emerging subject. There is little pre-existing work in terms of protocols describing them. The same messaging pattern is applied here also, for describing contacts. The channel represents an object in the virtual world (sphere, teapot etc...). A contact is created when agent objects (hand avatars, sticks...) collide with the channel.

{
    "type": "create",
    "component": "vr1",
    "channel": "sphere1",
    "instance": "123e4567-e89b-12d3-a456-426655440000"
    "parameters": {
        "incidence": 0.1765,
        "velocity": 10.7534,
        "hardness": 0.3342
        "wood": 0.2325,
        "metal": 0.4592
    }
}

There are two required parameters which are the incidence (see angle of incidence) and the velocity. Note that in this case 3D vectors are abstracted away, incidence and velocity should be enough dynamic information. This means that the same message format can be used for both 2D and 3D environments, there is no dimensionality change.

Other parameters are defined by the surface properties of the colliding objects at the point of collision. Although the value of these surface properties can be encoded in the color components or UV coordinates, they should really be translated into behavioural meaning by the emitting component which will help the mappings to other domains. In other words, parameter names such as wood, metal and hardness should be preferred to red, or . Once the contact is initiated, it can persist if the colliders stay within a given range of each other, in order to emulate actions like scratching, scraping, rubbing. During that time the parameters can be constantly updated by factory control messages.

{
    "type": "control",
    "component": "vr1",
    "channel": "sphere1",
    "instance": "123e4567-e89b-12d3-a456-426655440000"
    "parameters": {
        "incidence": 0.4982,
        "velocity": 3.7534,
        "hardness": 0.6732
        "wood": 0.5312,
        "metal": 0.4232
    }
}

When the colliders overrun the set contact distance threshold, the contact is destroyed following the factory messaging pattern (note the negative velocity implying that the colliders are moving away from each other).

{
    "type": "destroy",
    "component": "vr1",
    "channel": "sphere1",
    "instance": "123e4567-e89b-12d3-a456-426655440000"
    "parameters": {
        "incidence": 0.4982,
        "velocity": -3.7534
    }
}

Extending to Other Domains

These are only a few concrete simple examples, the protocol can be applied to any kind of system which can expose input and output channels with numerical parameters.