Since 2016 I have been presenting an interactive installation controlled by software that I have written (initially I used Processing, but now, as of 2019 I have migrated to full Java – I am also using SuperCollider for the audio and Arduino for the sensors).
Participants are initially presented with a projection screen, speakers and a touch sensor table. The sensors control the visual material that is presented on the screen and the audio material heard through the audio system.
(the two photos above are from Nuit Blanche North 2017)
More details about this project are available at each of the links below but briefly, what this project is about:
I am interested in studying human nature, consciousness, living presence and I am using this installation as the medium to accomplish this. A theme and a source image to represent it is chosen and then triangulated. This process of triangulation is just one way to translate living presence into a precise numerical form.
This triangle mesh, the ‘NETTT’, also serves as the interface, the instrument through which all activity takes place. Sensors on a table are activated by participants touching them triggering events: triangles filling in with colours and images along with the playing of accompanying samples or notes. ‘Playing the NETTT’ – playing the essence of the original source image as an instrument, as a non-linear sample player. The sonic material that populates the triangles is a composition I have created and sounds I have collected all relating to a theme – likewise for the images. The triangle mesh is gradually filled up by the participant’s actions thereby generating a collective artwork, a collective re-mix of the material. While this is going on, the software is recording which sensors were activated and when producing a table of data – data which serves as a precise document of collective psyche. All of the factors that contribute to this activity: the initial theme, the audio and visual work I have produced to communicate that theme and that populates the triangle mesh, the environment within which the installation is presented, the participant’s response to that theme through their interactions – all of these factors are encoded in the fairly lean table of data generated by the sensors and recorded by the software. Below is a screenshot of that data.
An example of a portion of a recording in data form (text file is 370 lines long). Numbers on the left are which triangles were activated and the numbers on the right are when, in milliseconds since the start time.
This is one of the main functions of this installation, to generate data based on the particpants’ activity – to generate a precise document, a table of numbers that represents living creative activity. The elaborate processes I have developed to acquire this data is so that it is produced by meaningful and inspired activity. The more focused the activity, the more vivid the results.
What you can do with this data is an open question and there are many avenues I am currently exploring to answer that question. It is no accident that I have chosen the graph as the core architecture. I am continually elaborating on its use as a network. I am exploring self-organizing, non-linear, dynamical systems and this installation was designed with that in mind – to serve as the foundation for staging these interactions and exploring their possibilities. Ultimately I seek to generate, through participant interaction, emergent phenomenon which I believe is the key to illuminating the unknown – the window into hidden worlds, worlds that constantly surround us but remain invisible, both inner and outer, with consciousness, psyche serving as the channel. This is the magic of art, science, programming. I use the triangle mesh as the instrument – the medium capable of recording, storing, transforming, and transmitting the data of living presence and the systems, algorithms, and mathematics that I am employing articulate these means.
Some early attempts at visualizing and sonifying the data can be found in the 2017 and 2018 links below – knots and trees for example. Each recording produces a different knot, a different tree, a different signature, a different ‘reading’.
Using artwork as a document of psyche.
an empty noise field – a rendering of nothing
first iteration – 2016:
The first presentation of this software. An initial source image was triangulated and the resulting mesh was used as the interface and the instrument through which all activity took place: recording, storage, transformation, and playback. Mouse interaction and simple triangle playback and recording. During development, a lot of attention went to translating a line drawing produced in Illustrator into a field of triangles that can be played like an instrument. A ‘non-linear sample player’.
second iteration – 2017:
Much was added to and elaborated upon this year: the addition of the controlling sensor table, two modes of analysis, sensors, and rgb, a new system for recording and playing back triangles, and a way of loading source images that resulted in a triangle mesh collage.
third iteration – 2018:
presented at two events:
A lot of time and energy went into refactoring the software. I added a ‘painting’ mode, whereby the triangles are replaced by ‘brushstrokes’ of colour that gradually fill in to form a painting. Two new modes of analysis were added: knots and trees along with vertices and edges for playback. Started using SuperCollider to generate synthesized sounds.
fourth iteration – 2019:
Currently under development. Expanding the software to allow participants to use the triangle mesh as an interactive synthesizer/sequencer. Also adding new modes of analysis such as 3-D trees. I will be presenting this installation at Electric Eclectics from August 2 to 4 this summer in Meaford, Ontario.