Art, technology and the Internet of Living Things

Abstract

Intelligence augmentation was one of the original goals of computing. Artificial Intelligence (AI) inherits this project and is at the leading edge of computing today. Computing can be considered an extension of brain and body, with mathematical prowess and logic fundamental to the infrastructure of computing. Multimedia computing-sensing, analyzing, and translating data to and from visual images, animation, sound and music, touch and haptics, as well as smell-is based on our human senses and is now commonplace. We use data visualization and sonification, as well as data mining and analysis, to sort through the complexity and vast volume of data coming from the world inside and around us. It helps us 'see' in new ways. We can think of this capacity as a new kind of "digital glasses". The Internet of Living Things (IOLT) is potentially an even more profound extension of ourselves to the world: a network of electronic devices embedded into objects, but now with subcutaneous, ingestible devices, and embedded sensors that include people and other living things. Like the Internet of Things (IOT), living things are connected; we call those connections "ecology". As the IOT becomes increasingly synonymous with the IOLT, the question of ethics that is at the centre of aesthetics and the arts will move to the forefront of our experience of and regard for the world in and around us.

Keywords: Immersive digital media; Implicit interaction; Internet of Living Things; Physical computing; Tacit knowledge.

Fig. 1

One of these networks takes in raw energy, transforms it into life, cools the Earth, cleanses water, absorbs CO₂, and releases oxygen. The other does the reverse. Photograph of Sao Paulo, Brazil by Ana Paula Hirama—Flickr: Passeio de Helicóptero em São Paulo, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=21405720. Photograph of Lower Lewis River Waterfall, https://www.peakpx.com/en/hd-wallpaper-desktop-orwom

Fig. 2

In Morocco Memory II (1999), a multimedia architectural installation, seeds and spices were placed inside 6 wooden boxes fitted with custom embedded systems as part of a real-time interactive process. When the boxes were opened and closed by a user, the system kept track of the number of boxes and order of choices, and responded by triggering navigation of story elements comprised of more than 10,000 texts, sounds, images, and movie fragments. Aromas from the spices mixed and triggered memories in the mind of the user, which informed the users’ choices. The result was a re-combinant multimodal narrative that due to the large number of elements resulted in an emergent system in constant transformation. Morocco Memory II was installed at the Interactive Frictions Exhibition as part of the Labyrinth Initiative on Interactive Narrative, Fisher Gallery, University of Southern California, June 5–18, 1999. Photographs by Vibeke Sorensen.

Fig. 3

Seeds are actual genetic memory, part of human life and culture, intricately connected with our survival and memory. The seeds in Morocco Memory II were spices that provided scent and taste. Photographs by Vibeke Sorensen

Fig. 4

Sanctuary (2005) installation at Gallery 1-1-1, University of Manitoba, Winnipeg, Canada. Plants, embedded systems, capacitance and custom hardware were included as primary elements of a real-time responsive environment. Any human touch of plants provided signals that the plant cells responded to in biofeedback with the system. Signals from the hands were sent through the leaves to the stems. A metal wire was placed into the dirt near the roots detected the plant’s signals, which was sent to the computer and converted into sound, light and navigation pathways for the mixing and display of lights, images, music and movies. There was also a bowl with water in it that detected proximity of users, and copper coated benches that sent signals to the system when people sat down or arose. The system traversed multimedia content in intersecting tree structures and included thousands of original recordings of spiritual and natural sanctuaries around the world. Sorensen received a 2001 Rockefeller Foundation Fellowship in Film/Video/Multimedia to support the creation of Sanctuary. Additional support was provided by the Midi@rte Laboratory, School of Fine Arts (EBA) at the Federal University of Minas Gerais (UFMG), Belo Horizonte, Brazil. Photograph by Vibeke Sorensen

Fig. 5

When no people were inside the Sanctuary installation, the system continued to run in a “dream” state that associated and layered mediated memory fragments as a montage. In this case, the photographer was detected near the water bowl (not seen) which “awakened” the system and responded with images of the Amazon river in Brazil. Small pubescent or “fuzzy” plants networked to the system can be seen on the low table in the middle. Photograph by Vibeke Sorensen

Fig. 6

“…everything thrives throughout the whole world. And the king abstains from killing living beings, and other men and those who are huntsmen and fishermen of the king have desisted from hunting. … in opposition to the past also in the future, by so acting on every occasion, they will live better and more happily.” Khandahar Rock Edict of Emperor Asoka, 268–232 B.C.E. Photograph: https://commons.wikimedia.org/wiki/File:AsokaKandahar.jpg

Fig. 7

“Formerly in the kitchen of King Devanampriya Priyadarsin (one of the titles of Asoka) many hundred thousands of animals were killed daily for the sake of curry. But now, when this rescript on morality is caused to be written, then only three animals are being killed (daily), two peacocks (and) one deer, but even this deer not regularly. But even these animals shall not be killed (in future).” First Kalsi Rock-Edict of Emperor Asoka, 268–232 B.C.E. Translation: Hultzsch, Inscriptions of Asoka. New Edition. In: Corpus Inscriptionum Indicarum vol. I. Oxford: Clarendon Press, 1925. Photograph by Mishra.prashant89—Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=51442997

Fig. 8

Illuminations (2013) Interactive architectural installation by Vibeke Sorensen. An illuminated folding screen, 30 m wide and arched, it included living plants where their O₂/CO₂ was detected as part of a multimedia biofeedback system. Plant biorhythms interacted with real-time visual music that included Tibetan singing bowls. The plants “sang” melodies that affected changes in colours, textures, 3D shapes, and movement, which in turn affected their growth and biological rhythm. Photograph of Christine Veras playing the singing bowls, by Vibeke Sorensen

Fig. 9

From the Illuminations installation. Selection of plants for biofeedback was informed by the traditional Tibetan Buddhist medicine mandala on the left, which was also on the cover of the book, A Clear Mirror of Tibetan Medicinal Plants (1999) by Dawa Galypo

Fig. 10

From the Illuminations (2013) installation at the ADM Gallery, NTU Singapore. Some of the plants are visible on the bottom left. Photograph by Vibeke Sorensen

Fig. 11

Plants with sensors and interactive Pure Data (Pd/GEM) programme used for the Illuminations (2013) installation at ADM, NTU Singapore. Biofeedback from the plants was mapped to real-time computer music and animation using CO₂/O₂ sensors, Arduino and Pure Data (Pd/GEM) software. Photograph by Vibeke Sorensen

Fig. 12

The Mood of the Planet (2015) installation was inspired by a Pre-Columbian archaeological site in Tiwanaku, Bolivia which connects the movement of the heavens with life on Earth. On the winter solstice in the southern hemisphere, June 21, the sun comes up in the middle of a portal, casting light onto people present who raise their hands in ritual to take in the Sun’s renewed energy and put them into harmony with nature and the Universe. Photograph of Vibeke Sorensen in Tiwanaku on 21 June 2002 by Heitor Capuzzo of UFMG, Brazil

Fig. 13

The Mood of the Planet installation was laid out as a portal or arch, like Tiwanaku, and represents a passage to greater understanding of our existence on the Earth and in the Universe as a metaverse where people of all walks of life are increasingly interconnected to each other and the vast physical-digital environment, like the Asian concept of One-ness. Diagram copyright 2015 Vibeke Sorensen

Fig. 14

The Mood of the Planet under development. The larger image on the right shows the inside of the arch, including “digi-tiles” made of crushed recycled glass, LEDs, and custom electronics. The 3 small vertical images on left show the “tunnel” created by reflections in mirrors on either side of the arch sculpture. Photographs by Vibeke Sorensen

Fig. 15

The Mood of the Planet (2015) installation in the ADM Gallery, NTU Singapore. Photograph by Nagaraju Thummanapalli

Fig. 16

The Mood of New York (2016) installation, at the Experiments in Art and Digital Technologies (EADT) Exhibition in 2016, a 2 m × 0.5 m LED custom display with original software, animation and music by Vibeke Sorensen. PureData (Pd/GEM) programming by Vibeke Sorensen. Custom Python programming and hardware development by Nagaraju Thummanapalli, and with assistance from Dennis Low Peng Thiam. Photograph by Suzanne Ball

Fig. 17

Digital Amulet: Smart Necklace (2017). Photograph by Vibeke Sorensen

Fig. 18

In Other Wor(l)ds (2018) installed at the Beyond Festival at ZKM in Karlsruhe, Germany. Digitally printed tapestries with embedded LEDs and networked fans. 6 layers of large (6 × 5.3 m) 2-sided fabrics were created to display real-time temperature and wind data from 5 global environmental biomes as an immersive multimodal experience. (2 of the 6 layers were exhibited at ZKM.) Colour was coded to temperature (blue to white = cool to cold, and yellow to red = warm to hot). Wind speed was mapped to the speed of movement of coloured light across the LEDs. An online app allowed viewers both to track and select the biomes. By translating live sensor data into moving coloured lights and air movement, the fabrics could move in the wind like curtains in a window and brush against the skin, with LEDs bathing the viewer in coloured light. Photograph by Vibeke Sorensen

Fig. 19

Anthropologist J. Stephen Lansing with Punan people in Borneo, 2019. Photograph copyright J. Stephen Lansing

Fig. 20

37 GPS tracks from 27 different Punan adults in their forests wearing Sorensen’s belts with GPS trackers, collected over 3 four-week periods (5 blue shades—collection 1, 15 red shades—collection 2, 17 purple shades—collection 3). Image courtesy of Dr. Guy Jacobs

Fig. 21

Punan woman wearing bark cloth in the rainforest of Borneo, Indonesia, 2019. Photographs by J. Stephen Lansing

Fig. 22

The Tree Dress (2020). A tree in the rainforest in Singapore was photographed using panoramic digital techniques, and the images were digitally stitched together and printed onto sustainable silk fabric. The same tree was tracked in real-time using networked sensors at the site of the tree. Data was transmitted wirelessly to a remote receiver, and custom Arduino/Lilypad soft circuits sewn into the dress converted it into changing colours displayed on the LEDs embedded in the dress, with data sonification and music played in wirelessly connected small speakers. A web app allowed people to track the tree and the Tree Dress online in real-time. Photographs by J. Stephen Lansing and Vibeke Sorensen