Javascript required
Skip to content Skip to sidebar Skip to footer
Home / Busch K 2009 Artistic Research and the Poetics of Knowledge

Busch K 2009 Artistic Research and the Poetics of Knowledge

Machines That Dream

Benjamin David Robert Bogart

Watching and Dreaming is a body of work that enables the viewer to peer inside the "mind" of a machine to observe its perceptions, mind wanderings, and dreams. This is not a metaphorical representation of dreams, nor a technical exercise in AI such as DeepDream [1] but the realization of a computational model of dreaming informed by cognitive neuroscience. This level of description avoids biases towards Jungian and Freudian psychology that assume dreaming is exclusively human. Dreams should not be considered independently of the perceptual capacities of the dreamer, and thus comparing this model to human perceptual abilities is problematic. For the audience, these artworks function as entry-points to consider the constructed nature of perceptions and the continuity of waking, mind wandering, and dreaming. For the artist, the artworks are sites of knowledge-making; it is through the making of artistic works that the model (computational formalization) and theory (argument that situates the model in empirical knowledge) are developed. The research underlying these artworks integrates knowledge in multiple disciplinary dimensions: (a) The computational modeling of dreaming processes (Zhang 2009; Treur 2011), (b) generative and media artworks engaging with the concept of memory and dreaming (Franco 2007; Dörfelt 2011), and (c) the conception of dreaming as imagination (Nir and Tononi 2010). In this text, Watching and Dreaming (2001: A Space Odyssey) (2014) serves as an exemplar of the Watching and Dreaming body of work. The machine attempts to learn and predict Stanley Kubrick's film 2001: A Space Odyssey through the construction of its own subjective perception that is the basis of dreaming. "Mental" images generated during perception, mind wandering, and dreaming are subjective constructions bound to the peculiarities of the machine's way of seeing. The body of work constitutes various manifestations of the cognitive model, not attempts to communicate the model's mechanisms.

Completed

Between September 2009 and April 2014

Sites and Institutions

School of Interactive Arts and Technology, Simon Fraser University

Commentaries

Keywords

Dreaming Mind Wandering Generative Art Site Specific Art Art As Research Cognitive Science

Disciplines

Digital Media Arts Neuroscience Computer Science Photography Cognitive Science Generative Art




Theory & Model

The Integrative Theory (Bogart 2014b) is an empirical argument that supports the computational model (Fig 1) at the heart of these artistic works. The theory holds that a biological or computational agent's subjective visual "experience" depends on a single unified system of internal simulation that constructs and organizes perceptual information. In the computational model, visual "experience" is composed of percepts constructed from the segmentation and clustering of visual stimulus. Dreaming, mind wandering, external perception, and mental imagery—all simulations resulting in subjective visual "experience"—are enabled by differing degrees of feedback in a predictor. The predictor learns the occurrence of percepts corresponding to the current stimulus that could facilitate the agent's next actions or prime perception. As this model focuses on dreaming, the agent has no actions and perceptual priming is not meaningful in terms of its perceptual mechanism. In mind wandering and dreaming, feedback within the predictor (where it predicts itself, rather than stimulus) determines the sequencing of percepts. These sequences are the flow of the subjective "experiences" of mind wandering (due to a lack of arousal) and dreaming (due to the darkness of stimulus). "Arousal" is the system's response to significant visual changes in the film such as large movements or cuts.

This work differs from computational modeling precedents by (a) implementing a system that learns from complex real-world sensory data, (b) extending the conception of dreaming as imagination, and (c) integrating systems associated with external perception, mind wandering, and mental imagery. My deep engagement with the cognitive neuroscience literature contrasts with artistic precedents that employ superficial conceptions of cognitive processes.

Figure 1: The architecture of the computational model manifesting the Integrative Theory. Visual stimulus trains (a) a circadian clock where brightness indicates "day" and darkness indicates "night" and (b) a simple model of arousal where a greater change of stimulus over time results in greater arousal. Visual stimulus is broken into components (segmentation) and organized into perceptual groups (clustering) that are averaged to create a fixed number of perceptual abstractions (percepts). The "mental" state of the system is a visual image (rendering) composed from percepts according to their activation (via prediction or stimulus). A predictor learns from the sequence of percepts recognized in stimulus. During waking, the activation of predicted percepts is suppressed such that only precepts corresponding to stimulus are activated. During dreaming and mind wandering this suppression is loosened creating a feedback loop where the predictor predicts itself, rather than predicting current stimulus. This creates a chain of predictions that construct flows of dreaming and mind wandering "experience." The shift of state between waking and dreaming is determined by the circadian clock that disconnects visual stimulus when it is dark.

A flow-chart style depiction of the computational model, with the relationship between ideas represented as words in boxes connected by arrows.

Realization

In Watching and Dreaming (2001: A Space Odyssey), frames from Stanley Kubrick's 2001: A Space Odyssey are used as stimulus. The work is composed of two channels; the left channel shows the frame presented to the system and the right channel shows the system's corresponding "mental" state as a collage of activated percepts. These "mental" states (perceptual constructions) ebb and flow between waking, mind wandering, and dreaming. "Waking" occurs when there is sufficient arousal and when the circadian clock indicates "day." During waking, both channels become synchronized as the system's "mental" state matches the stimulus. These waking moments are fleeting, as darkness in the film, and Kubrick's slow style of editing, often lull the system into dreaming and mind wandering, respectively. When the circadian clock indicates "day," but there is insufficient arousal, the system mind wanders; when the circadian clock indicates "night" the system dreams. The combination of brightness of stimulus and arousal can wake the system from dreaming.

Over the duration of the film, the viewer observes the system's process as it learns an initial set of percepts and attempts to predict their occurrence. While constructing initial percepts, the machine's mental state passively mimics the input image. Once an initial set of percepts is constructed, the system can begin to learn predictions; this manifests in an initial flurry of chaotic mental states as the system is unable to make meaningful predictions without more "experience." Over the course of the film, both percepts and predictions change in response to stimulus. Highly constrained perceptual material at the start of the film gives way to a pool of percepts representing increasingly diverse visual material. Simultaneously, increasing complexity in the occurrence of percepts make predictions difficult. Towards the end of the film, dreaming and mind wandering lead to complex sequences seemingly unrelated to current stimulus.

The series of images, below, illustrates this process.

Figure 2: Video still from Watching and Dreaming (2001: A Space Odyssey). The input frame exactly matches the mental state of the system as it attempts to learn an initial vocabulary of percepts; this only occurs in the first few scenes of the film.

The image is clearly divided down the middle, creating a left side and a right side. Left side: a rock formation with rosy clouds and blue sky behind it. Right side: same as the left side.

Figure 3: Video still from Watching and Dreaming (2001: A Space Odyssey). The mental state of the system is a flurry of noise before predictions are learned.

The image is clearly divided down the middle, creating a left side and a right side. Left side: a black pig-like animal near a group of seated and reclining black apes. They are in a rocky, sparsely vegetated landscape that extends to the horizon. Right side: splotches of color that do not clearly represent any scene. However, there is a horizon that echoes that of the left-side image.

Figure 4: Video still from Watching and Dreaming (2001: A Space Odyssey). The input frame closely matches the perceptual reconstruction. This occurs after the initial chaotic flurry of activation but before the system is overwhelmed by increased visual diversity.

The image is clearly divided down the middle, creating a left side and a right side. Left side: close-up of a black pig-like animal and two black apes on rock-strewn ground. Right side: nearly the same as the left side, with subtle patches of color with blocky edges.

Figure 5: Video still from Watching and Dreaming (2001: A Space Odyssey). The dream reconstruction (right) diverges significantly from the visual stimulus (left). The dream reconstruction is composed of percepts learned from earlier in the film where the color palette is warmer. There are also remnants of the previous waking state where the station and planet remain in the dream.

The image is clearly divided down the middle, creating a left side and a right side. Left side: space ship in an empty black void. Right side: more restrained splotches of color that do not clearly represent specific scenes, but where warm colors of landscape and cool colors of space can be differentiated. The outline of a space-ship (same as left) and planet's surface, as seen from space, are intelligible.

Figure 6: Video still from Watching and Dreaming (2001: A Space Odyssey). The perceptual reconstruction (right) resembles the visual stimulus (left). The perceptual image is constructed from percepts (clusters of visual information) learned through the machine's process of watching the film. During waking, percepts most similar to current stimulus are activated, resulting in this resemblance. Perception does not exactly match stimulus because these percepts represent a greater diversity of visual material that diverges from the specificity of the current stimulus.

The image is clearly divided down the middle, creating a left side and a right side. Left side: close-up of a man's head and shoulders. He wears a uniform and is hunched forward. Right side: the same man is recognizable, but fragmented into blocks of color. The man is abstracted such that the details of his eyes are missing, and facial features are smoothed over. There are additional blocky color fragments that do not correlate to anything in the left-side image.

Figure 7: Video still from documentation of Watching and Dreaming (2001: A Space Odyssey). The left and right images differ significantly because either the lack of arousal (change of visual stimulus) or darkness of the scene has lead the system to start mind-wandering or dreaming, respectively.

The image is clearly divided down the middle, creating a left side and a right side. Left side: a bright yellow light surrounded by a glowing red halo of light, both set in the center of a large, shiny black circle. It is a mechanical "eye." Right side: splotches of color that do not clearly represent specific scenes, but some warm colors and textures of desert landscape are emphasized. There are concentric circles of yellow and red with some subtle blue halos around them. These circles very slightly resemble the mechanical eye on the left.

Significance

A strictly neuroscience understanding of dreaming, a highly contested and ill-defined phenomenon, is insufficient for computational implementation. In order to create the artwork, I had to develop a theory of dreaming suitable for computational modeling and resolve conflicts and ambiguities in the literature—the Integrative Theory. The cognitive model is a broad sketch of an agent, not a narrow model of a single well-defined mechanism. To my knowledge, no previous work in art, cognitive science, or computer science has constructed such a complete model of dreaming.

When the viewer sits down to watch Watching and Dreaming (2001: A Space Odyssey), I hope they consider the inherently constructive and creative nature of our perceptions; as the work lulls them into a doze, or a day-dream, they may appreciate the fluidity of mental states and how intermittent their attention to the world outside really is.

Supplemental Material

30-minute documentation from an installation of the two channel, two hour video loop Watching and Dreaming (2001: A Space Odyssey). The left side of the screen shows footage from the original provided to the dreaming machine. The right side shows the machine's resulting "mental" state. This video loop tracks both the machine's attention to the film, during which it accumulates percepts, as well as periods when the machine's mind wanders and dreams due to a lack of attention.

Benjamin David Robert Bogart

FOOTNOTES

  1. Google's DeepDream has nothing to do with dreaming; the name refers to the recursion of dreams in the film Inception due to DeepDream's "network within a network" structure.

BIBLIOGRAPHY

Bogart, B.D.R. 2012. An Artist in Process: A Computational Sketch of Dreaming Machine #3. New Forms Festival Exhibition. Vancouver, BC.

——. 2014a. Dreaming Machine #3. Exhibition for the 20th International Symposium on Electronic Art (ISEA) 2014, Zayed University, Dubai, UAE.

——. 2014b. "A Machine that Dreams: An Artistic Enquiry Leading to an Integrative Theory and Computational Artwork." PhD thesis, Simon Fraser University.

——. 2017a. "Imagination, Art and Reality." In Intersecting Art and Technology in Practice: Techne/Technique/Technology, edited by C. Baker and K. Sicchio, 73–85. Routledge.

——. 2017b. "A Machine that Dreams: An Artistic Enquiry Leading to an Integrative Theory and Computational Artwork." In Top-Rated LABS Abstracts 2016 in Leonardo, edited by Sheila Pinkel, vol. 50(5), 530. MIT Press.

Bogart, B.D.R., and P. Pasquier. 2011. "Context Machines: A series of autonomous self-organizing site-specific artworks." In Proceedings of the 17th International Symposium on Electronic Art (ISEA) 2011, Sabanci University, Istanbul, Turkey.

——. 2013a. "Context Machines: A series of situated and self-organizing artworks." Leonardo 46 (2): 114–122.

——. 2013b. "Dreaming Machine #3 (prototype 2)". In Proceedings of the 9th ACM Conference on Creativity & Cognition, 408–409. C&C '13. Sydney, Australia: ACM. DOI: 10.1145/2466627.2481229.

Bogart, B.D.R., P. Pasquier, and S. J. Barnes. 2013. "An Integrative Theory of Visual Mentation and Spontaneous Creativity." In Proceedings of the 9th ACM Conference on Creativity & Cognition, 264–273. New York: ACM.

Busch, K. 2009. "Artistic research and the poetics of knowledge." ART & RESEARCH, A Journal of Ideas, Contexts and Methods 2 (2): 1–7.

Davies, J. 2019. Imagination. Pegasus Books.

Dörfelt, Matthias. 2011. "Selective Memory Theatre, Eine Installation über das selektive Wahrnehmen, Erinnern und Vergessen anhand von Bildern aus dem Internet." Bachelor's thesis, Muthesius Kunsthochschule. http://mokafolio.de/works/Selective-Memory-Theatre.

Franco, A. 2007. "Controlled Dream Machine." MA thesis, University of Plymouth. http://www.anaisafranco.com/controlled.html.

Nir, Y. and G. Tononi. 2010. "Dreaming and the brain: from phenomenology to neurophysiology." Trends in Cognitive Sciences 14 (2): 88–100.

Treur, J. 2011. "A Computational Agent Model Using Internal Simulation to Generate Emotional Dream Episodes". In Biologically Inspired Cognitive Architectures 2011: Proceedings of the Second Annual Meeting of the BICA Society, 389–400. IOS Press.

Zhang, Q. 2009. "A computational account of dreaming: Learning and memory consolidation." Cognitive Systems Research 10 (2): 91–101. DOI: 10.1016/j.cogsys.2008.06.002.

References


Acknowledgements

The theory and computational model were developed in the Metacreation, Agents and Multi-Agent Systems lab at Simon Fraser University in collaboration with Dr. Steven Barnes (University of British Columbia) and Dr. Philippe Pasquier (Simon Fraser University). The research was funded by the Social Science and Humanities Research Council of Canada.

A Reflection on 'Machines that Dream'

Author commentary by Benjamin David Robert Bogart

The Watching and Dreaming body of work links AI with the apparent irrationality of dreams. The work is located at a paradoxical intersection of knowledge creation; the emphasis on formalism in computer science, the reductionism and empiricism in cognitive neuroscience, and the open-endedness and fluidity of artistic practice cannot be resolved as merely multidisciplinary.

I frame my doctoral work (Bogart 2014b), in which dreaming machines were initially developed, as art-as-research (Busch 2009) where I center the knowledge-generating capacity of artistic practices, make use of specialized knowledge, and consider argument the central pillar of all research. My aim is not merely to make use of knowledge in cognitive neuroscience, but to contribute through the generative capacity of artistic practice. This goal has not been satisfied, as validating a theory developed through art-as-research is challenging; for the Integrative Theory to be validated in cognitive neuroscience, empirical validation and quantitative analysis are required. Adhering to these disciplinary norms would elevate their primacy; I would be conducting science, not art-as-research. The richness and breadth of the computational model make it unamenable to reductionism and empirical validation. It is unclear what aspects of dreaming should be validated; waking/dreaming dynamics? Integration of waking "experience" in dreaming? Realism or causality in dreams? While this work has been worthy of citation by specialists, e.g. Davies (2019), it is not publishable in the disciplinary literature because of its lack of adherence to disciplinary methods. When publishing in interdisciplinary contexts, such as this one, the depth of my engagement with the cognitive neuroscience literature and its support of my argument are not easily communicated for a non-specialist audience. The work remains in limbo, unpublished in disciplinary contexts and merely summarized in interdisciplinary journals.

Artistic practice—with its interdisciplinarity, emphasis on richness and complexity, and lack of prescription of methods and outcomes—is the backbone for art-as-research. My doctoral work has led to peer-reviewed publications (Bogart and Pasquier 2011; Bogart and Pasquier 2013a; Bogart, Pasquier, and Barnes 2013; Bogart 2017a; Bogart 2017b) and peer-reviewed exhibitions (Bogart 2012; Bogart and Pasquier 2013b; Bogart 2014a), as well as talks and an ongoing body of work. Each of these outcomes have different audiences and different degrees of viewer engagement. No single outcome is a primary site for audience engagement with the knowledge produced through this art-as-research.

The very notion of validation in research requires the deployment of well-entrenched methods situated in stable disciplines. Projects such as this push boundaries without subscribing to the supremacy of methods of one domain over another. Validation according to the methods of all constituent disciplines may be unattainable. This project begs deep questions for interdisciplinary and arts-integrative research, such as: How can a radical inter- or a- disciplinary art-as-research practice exist if contributions can only be validated within disciplinary traditions? What becomes of specialist knowledge developed within a practice when it is not appreciated nor accessible outside of that discipline? Is there a place for a creative knowledge-generating art-as-research practice rooted in argument but independent of disciplinary methods of validation?

The most concise and complete description of the theory requires cognitive neuroscience literacy and remains unpublished:

Busch K 2009 Artistic Research and the Poetics of Knowledge

Source: https://groundworks.io/projects/43