TOLAS: Terrain of Light and Sound

Anna M. Chupa

Department of Art and Architecture, Lehigh University

anna.chupa@lehigh.edu

Michael A. Chupa

GeoResources Institute, Mississippi State University

chupa@gri.msstate.edu

Abstract

We have previously explored generative techniques in pattern application in a planar tiling, and alluded to future work involving exploration of the Platonic clumsy butcher (Phaedrus 265 d,e) In the present work, a taxonomy describing texture source imagery is employed in a generative process to synthesize new textures based upon a library of preselected sources.

The setting of TOLAS: Terrain of Light and Sound is one in which generative and stochastic processes govern the presentation of potential choices to an artist. In the absence of any choice, this presentation unwinds in a tapestry-like animation. However, once the artist makes a selection, there is at once a collapse of the multiplicity of possible futures and a penetration into possibilities ignored or avoided in previous choices. The random processes guiding the temporal evolution continue to act even in the presence of conscious choice, insuring that there will be sufficient variety available to the artist.

To provide a sense of artistic immersion in TOLAS’ textured terrain, the animation engine maps texture imagery onto a logical grid created by selecting a mesh from two of the N dimensions available as texture descriptions. Navigation along this reduced dimension mesh then takes place automatically when the artist makes selections; some of these selections will transform the camera path to another pair of dimensions. To avoid a tedious march of textures along a flat surface, stochastic processes govern animation of mesh control points. The animation engine is also coupled with TOLAS’ audio generation engine, which provides loose synchronization between the sound and visual design elements in the animation view. Changes in the virtual ensemble of the audio engine track the navigation paths induced by the artist’s choices.

1. Introduction

We have previously explored generative techniques[1] in pattern application in a planar tiling, and alluded to future work involving exploration of the Platonic clumsy butcher (Phaedrus 265 d,e) In the present work, a taxonomy describing texture source imagery is employed to generate new materials for evolutionary texture generation. We amplify this taxonomy in our current work with a temporally based system that presents a tapestry of potential forms to an artist, who then makes serendipitous or fateful choices. Limitations of efficacy of arbitrary taxonomic descriptors is offset by the intelligent selector’s aesthetic choices, which are informed by that selector’s personal history and unconscious participation in the multiplicity of possible paths. As Socrates explains in the Phaedrus, “perfection [is] … partly given by nature, but may also be assisted by art.”[2]

This work presents an exploration of the innate tension between an autonomous generative process responsible for synthesis of instances and an observer who judges the aesthetic qualities of these novel instances. Since an artist typically has a role in both the generative process definition and in evaluating the output against some measure of aesthetic fitness, the tension alluded to above results from the feedback loop created by the artistic process. The generative process employed here is analogous to the intuitive enumeration of possibilities and selection of a particular course in executing a creative process.

2. Surface Textures and Texture Synthesis

We choose to explore the domain of two-dimensional texture maps in TOLAS. The choice of a texture map size is conditioned by several constraints: available graphics memory, frame rate requirements for interactive display, texture dimension limitations[3], and retention of sufficient surface detail to satisfy aesthetic prerogatives. A standard map of n by n pixels for our texture source material yields a universe of possible textures of cardinality U, with

(1)

where c is the number of channels in the color model employed, and b is the number of bits / channel. For our nominal standard size of 256 x 256 pixels in RGB with 8 bits per pixel, our universe of possible textures numbers about 1.6 million possible textures. A brute-force method for generating a new texture given two such textures as sources would involve exploration of a Cartesian product space with more than 10¹² elements. This purely reductionistic measure of our sample space is unmanageably large and fails to capture any semantic content of our texture images. Thus, we need to find effective methods for reducing the dimensionality of our texture descriptions while incorporating meaningful semantic content descriptors for later use in a generative process for texture synthesis.

Selecting tiles from file directories for bodies of work produced over a 10-year period established enough diversity in image content, formal considerations (e.g., color harmonies, symmetry types), and work method to assist in the development of a taxonomy sufficiently broad to test TOLAS against an idiosyncratic and largely intuitive method of constructing images.

With few exceptions, all texture tiles were derived from photographs. Although the identity of the photograph is often obscured by the subsequent abstraction that occurs from selecting and cropping photographic details, that identity is still critical to the construction of the content taxonomy. To simulate the intuitive process that occurs at the beginning of image development, a string of keyword associations was created, one of which identifies the photographic source. This list includes identification of figurative elements (e.g., goose, rocking horse, lemon) and their synectic analogies.[4,5] For example, {Rocking horse, children} is a more immediate and general association accessible to an audience; whereas {Rocking horse, Westbury, antique, sister, nightmare, protection} is an more idiosyncratic associative list that has personal significance as choices are made in using that rocking horse in combination with other images. This example reveals that a rocking horse or other toy is not an emotionally neutral object. Hence the content taxonomy is at once a literal listing of objects and the dates and places where they were photographed, and a memory trigger, together with emotional, social, political, philosophical and spiritual associations.

If abutting edges of two textures appear to seamlessly flow from one tile to another, the progression of pattern can meander through preconceived paths (see Figure 1). If a tile pair has not been composited for seamlessness, then clear boundary lines will be visible (see Figure 2). At this juncture, TOLAS can proceed in three possible ways: it can perform procedures to eliminate obvious seams (the easiest, albeit least aesthetically pleasing method would be to blur the edges), it can superimpose a border tile on top of visible seams (see Figure 3), or it can accept the seams and use predictable patterns (e.g., a checkerboard) to make it seem as if the abrupt change from one tile to the next is intentional simply because that change is repeated in a regular predictable pattern.

Figure 9. Seamless texture progression (L to R).

Figure 10. Visible texture seam.

Figure 11. Border pattern covers seams.

Table 1 below outlines selected texture metadata that is stored, along with the texture source material, in a database for use in the generative process.

Table 1. Example Texture Metadata

metadata label	data type	description
color1, color2, color3	RGB	principal, subsidiary colors
symmetry	Boolean tuple	axial and rotational symmetry flags
is_seamless_with	Boolean tuple	edges blend with other source tiles
is_figurative	Boolean	image content flag
mean_luminance	integer	average value over 2D image
keywords	string	semantic tags

While these selected variables provide a vast simplification in the dimensionality of our problem space, they do little to preserve the semantic content of texture images. To this end, we also store alpha channel image masks that can be applied in collage fashion to hide seams. Aleatoric selection here introduces a new layer of content to the already polyvalent semantic content. Similarly, the aforementioned border patterns which serve the same seam–hiding function as the image maps present recombinations of image content as well as architectural ornament. The latter make reference to the anthropomorphic valences of meaning George Hershey applies to his analysis of classical architecture where dentils, triglyphs and corbels trope teeth, thighs and ears.[6]

3. Animation

The initial scene in TOLAS is that of a single random seed texture applied to one mesh element, chosen from the library of predefined textures. After this initial display, a series of imaging transformations is performed on the seed texture; these transformed copies of the image are mapped onto adjacent mesh elements to form a 2 x 2 grouping, and then a 4 x 4 grouping. New textures then appear, and the generative routine is started; as tiles are created, they cycle into the background, and displace the already-generated portions. At any time, the user can navigate elsewhere within a large virtual space; new generative pipelines are started for display into the new viewing frustum. However, the generative process is still enabled for the other portions of the scene viewed earlier, and the user can navigate back to that region at any time.

Much of the metadata associated with textures is not used by TOLAS in texture synthesis. During the main generative loop, a suitable mapping from variables currently unused is enabled, and this mapping output is used to drive a mesh distortion routine localized in the current viewing frustum. This routine modifies the position of control points on the mesh. This output is also fed to the audio engine (described below).

4. Generative Audio Accompaniment

To determine an appropriate mapping of the texture generation process into the audio domain, we note that TOLAS’ texture synthesis takes as inputs two (potentially dissimilar) texture sources, yielding a novel result. This was reminiscent of an compositional technique called hocketing, where two voices alternate notes in rapid succession.[7] Hocketing is commonly used in the Ugandan amadinda (pentatonic-tuned xylophone) tradition[8] to subdivide an overall melodic line between multiple players. An actual wooden amadinda had been constructed for an earlier live performance, and we digitized samples of the instrument’s keys being struck. As noted earlier, only a few variables are typically used as arguments to the texture generation function. The audio track provides an opportunity to map several other variables into a virtual amadinda performance ensemble coded as a Csound[9] orchestra, with real-time score events dispatched to the Csound renderer via inter-process communication with the animation application. The Csound orchestra also includes sampled speech fragments for use as an audio analogue to the various edge-blending techniques employed in the visual channel; these fragments also are loosely coupled to selected keywords in the texture metadata. To further bind the visual and audio events in TOLAS’ animation sequence, events driving the mesh displacement mapping are redundantly mapped into melodic and dynamic qualities in the Csound amadinda ensemble.

References

1. Anna M. Chupa and Michael A. Chupa, “Generative Texture Maps for Computer Animation,” Proc. Generative Art ‘99.

2. Plato’s Phaedrus. translated with Introduction and commentary by R. Hackworth. New York: The Bobbs Merrill Company, Inc. a subsidiary of Howard W. Sams & Co., Inc. 1952.

3. OpenGL Architecture Review Board, Mason Woo, Jackie Neider, Tom Davis, and Dave Shreiner, OpenGL Programming Guide, 3^rd edition. Reading, MA: Addison-Wesley, 1999 p. 363.

4. Nicholas Roukes. Art Synectics: Stimulating Creativity in Art. Worcester MA: Davis Publications, Inc. 1984.

5. Nicholas Roukes. Design Synectics: Stimulating Creativity in Design. Worcester MA: Davis Publications, Inc. 1988.

6. George Hershey, The Lost Meaning of Classical Architecture. Cambridge, MA: MIT Press, 1989. pp. 31, 38, 40.

7. Ned Sublette, Cuba and Its Music: From the First Drums to the Mambo. Chicago, IL: Chicago Review Press, 2004. p. 49.

8. Gerhard Kubik, A Theory of African Music, vol. 1. Wilhelmshaven, Germany: Florian Noetzel Verlag, 1994. pp. 47–85.

9. Richard Boulanger, ed. The Csound Book. Cambridge, MA: MIT Press, 2000.