CoG2020 Tutorial

Sketching and Bio-sensing for "Wellness Entertainment"

August 2020 - Kinki University, Japan

Yoichi Nagashima (SUAC/ASL)

Description of tutorial

This is a lecture intended for the designer/artist/therapist of the interactive system for "wellness entertainment". In recent years, the fields of serious games and medical/welfare/rehabilitation applications are gaining attention in the field of entertainment computing. The presenter has been promoting R & D activities in this area for 30 years, and recently targeting "Wellness" or "Well-being" entertainment using bio-sensing technology. This lecture contains four parts of practical techniques/ideas to realize effective interactive system for "wellness entertainment" - environment & interface, concept of biofeedback, special techniques using commercial bio-sensors and cutting-edge topics.

The first part is short lecture of "Max8"(cycling'74) - famous and powerful platform/environments to develop realtime multimedia system with sensors. This "Sketching"(physical computing) lecture with many effective examples will help future development. The popular interface "Gainer" had retired few years ago, so this lecture also contains three types of "post-Gainer" new sensing/interface techniques using Arduino.

The second part is a lecture about "biofeedback and human wellness" related to "Wellness Entertainment" System Design. The idea of biofeedback is well-known in the rehabilitation and welfare field, and recent bio-sensing technology and multimedia technology enable to realize effective biofeedback system easily. Generally, human mental power for homeostasis works in emotion/feeling channel - which leads us "Wellness" or "Well-being" status. Especially, reaction information from unconscious muscles is gaining attention as "interoceptive sense" leading to emotion/feeling - this is the reason the organizer pays particular attention to the EMG sensing.

The third part is the special techniques using bio-sensing systems which are commercially available. (1) The "Myo" armband - three original techniques will be opened - how to use 2 or 3 Myo at the same time, how to avoid Myo's sleeping, and how to get directly data such as 8ch-EMG and 9-D data via Bluetooth. (2) The "MUSE" headband - how to get directly data such as 4ch-brain waves and 3D-direction via Bluetooth. (3) The combination application using "double Myo" and "MUSE" will be shown with the detail of Max8 programming - as a realtime interactive multimedia performance work in media art.

The fourth part is two cutting-edge bio-sensing topics. (1) The presenter and collaborator developed a novel 4ch EMG sensor system called "VPP-SUAC". The whole development data is open-source in the Web. While Myo is an "exclusive for arm" EMG sensor, our sensor focuses on sensing legs and lower body EMG which are important in welfare and rehabilitation area. (2) The other is a new special tactile/rubbing sensor called "PAW sensor", which is a small PCB with a cylinder of urethane foam on it. The output information of this sensor is four channel voltages which is time-shared conversion, which means the nuances of rubbing/touching the urethane foam with fingers. In this lecture, the attendees will experience the special eight sensors and its experimental application which generates 3-D realtime sounds/graphics for healing entertainment or prevention of dementia.

Background/relevance of tutorial topic

Outline of tutorial workshop

Part 1 Short/dense Introducing Lecture of "Max8" and "post-Gainer" Techniques
	(1) Max/MSP/jitter Introduction Demos
	(2) "post-Gainer" Techniques - Firmata+maxuino, Arduino2Max, USB-MIDI

Part 2 Lecture/discussion about "Biofeedback and Human Wellness"
	(1) Biofeedback and rehabilitation/habilitation
	(2) interoceptive sense

Part 3 Special Techniques using Bio-sensing Systems which are commercially available
	(1) Myo and DoubleMyo
	(2) Muse and MuseOSC

Part 4 two cutting-edge bio-sensing topics
	(1) PAW sensor (tactile sensor) - single test, "MRTI2015", "PAW-eight", "PAW-double"
	(2) Introduction for original EMG/EEG sensor "VPP-SUAC"

Duration

3 hours ?

Language

English (if requested, Japanese a little)

Target participants, including level of experience

Expected tutorial workshop outcomes

Sample materials

Examples of interactive works are online - for example: http://nagasm.org/1106/PhysiCom/

CV information of the tutorial presenter

Yoichi Nagashima, composer/researcher/PE, was born in 1958 in Japan. He learned and played some instruments: violin, recorder, guitar, keyboards, electric bass, drums, and vocal/choral music for 45 years. He was the conductor of Kyoto University Choir and composed over 100 choral music, and studied nuclear physics there. As the engineer of Kawai Musical Instruments, he developed some sound generator LSIs, and designed some electronic musical instruments, and produced musical softwares. From 1991, He has been the director of "Art & Science Laboratory" in Japan Hamamatsu, produces many interactive tools of realtime music performance with sensor/MIDI, cooperates some researchers and composers, and composes experimentally pieces. He is also the key-member of Japanese computer music community. From 2000, he has been also the associate professor of SUAC(Shizuoka University of Art and Culture), Faculty of Design, Department of Art and Science, and teaches multimedia, computer music and media art. As a composer of computer music, he collaborates many musicians in his composition: Piano, Organ, Percussion, Vocal, Flute, Sho, Koto, Shakuhachi, Dance, Percussion, etc. He organized and was the General Chair of NIME04. He became a professor on April 2007. He organized a tutorial “Parallel Processing Platform for Interactive Systems Design” at ICEC2009 (Paris).

http://nagasm.org/ASL/profile/

Lecture/Workshop of Yoichi Nagashima (in English)

  1. Lecture "Technology for Computer Music", Seoul, Korea, 1997
  2. Workshop "Sensors for Interactive Music Performance", ICMC2000, Berlin, Germany, 2000
  3. Workshop "Sensors for Interactive Music Performance", Studio CCMIX, Paris, France, 2001
  4. Lecture "Composition of [Visional Legend]", International Workshop on "Human Supervision and Control in Engineering and Music", Kassel, Germany, 2001
  5. Lecture "Interactive Media Art with Biological Interfaces", STEIM Open Lecture Concert Series, Amsterdam, Holland, 2004
  6. Lecture "Interfaces for Interactive Media Art", IRCAM Open Lecture, Paris, France, 2004
  7. Lecture "Interactive Art with Bio-Interfaces", La Kitchen Open Lecture, Paris, France, 2004
  8. Lecture "Interactive Multi-Media Performance with New Interfaces", International Workshop on Interactive Media Art, Kai-Nan Univ., Taiwan, 2007
  9. Lecture "Interactive Multi-Media Art with New Technology", Tsukuba Univ., Japan, 2007
  10. Lecture "SUAC Installation - Case Studies as Physical Computing -", Sketching in Hardware 2008, Rhode Island School of Design, USA, 2008
  11. Lecture "Some interactive works ('08-'09) - case study of my sketching”, Sketching in Hardware 2009, University College of London, UK, 2009
  12. Tutorial "Parallel Processing Platform for Interactive Systems Design”, International Conference on Entertainment Computing, Conservatoire National des Arts et Metiers, Paris, 2009
  13. Lecture "Technology for Computer Music / Interactive Multi-Media Performance with New Interfaces", International Festival/Competition SYNC. 2010 Tutorial(1), The Ural State Conservatory, Yekaterinburg, Russia, 2010
  14. Lecture "SUAC Installation - Case Studies as "Physical Computing", International Festival/Competition SYNC.2010 Tutorial(2), The Ural State Conservatory, Yekaterinburg, Russia, 2010
  15. Lecture "Interactive Art with Bio-Interfaces", International Festival/Competition SYNC.2010 Tutorial(3), The Ural State Conservatory, Yekaterinburg, Russia, 2010
  16. Lecture "Sketching as entertainment of design”, Sketching in Hardware 2012, University of Oregon, USA, 2012
  17. Lecture "New Platform for Design Entertainment”, Sketching in Hardware 2013, PARC, California, USA, 2013
  18. Lecture "Against the boom of image recognition sensors”, Sketching in Hardware 2015, Biosphare 2, Arizona, USA, 2015
  19. Lecture "Biofeedback Instruments and Improvisation”, 4th tempora International Meeting, Bordeaux, France, 2016
  20. Lecture "Interactive Media Arts - New Ideas and New Technologies", General Lecture for Ekaterinburg's Contemporary Art Academy, Gorky library, Yekaterinburg, Russia, 2016
  21. Lecture "Interactive Media Arts - New Ideas and New Technologies", General Lecture - open public, Ural branch of the National Center for Contemporary Arts, Yekaterinburg, Russia, 2016
  22. Workshop "Introduction of Programming - Creating Art Objects", Practical Workshop for Art Academy's Designer Students, Center of Culture “Ordzhonikidzevsky”, Yekaterinburg, Russia, 2016
  23. Workshop "Interactive System Design - Creating Media Arts", Practical Workshop for Art Academy's Programmers Students, Center of Culture “Ordzhonikidzevsky”, Yekaterinburg, Russia, 2016
  24. Lecture "Human-Computer Interaction and Media Arts (part1/2)”, Lecture Workshop at MARS Gallery, MARS Gallery, Moscow, Russia, 2016
  25. Lecture "Bio-Sensing and Bio-Feedback Interfaces”, Conference “Musical Interfaces and Robotics”, Lecture Workshop at Tokyo University of Arts, Japan, 2017
  26. Workshop/Tutorial "Bio-Sensing Platforms for [Wellness Entertainment] System Design", International Conference on Entertainment Computing, Poznan University of Technology, Poland, 2018

YouTube Demo movies

Presentation and papers

VPP-SUAC (sorry, currently in Japanese only - please use "Google Translate")

PAW-double (sorry, currently in Japanese only - please use "Google Translate")