ElectroEncephaloGraphics: A Novel Modality For Graphics Research - Maryam Mustafa - ebook

ElectroEncephaloGraphics: A Novel Modality For Graphics Research ebook

Maryam Mustafa



My work uses EEG data to determine the perceptual quality of videos and images which is of paramount importance for most graphics algorithms. This is especially important given the gap between perceived quality of an image and physical accuracy. This thesis begins by introducing the fundamentals of EEG measurements and its neurophysiological basis. Following this introduction, I present a novel method for determining perceived image and video quality from a single trial of EEG data in response to typical rendering artifacts. I also explore the use of EEG for direct neural feedback and present a neural-feedback loop for the optimization of rendering parameters for images and videos. I conclude with an outlook on what the future of EEG in graphics may hold.

Ebooka przeczytasz w aplikacjach Legimi na:

czytnikach certyfikowanych
przez Legimi

Liczba stron: 115

Odsłuch ebooka (TTS) dostepny w abonamencie „ebooki+audiobooki bez limitu” w aplikacjach Legimi na:


Contributions of the Author

Clarification of my individual contributions to the publications that describe parts of my thesis; The papers are ordered according to the structure of this thesis.

1. Maryam Mustafa, Lea Lindemann, Marcus Magnor. EEG analysis of implicit human visual perception. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 513-516, 2012

I developed the ideas and experimentation implemented in this paper. L. Lindemann and I discussed ideas for artifact selection, data analysis and experimental setup. The paper was written and presented by me. M. Magnor guided the project with many suggestions and gave advice concerning ideas and content of the paper. The contributions of this paper are part of Chapter 3.

2. Maryam Mustafa, Stefan Guthe, Marcus Magnor. Single-trial EEG classification of artifacts in videos. ACM Transactions on Applied Perception (TAP), 9(3):12, 2012.

This paper was selected as one of the best 3 papers of the ACM Symposium on Applied Perception (SAP 2012) conference and therefore was accepted by and published in the TAP journal. It was presented at SAP 2012. The ideas presented in this paper are part of Chapter 4 and all experimental work, data analysis and paper writing was done by me. S. Guthe was responsible for the wavelet based classification.

3. Maryam Mustafa, Marcus Magnor. ElectroEncephaloGraphics: Making Waves in Computer Graphics Research. Computer Graphics and Applications, IEEE , 34(6):46 - 56, 2014.

This paper was selected for a special issue of the Computer Graphics and Applications Journal on The Next Big Thing. I was responsible for the ideas, conducting the experimental work, data analysis, creating the neuro-feedback loop and writing the paper. Contributions from this work are presented in Chapter 5. Parts of this work are based on earlier work (Chapter 4) with S. Guthe who supported me with advice and suggestions for the implementation and evaluation of the algorithms. He was also responsible for the wavelet based classification. M. Magnor oversaw the project.


Neuroimaging and brain mapping techniques can provide meaningful insights and guidance for graphics related problems. This is particularly true given that most of the output from graphics algorithms and applications is for human consumption.

In this thesis I present the application of ElectroEncephaloGraphy (EEG) as a novel modality for investigating perceptual graphics problems. Until recently, EEG has predominantly been used for clinical diagnosis, in psychology, and by the brain-computer interface (BCI) community. Here, I extend its scope to assist in understanding the perception of visual output from graphics applications and to create new methods based on direct neural feedback. My work uses EEG data to determine the perceptual quality of videos and images which is of paramount importance for most graphics algorithms. This is especially important given the gap between perceived quality of an image and physical accuracy.

This thesis begins by introducing the fundamentals of EEG measurements and its neurophysiological basis. Following this introduction, I present a novel method for determining perceived image and video quality from a single trial of EEG data in response to typical rendering artifacts. I also explore the use of EEG for direct neural feedback and present a neural-feedback loop for the optimization of rendering parameters for images and videos. I conclude with an outlook on what the future of EEG in graphics may hold.


In dieser Arbeit präsentiere ich die Anwendung von Elektroenzephalografie (EEG) als eine neuartige Modalität zur Untersuchung von Wahrnehmungsfragen in der Computergraphik. Bisher wurde EEG vorwiegend für die klinische Diagnostik, in der Psychologie und in der BCI-Community verwendet. Ich erweitere den bisherigen Anwendungsbereich um die Untersuchung von perzeptueller Qualität bildgebender Verfahren auf Basis von neuronalem Feedback.

Da die Ergebnisse der meisten graphischen bildgebenden Verfahren für die Betrachtung durch Menschen bestimmt sind, ist bei der Bildsynthese neben der physikalischen Genauigkeit ebenso die durch den Betrachter tatsächlich wahrgenommene Qualität von großer Bedeutung. Um die tatsächliche wahrgenommene Qualität von Videos und Bildern zu ermitteln, setze ich in meiner Arbeit mit EEG gemessene Daten ein.

Diese Arbeit beginnt mit einer Einführung der Grundlagen der EEG-Messungen und ihrer neurophysiologischen Basis. Nach dieser Einführung stelle ich eine neue Methode zur Bestimmung wahrgenommener Bild- und Videoqualität vor. In diese Methode ermittele ich ein Maß für die wahrgenommene Bildqualität, in dem die EEG-Daten von Probanden als Reaktion auf typische Rendering-Artefakte aufgezeichnet werden. Weiterhin erforsche ich die Nutzung des EEG für direktes neuronales Feedback und präsentiere eine Neuronale-Feedback Schleife zur Optimierung von Rendering-Parametern für Bilder und Videos. Ich schließe diese Arbeit mit einem Ausblick auf die zukünftigen Möglichkeiten, die das EEG der Computergraphik bereitstellen könnte.

To Halah,

my parents and in memory of Haroon


While the contribution of this dissertation is my own, I wouldn’t have been able to finish it without the support of many.

I would like to express my sincere thanks to my advisor, Marcus Magnor for his constant support, ideas and all the valuable and constructive comments during each step of this work. Without his guidance, understanding, feedback and often times compassion this work would truly not have been possible. Thank you Marcus.

I would also like to thank Douglas Cunningham for always making the time for discussions about my work and for being the voice in my head cautioning me and expecting a stricter standard of scientific research. For that I am a better researcher and truly grateful.

Stefan Guthe also deserves my gratitude not only for his help in my work but for being the sounding board for my ideas and for always having a solution to my mathematics related problems.

I would like to express my very great appreciation to present and former colleagues of the Computer Graphics Lab at TU Braunschweig for creating a wonderful work environment. In particular, I would like to thank Martin Eisemann, Felix Klose and Kai Ruhl for the ‘interesting’ discussions in the kitchen and the endless hours of entertainment. Thanks Anja Franzmeier for supporting me not only in the administrative work but making me feel welcomed in this department and country. Thank you Carsten for maintaining all computers and solving many technical problems.

Further thanks goes also to my student assistant Julia Duczmal for her endless patience in the midst of all the EEG experiments.

Given that there exists a world beyond doctoral research work, I would like to thank my adopted family, the Prekazi’s and Tahera, for being there through the good and the bad and for making me feel less alone. Thank you Ariana for being friend, sister, baby-sitter, aunt, therapist and so much more.

Many other friends have also provided support through the years. I can’t name all of them individually, but I appreciate every single one of them. In particular, I’m grateful to Asli, Ghadah and Jeanne for providing me with a certain sense of normalcy and sanity in the last months of this work.

Finally, I would like to thank my family for the support they provided me through my life. Particularly, my parents who have been a constant support in everything I have achieved. I would especially like to thank Fatima for dropping everything to always be there so I could do my work. Thank you.

A very special thanks goes to Halah for being my partner in the many ups and downs of this journey. Only she can truly comprehend the toll of this journey and only she can truly appreciate its culmination in this work.



1.1. Thesis structure and contribution


2.1. EEG Basics

2.2. EEG and Graphics

Image and Video quality assessment using EEG

3.1. Introduction

3.2. Related work

3.3. Experiment Procedure

3.4. Results

3.5. Discussion

3.6. Summary

Single Trial Analysis of EEG data

4.1. Introduction

4.2. Related Work

4.2.1. Perception Based Rendering Algorithms

4.2.2. Wavelet-Based Analysis

4.3. Artifact Classification

4.3.1. Wavelet Transformation

4.3.2. Shift Invariant Transform

4.3.3. Support Vector Machine Classification

4.4. Experiment

4.5. Results

4.5.1. Response from Frontal Electrodes

4.5.2. Wavelet Based Classification

4.6. Discussion

4.7. Summary

The Human in the Loop : EEG-driven Rendering Parameter Optimization

5.1. Introduction

5.2. Related Work

5.2.1. Perceptual graphics

5.2.2. Visual quality metrics

5.2.3. Non-photorealistic rendering

5.2.4. EEG in BCI

5.2.5. EEG and emotion

5.2.6. EEG in visual perception

5.2.7. Rendering parameter optimization

5.3. EEG Optimization Loop

5.3.1. Classifier

5.4. Evaluation

5.4.1. Application Scenario: Photo Personalization

5.4.2. Application: Guided Image Filter Parameter Optimization

5.5. Discussion

5.6. Summary

Conclusion and Future Directions

A. Two-Tailed T-Test

B. Test Results for Neural feedback loop


1. Introduction

“All our knowledge has its origins in our perceptions."

—Leonardo da Vinci

The ultimate purpose of computer graphics is to produce images and videos for human observers. Thus the success of any graphics application depends on how well it conveys relevant information to a human viewer. However, the inherent complexity of the physical world and limitations of computing hardware make it an impossible task to replicate the view of the real world. In fact, over 150 years of research on perception shows that no organism has a perceptual system that tries to create an exact representation of the real world. All perceptual systems, including the human perception system, make many assumptions in order to make sense of the real world. This can often lead to a gap between how we perceive the world and how it really is. Take for example the Café Wall illusion (Fig.1.1) where our perceptual system convinces us that the walls are not parallel when in fact they are. Given the vastly complex task of creating physically accurate imagery the best that graphics algorithms can aim for is to create perceptually accurate images. To achieve this goal it is essential to understand and analyse how the Human Visual System (HVS) perceives the world around it. This understanding will allow computer graphics practitioners to take advantage of the flexibility and robustness associated with human perception to decrease the gap between hardware performance and desired performance. In rendering, for example, resources can be allocated to areas of a scene that matter most to human observers, saving computation time. Similarly, visualization techniques have benefited from perceptual measures such as processing speed, which determines how quickly humans perceptually process features such as colour and texture. The integration of methods and techniques from perception research into graphics is particularly important if the goal is to create realistic imagery for movies, games, and immersive environments.

Figure 1.1.: The Café Wall illusion first discovered by Gregory and Heard deceives the human perceptual system that the lines are not parallel [GH79].

The study of perception is, however, a complicated task. The main difficulty is that it can not be directly measured or observed. Given this, the only way to study human visual perception is through indirect measures. Any perceptual process does, to a certain extent, influence human behaviour and it is through the study of this behaviour (covert or overt) that we can create models of perceptual systems.

One of the main methodologies used by perception researchers to study and understand hidden perceptual processes are psychophysical experiments [MBB12]. Psychophysics is the empirical study of the relationship between physical stimulus and the resulting perceptual or sensory responses [Ges13]. It was first introduced in Fechner’s Elements of Psychophysics which presents methods and theories for studying sensation [Fec48]. Psychophysical experiments are conducted in a highly controlled environment with an absolute control over as many factors as possible. Because of the desire for absolute control over the experiment, most psychophysical experiments use simple abstract stimuli. This makes it difficult to not only conduct these experiments but also to model the results.

Another technique typically used for perceptual research is the eye-tracker. Researchers have employed eye-trackers to investigate what kind of visual information people focus on while viewing images, videos, or visualizations. This can provide valuable information on which parts of the stimulus are important and on the order in which an image is scanned. These tools have become part of mainstream graphics research and have provided unique insights.

Brain imaging techniques are another class of measurement methodologies used for studying perception. Neuroimaging technologies allow researchers to visualize the processing of information in the brain directly. There are a number of safe imaging techniques in use throughout the world for research and medical purposes. Functional magnetic resonance (fMRI) works by detecting the changes in blood oxygenation that occur in response to neural activity. Technological advances in the acquisition of fMRI data and its processing have made it possible to analyse neural activity as quickly as the images are acquired allowing this data to be fed back to the subjects. It is then possible for subjects to voluntarily learn to modulate their own brain activity using real-time fMRI data (rtfMRI) [BHVH12]. Other signal acquisition techniques include magnetoencephalography (MEG) where brain activity is mapped by recording magnetic fields produced by electrical currents occurring naturally in the brain using very sensitive magnetometers. However, one of the most popular brain mapping techniques is Electroencephalography (EEG) which measures the electrical activity of large numbers of neurons close to the brain surface. EEG has been particularly important for sleep research and for detecting abnormalities in brain waves [RK87, FKH67, GAL90]. EEG has been used extensively in clinical psychiatry, brain-computer interface (BCI) research and perception due to its high temporal resolution, inexpensiveness and ease of use compared to the other imaging techniques.