Journal Search Engine
Search Advanced Search Adode Reader(link)
Download PDF Export Citaion korean bibliography PMC previewer
ISSN : 1598-7248 (Print)
ISSN : 2234-6473 (Online)
Industrial Engineering & Management Systems Vol.18 No.4 pp.701-709
DOI : https://doi.org/10.7232/iems.2019.18.4.701

Comparison of Workload Perception for Original and Modified Cognitive Tasks

Shuxun Chi, Manida Swangnetr Neubert*, Orawan Buranruk, Weerawat Limroongreungrat, Wichai Eungpinichpong, Rungthip Puntumetakul
Research Center in Back, Neck, Other Joint Pain and Human Performance, School of Physical Therapy, Faculty of Associated Medical Sciences, Khon Kaen University, Khon Kaen, Thailand
Research Center in Back, Neck, Other Joint Pain and Human Performance, Program of Production Technology, Faculty of Technology, Khon Kaen University, Khon Kaen, Thailand
Research Center in Back, Neck, Other Joint Pain and Human Performance, School of Physical Therapy, Faculty of Associated Medical Sciences, Khon Kaen University, Khon Kaen, Thailand
College of Sports Science, Mahidol University, Salaya, Nakhon Pathom, Thailand
Research Center in Back, Neck, Other Joint Pain and Human Performance, School of Physical Therapy, Faculty of Associated Medical Sciences, Khon Kaen University, Khon Kaen, Thailand
Research Center in Back, Neck, Other Joint Pain and Human Performance, School of Physical Therapy, Faculty of Associated Medical Sciences, Khon Kaen University, Khon Kaen, Thailand
Corresponding Author, E-mail: manida@kku.ac.th
May 7, 2019 September 23, 2019 October 18, 2019

ABSTRACT


Memory search performance is commonly required in many daily life and professional careers. Sternberg Memory Search Task (MST) is originally designed to test working memory (WM) retrieval function. However, in different scenarios, memorizing a string of letters (i.e., stimulus items in original (MST) may not be able to represent the real task characteristics, in this case for a specific soccer player population. Therefore, a modified MST was designed based on literature review and interview with soccer players. With the same numbers of WM requirements with the original MST, the modified MST was designed to adopt numeric (player numbers) and color codes (team uniform) as stimuli. The objective of this study was to preliminarily compare subjective workload perception during the original and modified MST performance. Seven healthy male university students (average age of 30.7±8.0 years) were recruited to participate in this study. All participants were asked to perform both the original and modified MST including low and high workload intensity conditions. All subjective workload scores were collected by using NASA-Task Load Index (NASA-TLX). In general, the results showed increase in workload intensity to induce higher workload perception for both test conditions. Mental Demands and Effort contributed as major demanding components of this cognitive task type. Although overall workload of both tests was comparable, subscale workload scores of the modified MST found to be lower than the original MST. The findings suggested that the modified MST can be used to test similar demanding workload component target, as compared with the original MST. However, since different types of symbolic information posed different levels of mental workload perception, the task stimuli should be carefully designed to represent actual characteristics of the task of interest.



초록


    1. INTRODUCTION

    Memory is the human ability to receiving, modifying, storing, and retrieving information (Klatzky, 1975). In general, there are two types of memory storage: working memory (WM) and long-term memory (LTM) (Klatzky, 1975). LTM can store a relatively large quantity of information for many years. WM, in contrast, can only hold about 7±2 chunks of information for a limited time without any rehearsal or reviewing (Klatzky, 1975). Individual differences in the capacity of WM were found to be associated with variation of control of attention (Kane et al., 2007), non-verbal reasoning ability (Kyllonen and Christal, 1990) and academic performance (Gathercole et al., 2003). WM perceives or retrieves information from LTM and temporality stores it while the necessary information is actively manipulated for complex cognitive tasks (Schacter and Tulving, 1994). An increase or decrease of cognitive performance is most likely, due or at least closely linked to retrieval failure from either WM or LTM. WM can be, generally, modelled to consist of 3 components including a central executive, visuospatial sketch pad, and phonological loop (Baddeley, 1997). The central executive is assumed to be an attentionalcontrolling system for coordinating information between WM and LTM. The visuospatial sketch pad holds and manipulates spatial form of information (e.g., visual images); while the phonological loop stores and rehearses verbal information.

    Different functions of WM are required in many daily life and professional careers. Recreation and occupations require mental demands in encoding, storage and recall information in order to complete both simple and complex tasks. Many cognitive task batteries have been proposed and applied to test a performance of these WM functions. Sternberg Memory Search Task (MST) is based on the experimental paradigm proposed by Sternberg in 1969 (Sternberg, 1969). The test is a standardized task designed to place variable demands on human information processing resources dedicated to WM retrieval functions (Shingledecker, 1984). In this test, participants were initially required to memorize a set of letters. Subsequently, participants were presented with a series of single letters and asked to identify whether the presented letter belonged to the previous set of letters. Based on this approach, human factors and human performance specialists can gather information regarding capacities and limitations of the human operator (Shingledecker, 1984;Pesonen et al., 2006). However, in different scenarios, memorizing a string of letters, used as stimulus items in original MST, may not be able to represent the real task characteristics. Frequently, individuals may encounter situations that require memorizing pictorial symbols and colors rather than solely a set of letters. For example, soccer players need to remember the opponent team’s clothing color and the marked number of players (Williams et al., 1993). As different types of spatial and verbal information codes required different mental resource-demanding (Wickens et al., 1998;Frackowiak, 2004), various MST stimuli might pose different levels of mental workload perception. Therefore, it might be essential to design MST stimuli according to types of information presented in the real situations.

    Mental workload has been assessed by a variety of different techniques, including physiological, performancebased, and subjective measures (Xiao et al., 2005;Cao et al., 2009;Christ et al., 1993;Donaldson et al., 2000;O’donnell and Eggemeier, 1986). Event-related potentials (ERP), heart rate, heart rate variability (HRV), and blood pressure have been widely used as physiological measures of workload. In general, physiological measures allow for continuous workload monitoring and are sensitive to changes in workload demand. However, some measures might be physically obstructive to task performance (Christ et al., 1993;O’donnell, 1986). Performance-based measuring can be conducted by recording accuracy and time spent on specific tasks. Performance reflecting mental workload can be measured using primary or secondary task measurement methods. Subjective workload assessment evaluates participants’ perceived workload using self-rating questionnaires. This technique assumes that an increased demand is associated with a perception of effort, which can be appropriately assessed by individuals (Christ et al., 1993). Although the measurement of workload perception might be subjective and therefore might not correspond with actual performance (Andre and Wickens, 1995), it represents the easiest technique to conduct and the most intuitive measure of mental workload (Wickens et al., 1998). The most widely used questionnaires are the Subjective Workload Assessment Technique (SWAT) and the National Aeronautics and Space Administration-Task Load Index (NASA-TLX) (Christ et al., 1993). SWAT is a subjective rating technique of workload composed of three dimensions, specifically: time, mental effort, and psychological stress load (Reid and Nygren, 1988). However, NASA-TLX has been indicated to be substantially easier to accomplish, and more practical to apply in operational environments, as compared with SWAT (Hart and Staveland, 1988). NASA-TLX has been reported to be of high reliability and validity for subjective workload measurement (Cao et al., 2009;Hart and Staveland, 1988). It also has been proved to be able to distinguish possible relationships between workload and stress states (Matthews and Campbell, 2010), and correlate with other physiological signals, such as blinks of surgeons (Zheng et al., 2012). NASA-TLX has been previously applied in many WM task studies (e.g., mental arithmetic, tracking task, continuous recall tasks) (Ryu and Myung, 2005;Anthony and Biers, 1997;Hancock, 1996). In addition, this tool has been successfully applied in previous MST studies (e.g., Haga et al., 2002;Rubio et al., 2004;Matthews and Westerman, 1994;Proctor et al., 1998). Earlier investigations showed that the ratings of NASA-TLX were higher as the task difficulty increased when participants performing MST.

    The objective of the present study was to perform a preliminarily comparison of workload perception during original and modified MST performance. The modified MST was designed for cognitive demands representing actual characteristics of the specific scenario, in this case for a soccer match simulation, based on a structure interview and a review of the relevant human factors literature. The soccer match scenario was selected since the types of information required to be memorized by the players are largely dissimilar to the string of letters presented in the original MST. It was expected that the stimuli in the modified MST would pose different levels of mental workload perception, as compared with the original MST. However, both MST tests were hypothesized to be similar in terms of their capability to test the same components of workload (i.e., certain NASA-TLX workload subscales). The sensitivity to changes in workload was also expected to indicate a similar direction for both MST conditions. Specifically, the workload perception was anticipated to increase when workload demand changed from low to high intensity.

    2. METHODS

    2.1 Participants

    Seven healthy male university students were recruited from a university in Thailand. Participants’ age ranged from 22 to 41 years. All participants had normal or corrected 20/20 vision. All of them were right-handed. Table 1 displays details of demographic characteristics of participants.

    Before participating the experiments, participants were asked to abstain from: (1) alcohol use for 48 hours and; (2) caffeine-containing products for 12 hours. All participants had no traumatic brain injury, chronic or acute sleep deprivation, a fever or dizziness within 24 hours before the trial; or any severe psychological disorders. Four participants were dropped from the study due to achromatopsia (n = 1), smoking more than 3 times/week (n = 2) and alcoholism (n = 1). Participants were informed regarding objectives, procedures, benefits and risks of the present study before giving written informed consent. All study procedures were approved by the Khon Kaen University Ethics Committee in human research based on the Declaration of Helsinki and the ICH Good Clinical Practice Guidelines (Reference No. HE602170).

    2.2 Memory Search Task (MST) Design

    Task demand loadings investigated in this study were divided according to numbers of WM chunks, including: 1). low- demanding condition ≤5 WM chunks and; 2). high- demanding condition >5 WM chunks (Haga et al., 2002). When more than 5 chunks of information are required to be memorized in WM, prior research also found that cognitive overload and potential errors are more likely to occur (Kieras et al., 1998;Lerch et al., 1989).

    2.2.1 Original MST

    Stimulus items in the original MST were visually presented in English capital letters. Seventeen of the alphabets, ABCEFGHIJLOQRSXYZ, were used in the test to prevent the acoustic confusability (omitted 9 alphabets including DKMNPTUVW). Four and seven letters memory sets were used as low and high intensity levels of workload, respectively. If these memorized letters appeared on the screen, subjects were instructed to press the left button. If other remaining letters (i.e., negative set) appeared, they should press the right button. Table 2 shows examples of original MST rules. In this example, for low workload intensity, participants were initially presented with a set of letters: B, E, O, and G. The memory set was not sorted in alphabetical order. Participants were then presented with a single letter, one at a time. If B, E, O, or G appeared on the screen, subjects were asked to press the left button. If any of the other letters appeared, participants were instructed to press the right button. Task difficulties, using 4 and 7 letters, were designed to require 4 and 7 numbers of chunks in WM, respectively.

    2.2.2 Modified MST

    The modified MST was designed based on literature review and interview with soccer coaches and players. A structured interview was conducted with 24 soccer professionals and 9 coaches from professional football clubs in Thailand regarding cognitive demands perceived in training and during a match. Based on the interview, soccer athletes were trained to memorize specific players both from their team and the opponent team in order to perform certain movements. For example, when a player encountered a certain opponent, he was instructed to pass a ball to a specific teammate. However, for some other opponents, the player might try to dribble past them without passing the ball. According to a literature review, rulebased decision making (Rasmussen, 1983) model of naturalistic decision making; (Salvendy, 2012) was identified as a type of cognitive demand. Subsequent review on possible psychomotor tests indicated Sternberg MST (Sternberg, 1969) can be applied to represent this type of cognitive demand. The modified MST was designed to simulate the real characteristics of the task during soccer match by adopting numeric (i.e., player numbers) and color (i.e., team uniform) codes as the stimulus items. The task difficulty of the modified MST was designed with decision making rules that required the similar numbers of WM requirements with the original MST test.

    Table 3 shows an example of modified MST rules. In this example, the foe team color was red and the friend team color was yellow. The marked numbers of foe players were 3 and 5. For low workload intensity, if the foe player number 3 (red color) appeared in the screen, subjects should press the left button. If the foe player number 5 (red color) appeared, they should press the right button. No pressing of any button was expected when unmarked numbers of foe players or friend players appeared. For high workload intensity, in addition to the marked number of foe players, participants were also required to remember the marked number and the position of the friend team. The simulation would include 3 players at the same time. In this example case, the marked friend number was 9. If the foe player number 3 (red color) appeared in the middle of screen and the friend player number 9 (yellow color) appeared on the left of screen, subjects should press the left button. If the foe player number 5 (red color) appeared in the middle of screen, along with the friend player number 9 (yellow color) appeared on the left of screen, subjects should press the right button. For other cases, participants were instructed to not press any buttons.

    2.3 Measurement of Workload

    NASA-TLX was used for assessing perceived mental workload in the present study. NASA-TLX consists of 6 subscales, including: Mental Demand (MD), Physical Demand (PD), Temporal Demand (TD), Performance (PE), Effort (EF), and Frustration (FR) (Cao et al., 2009). Following the standard experimental procedure of NASA-TLX, the scale definition was provided and explained to participants. MD indicates how much mental and perceptual activity (e.g., remembering, thinking, looking) is required for performing the task. PD refers to perception of physical demands influenced by the task. TD is associated with time pressure and the pace of the task. PE is rated based on degrees of accomplishment perceived by the participants. EF combines physical and mental effort to attain a desired performance. Lastly, the level of FR is indicated by the extent of frustration, stress and irritation experienced by the participants during task performance (Cao et al., 2009).

    Initially, participants were asked to choose the more important NASA-TLX subscale, when performing the given type of task, between all possible pairs of subscales. After completion of each task condition, they were asked to provide ratings on the subscales according to their perceived task workload. Finally, the weights of the subscales were tallied from the results of the paired-comparison, and the ratings of workload of the 6-subscales were combined into an overall workload score.

    2.4 Apparatus

    A high definition television (LG® model 43LK5000PTA, 968×625×217.8 mm) with HDMI (High-Definition Multimedia Interface) digital port was used to display both MST conditions. The DELL laptop with Windows 7 Ultimate systems (32 bit) was connected to the television to transmit videos of the test scenarios. Two buttons were used for left and right hand pressing, conveying positive responses. All hardware and software components of this device were obtained from Arduino, which is an opensource electronics platform. Figure 1 shows the experimental apparatus setup and an illustration of both MST versions delivered in this study.

    2.5 Procedures

    Prior to the experiment trial, participants were asked to read and sign an informed consent. Demographic information was then obtained through a self-administered survey. Subsequently, each participant was introduced to the simulated task apparatus and instructed on how to perform the MST. The subjects then underwent a randomization process to choose which version of MST to participate first. They also randomly chose which level of intensity (low or high intensity) they would perform first in each MST test.

    In this study, for both MST tests, the “memory set” was first presented to the subjects for memorization. A series of single test items was then presented to each participant one at a time. Participants were asked to respond positively (by pressing left or right button) if the test item was contained in the “memory set”, or negatively (no pressing of button) if not. Test items in both versions were randomly generated with the restriction that positive and negative set items were drawn with equal probability. Each version of the MST task was composed of 2 comparable demand levels, produced by variations in the number of items to be memorized (i.e., numbers of WM requirements). Based on a guideline of the original test, practice effects were eliminated using seven sets of 3-minutes training for each loading level. Participants were encouraged to respond as rapidly and accurately as possible. A maximum acceptable reaction time in the training mode was 15 seconds. If the subject did not respond within 15 seconds, the next item was automatically presented. In the testing mode, reaction time deadlines were reduced to 2.0 seconds for low intensity, and 2.5 seconds for high intensity. During test trials, subjects received feedback concerning the accuracy of their performance after each response. The letters or numeric items were set as approximately 0.5 ×0.7 cm and can be viewed from a distance of roughly 60 cm. Subjective workload rating using NASA-TLX was performed immediately after each trial. A 5-minutes rest period was set between two intensity levels and conditions. Figure 2 shows experimental procedure in this study.

    3. RESULTS AND DISCUSSION

    Table 4 summarizes the overall and subscale scores of NASA-TLX after performing the original and modified MST under two different levels of mental workload demand. The results show that an increase of workload intensity induced higher workload perception for both MST conditions. MD and EF were rated as most and second most important subscales, respectively, for original and modified MST tests. Participants also perceived high demand with respect to these subscales when performing both MST conditions with both low and high workload intensity. Therefore, it can be inferred that MD and EF contributed as major demanding components of this memory retrieval task type. When comparing NASATLX scores between original and modified MST tests, the overall workload, PD and FR scores were comparable for both low and high workload intensity (difference < 5%). However, MD and EF subscale workload scores of the modified MST found to be lower than the original MST for low workload intensity. TD scores of both level of intensity of the modified MST found to be higher than the original MST.

    In the present study, the NASA-TLX was proven to be sensitive to the change of task difficulty as the score of the high-intensity condition was found to be higher than that of the low-intensity condition in both MST tests. In line with the hypothesis, the higher mental resources demand in the high-intensity condition made subjects perceive higher workload (Haga et al., 2002). The present results were also in accord with previous MST studies (e.g., Rubio et al., 2004). It is important to note that, in the present study, participants reported a sleep duration between 7-9 hours prior to the experiment. They were also instructed to restrain from alcohol use for 48 hours and caffeine for 12 hours. Therefore, the performance and workload perception were neither affected by sleep deprivation, nor by alcohol and caffeine consumption.

    In agreement with our expectation, the comparable overall workload perception and similar demand for workload components of MD and EF suggested that the modified MST can be successfully applied to test WM retrieval function, similar to the original MST. The use of resources available in WM might lead MST to be recognized as mentally predominant demand. From an EF perspective, since none of the participants in this study had relevant experience with MST tests, the feeling of high mental demand might translate into increased effort in order to retain a good performance in task execution. The major demanding components of MD and EF for both original and modified MST in this study were similar to those identified in previous studies. For example, Rubio et al. (2004) also found that MD received the highest estimations for all conditions, followed by EF and TD, in their Sternberg MST study. In other context, Akyeampong et al. (2014) found an MD component to be perceived as the highest demand followed by OP and EF for a simulated hydraulic excavator operation task. Sirevaag et al. (1993) also reported MD and EF to be the two highest score components in both low and high load conditions when rotary wing aircraft pilots performed a reconnaissance task.

    Little demand was required in terms of PD and FR for this WM test type. The experimental setup in this study involved a seated position and required merely slight physical activity in pressing handheld buttons. Therefore, such low observed PD ratings appear completely reasonable. This result can also be useful for differentiating physical and mental workload perception when, in the future, additional physical requirement will be included in order to better represent the real scenario of a soccer match. Regarding FR perception, the low ratings might be related to the demographic characteristics of the participants, who were exclusively male university students. Participants might be familiar to similar stimuli used in this study, including the English alphabet, soccer player numbers and uniform colors, experienced during their studies and leisure time activities. Therefore, such types of stimuli might not influence frustration perception in the participant group.

    However, when considering MD and EF subscale workload scores, the modified MST found to induce lower ratings scored by participants than for the original MST during performance of low workload intensity. It is, however, in agreement with our hypothesis that the stimuli in the original and modified MST would induce different levels of mental workload perception. Such differences might be due to different types of spatial symbolic information posing different levels of mental demands (Wickens et al., 1998). In this case, the color code might be easier to remember and retrieve than letters and numeric information since it can be encoded in a semantic way (i.e., associated with opponent or friendly teams.) The strength or number of associations aids information in the LTM to become more available to be retrieved by the WM (Baddeley, 1997;Wickens et al., 1998). On the other hand, this study showed TD scores of both levels of intensity of the modified MST to be higher than for the original MST. This result might be due to the fact that the modified MST stimuli were not displayed as static targets (video of players jogging on the spot.) Participants might perceive less time available for dynamic motion in the modified MST condition, as compared with static figure in the original MST. Besides, it is also possible that participants were just unaware of any increase or decrease in the level of workload, since the NASA-TLX still remains a subjective workload measurement tool (O’donnell and Eggemeier, 1986;Rubio et al., 2004). As mentioned earlier, certain physiological measures were also proven to be sensitive to changes in workload intensity and supported continuous workload monitoring (Christ et al., 1993;O’donnell and Eggemeier, 1986). Future research should apply objective measures of workload, such as ERP and HRV, for more accuracy and real-time results.

    4. CONCLUSION AND RECCOMMENDATION

    Workload may reflect the relationship between resource supply and task demand (Christ et al., 1993). For most modern occupations, an accurate assessment of workload is necessary to ensure high decision accuracy and low error rates (Donaldson et al., 2000). Generally, the results in this study showed that an increase of workload intensity in both versions of the MST induced a higher workload perception. MD and EF were indicated as major demanding components contributing to the tasks requiring WM retrieval performance. The overall workload of both tests was also found to be comparable. The results obtained in this study thus support the applicability of the modified MST for testing WM retrieval functions, as has been previously established for the original MST.

    However, subscale workload scores of MD and EF of the modified MST were found to be lower than for the original MST. This may be explained by the fact that different types of symbolic information might require a different demand of mental resources, and therefore might lead to different levels of workload perception. In addition, static and dynamic stimuli might also contribute to different perception of time pressure and pace of the task. With such different levels of workload experienced by participants, our redesigned task stimuli would better represent actual characteristics of the task of interest (i.e., soccer match, in this case). Therefore, the modified MST was found appropriate to be used as stimuli for our subsequent experiment, which will include additional physical demand to more accurately approximate the real scenario of a soccer match.

    In a real situation, the stimuli are recommended to be carefully designed to represent actual characteristics of task. In addition, future research direction should include objective measures of workload for real-time measurement. The use of a combination of objective and subjective workload measures might give a more complete picture of the nature of the workload imposed by the MST test and relevant tasks.

    ACKNOWLEDGEMENTS

    This study was supported by a grant from the Office of Naval Research Global (ONRG) (No. N62909-18-1- 2082). The opinions expressed in this paper are those of the authors and do not necessarily reflect the views of ONRG.

    Figure

    IEMS-18-4-701_F1.gif

    Apparatus setup and illustration of: (a) original MST and; (b) modified MST.

    IEMS-18-4-701_F2.gif

    Study experimental procedures.

    Table

    Demographic characteristics of participants

    Example of original MST rule

    Example of modified MST rule

    NASA TLX weight sum score result of memory search task (MST) tests

    REFERENCES

    1. Akyeampong, J. , Udoka, S. , Caruso, G. , and Bordegoni, M. (2014), Evaluation of hydraulic excavator Human- Machine Interface concepts using NASA TLX, International Journal of Industrial Ergonomics, 44(3), 374-382.
    2. Andre, A. D. and Wickens, C. D. (1995), When users want what’s not best for them, Ergonomics in Design, 3(4), 10-14.
    3. Anthony, C. R. and Biers, D. W. (1997), Unidimensional versus multidimensional workload scales and the effect of number of rating scale categories, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, SAGE Publications Sage CA: Los Angeles, CA, 1084-1088.
    4. Baddeley, A. D. (1997), Human Memory: Theory and Practice, Psychology Press, UK.
    5. Cao, A. , Chintamani, K. K. , Pandya, A. K. , and Ellis, R. D. (2009), NASA TLX: Software for assessing subjective mental workload, Behavior Research Methods, 41(1), 113-117.
    6. Christ, R. E. , Hill, S. G. , Byers, J. C. , Lavecchia, H. M. , and Zaklad, A. L. (1993), Application and Validation of Workload Assessment Techniques, Analytics Inc Willow Grove Pa.
    7. Donaldson, M. S. , Corrigan, J. M. , and Kohn, L. T. (2000), To Err is Human: Building a Safer Health System, National Academies Press.
    8. Frackowiak, R. S. (2004), Human Brain Function, Elsevier Science, USA.
    9. Gathercole, S. E. , Brown, L. , and Pickering, S. J. (2003), Working memory assessments at school entry as longitudinal predictors of National Curriculum attainment levels, Educational and Child Psychology, 20(3), 109-122.
    10. Haga, S. , Shinoda, H. , and Kokubun, M. (2002), Effects of task difficulty and time‐on‐task on mental workload, Japanese Psychological Research, 44(3), 134-143.
    11. Hancock, P. A. (1996), Effects of control order, augmented feedback, input device and practice on tracking performance and perceived workload, Ergonomics, 39(9), 1146-1162.
    12. Hart, S. G. and Staveland, L. E. (1988), Development of NASA-TLX (task load index): Results of empirical and theoretical research, Advances in Psychology, 52, 139-183.
    13. Kane, M. J. , Brown, L. H. , McVay, J. C. , Silvia, P. J. , Myin-Germeys, I. , and Kwapil, T. R. (2007), For whom the mind wanders, and when: An experiencesampling study of working memory and executive control in daily life, Psychological Science, 18(7), 614-621.
    14. Kieras, D. E. , Meyer, D. E. , Mueller, S. , and Seymour, T. (1999), Insights into working memory from the perspective of the EPIC architecture for modeling skilled perceptual-motor and cognitive human performance. In A. Miyake and P. Shah (Eds.), Models of working memory: Mechanisms of active maintenance and executive control (pp. 183-223), Cambridge University Press.
    15. Klatzky, R. L. (1975), Human Memory: Structures and Processes, San Francisco: WH Freeman.
    16. Kyllonen, P. C. and Christal, R. E. (1990), Reasoning ability is (little more than) working-memory capacity?! , Intelligence, 14(4), 389-433.
    17. Lerch, F. J. , Mantei, M. M. , and Olson, J. R. (1989), Skilled financial planning: The cost of translating ideas into action, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM New York, NY, 121-126.
    18. Matthews, G. and Campbell, S. E. (2010), Dynamic relationships between stress states and working memory, Cognition and Emotion, 24(2), 357-373.
    19. Matthews, G. and Westerman, S. J. (1994), Energy and tension as predictors of controlled visual and memory search, Personality and Individual Differences, 17(5), 617-626.
    20. O’Donnell, R. D. and Eggemeier, F. T. (1986), Workload assessment methodology. In K. R. Boff, L. Kaufman, and J. P. Thomas (Eds.), Handbook of perception and human performance, Vol. 2. Cognitive processes and performance (pp. 1-49), John Wiley & Sons.
    21. Pesonen, M. , Björnberg, C. H. , Hämäläinen, H. , and Krause, C. M. (2006), Brain oscillatory 1-30 Hz EEG ERD/ERS responses during the different stages of an auditory memory search task, Neuroscience Letters, 399(1-2), 45-50.
    22. Proctor, R. W. , Wang, D. Y. , and Pick, D. F. (1998), An empirical evaluation of the SYNWORK1 multipletask work environment, Behavior Research Methods, Instruments, & Computers, 30(2), 287-305.
    23. Rasmussen, J. (1983), Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models, IEEE Transactions on Systems, Man, and Cybernetics, 13(3), 257-266.
    24. Reid, G. B. and Nygren, T. E. (1988), The subjective workload assessment technique: A scaling procedure for measuring mental workload, Advances in Psychology, 52, 185-218.
    25. Rubio, S. , Díaz, E. , Martín, J. , and Puente, J. M. (2004), Evaluation of subjective mental workload: A comparison of SWAT, NASA‐TLX, and workload profile methods, Applied Psychology, 53(1), 61-86.
    26. Ryu, K. and Myung, R. (2005), Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic, International Journal of Industrial Ergonomics, 35(11), 991-1009.
    27. Salvendy, G. (2012), Handbook of Human Factors and Ergonomics, John Wiley & Sons.
    28. Schacter, D. L. and Tulving, E. (1994), Memory Systems, MIT Press, Cambridge, MA.
    29. Shingledecker, C. A. (1984), A Task Battery for Applied Human Performance Assessment Research, Technical Report AFAMRL-TR-84-071, Air Force Aerospace Medical Research Laboratory, Wright-Patterson Air Force Base, OH.
    30. Sirevaag, E. J. , Kramer, A. F. , Reisweber, C. D. W. M. , Strayer, D. L. , and Grenell, J. F. (1993), Assessment of pilot performance and mental workload in rotary wing aircraft, Ergonomics, 36(9), 1121-1140.
    31. Sternberg, S. (1969), Memory-scanning: Mental processes revealed by reaction-time experiments, American Scientist, 57(4), 421-457.
    32. Wickens, C. D. , Gordon, S. E. , and Liu, Y. (1998), An Introduction to Human Factors Engineering (2nd ed.), Addison-Wesley and Longman, New York.
    33. Williams, M. , Davids, K. , Burwitz, L. , and Williams, J. (1993), Cognitive knowledge and soccer performance, Perceptual and Motor Skills, 76(2), 579-593.
    34. Xiao, Y. , Wang, Z. , Wang, M. , and Lan, Y. (2005), The appraisal of reliability and validity of subjective workload assessment technique and NASA-task load index, Zhonghua lao dong wei sheng zhi ye bing za zhi = Zhonghua Laodong Weisheng Zhiyebing Zazhi = Chinese Journal of Industrial Hygiene and Occupational Diseases, 23(3), 178-181.
    35. Zheng, B. , Jiang, X. , Tien, G. , Meneghetti, A. , Panton, O. N. M. , and Atkins, M. S. (2012), Workload assessment of surgeons: Correlation between NASA TLX and blinks, Surgical Endoscopy, 26(10), 2746-2750.