VR CogLoad Dashboard for Flight Simulator

Подробнее о видео

We are developing a virtual reality flight simulator and cognitive load estimation system to enhance pilot and passenger safety.
Initially, we worked on developing a desktop flight simulator and developed cognitive load estimation systems using eye tracking glasses. We undertook multiple simulation studies with desktop flight simulator to correlate ocular parameters with flight handling qualities measured through Duty Cycle and Aggressiveness of flying, which are in turn indicative of pilots’ interceptor workload.
The cognitive load estimation algorithms were validated with ocular data collected in actual flight environments. Ocular parameters found to be indicative of flight handling qualities and flight situations. For example, we reported a statistically significant correlation between fixation rate and rate of descend of aircraft during air to ground dive attacks.
We set up the VR flight simulator to test different collaborative and competitive situations that are difficult to simulate in actual flights. The VR flight simulator generates a more realistic and immersive environment for flight handling quality testing than desktop flight simulator. We set up target tracking tasks in the VR flight simulator and compared ocular parameters and EEG parameters of professional pilots and university students. We developed a cognitive load dashboard for visualizing change in ocular and EEG parameters as users interact with the flight simulator.
Before developing the 3D dashboard, we undertook a VR study to investigate 3D graphical primitives. In particular, we evaluated three graphical primitives for representing numerical data and two primitives for depicting nominal data. We compared six different 3D graphs involving two graph types and five graphical primitives. We analyzed ocular and EEG parameters of users while they undertook data interpretation task in 3D graphs. We designed and set up a VR environment scene using unity 3D game engine. The scene consist of a visualization chart and a set of four questions. User were requested to undertake the experiment by observing the chart and then answering the questions related to the chart. This is one example, where the numerical data are represented by the height of the bars and nominal data are represented by colors of the bars, while in this chart the opacity of bars depicts numerical value while colors depict nominal value. We found that primitives like size and color makes it easier to encode numerical and nominal data. We also found that bar and area charts are better than other charts for correct data interpretation. We further found that size incurs less motor action and cognitive load. We used the results of this study to develop the cognitive dashboard. For example, we used 3D bar and area charts to display EEG data, and all graphs of the dashboard used size primitive to represent numerical data while color was used to depict nominal data.
This dashboard helps us to visually analyze the ocular and EEG parameters to investigate gaze behavior and estimate cognitive load. The dashboard is divided into three modules: 2D graphs, 3D graphs and complex graphs. You can click on any of the following types of visualization to start the analysis. Let us click on the left of the main to analyze 2D graphs.
This is the 2D interface of the dashboard, the bubble graph here is showing fixation rate and duration. The size of the circle represents duration and y-axis represents fixation rate. You can also see a video of the task undertaken in an F16 flight simulator at the top of the screen. The pupil diameter, saccadic intrusion and EEG data while undertaking task are shown here in a 2D line graph and bar graph. You can also switch between the graphs by clicking on the switch graph button.
Let us now analyze 3D graphs. This is the 3D interface of the dashboard. The 3D bar charts here are showing EEG data of four different bands for five different electrodes. The corresponding video of the task is shown in the upper left corner. We can also select any specific band from the drop down box. 3D line graphs are showing pupil diameter and Saccadic intrusion while 3D radar charts shows four types of ocular parameters