Drowsiness Detection Using Image Processing

Table of Contents

Acknowledgement

Abstract

Abbreviation

List of Figures

List of Tables

Chapter 1: Introduction
1.1 Rationale of the topic
1.2 Problem identification
1.3 Aim and Objectives
1.3.1 Aim
1.3.2 Objectives
1.4 Scope and Limitations
1.4.1 Scope
1.4.2 Limitations
1.5 Report Overview

Chapter 2: Literature Survey
2.1 Types of image processing software.
2.2 Techniques to identify fatigue.
2.3 Fatigue identification follow ups
2.3.1 Non self-driving mode
2.3.2 Self-driving mode

Chapter 3: Methodology
3.1 Conceptual Design
3.1.1 Conceptual Design 01
3.1.2 Conceptual Design 02
3.1.3 Conceptual Design 03
3.2 Selecting the Optimum Design and Justification

Chapter 4: Design & Implementation
4.1 Software Implementation
4.1.1 Image acquisition
4.1.2 Image Processing
4.1.3 Detection
4.2 Hardware Implementation
4.2.1 Alarm
4.2.2 Speed Control
4.3 The System

Chapter 5: Testing & Analysis

Chapter 6: Conclusion & Further Development
6.1 Conclusion
6.2 Further Development

References

Project Management & Control

6.3 Gantt Chart
6.4 Meeting Records

Appendix 01

Appendix 02

Appendix 03

List of Figures

Figure 2.1 Literature Mapping

Figure 3.1.1 Flow Chart - Conceptual Design 01

Figure 3.1.2 Flow Chart - Conceptual Design 02

Figure 3.1.3 Flow Chart - Conceptual Design 03

Figure 4.1.1 Code for image acquisition and face detection.

Figure 4.1.2 Face detection

Figure 4.1.3 Code for face cropping and eye detection.

Figure 4.1.4 Eyes Detection

Figure 4.1.5 Code for isolating eye region and recognizing blinking

Figure 4.1.6 Image Processing with closed eyes

Figure 4.1.7 Code to recognize open eyes

Figure 4.1.8 Image Processing with open eyes

Figure 4.1.9 Code to detect drowsiness based on blinking rate.

Figure 4.2.1 Circuit design using Circuit.io

Figure 4.3.1 Systems integration

Figure 4.3.1 Incorrect pattern match

Figure 4.3.2 Drowsiness detection in Matlab

Figure 4.3.3 Accurate identification under less light

Figure 4.3.4 Incorrect detection with glasses

Figure 8.1 Gantt Chart

Figure 9.1 Test results under less light - open eyes.

Figure 9.2 Test results under less light - closed eyes.

Figure 9.3 Test results of open eyes with glasses

Figure 9.4 Test results of closed eyes with glasses

Figure 10.1 Eyes open - without glasses

Figure 10.2 Eyes closed - without glasses

Figure 10.3 Eyes open - with glasses

Figure 10.4 Eyes closed - with glasses

Figure 11.1 The camera with 50 degrees alignment and an externally integrated lighting

Figure 11.2 System integrated with alarm, proximity sensors and automatic braking system.

List of Tables

Table 4.3.1 Number of blinks per minute

Table 4.3.2 Test results

Table 4.3.3 Test results (long distance)

Table 8.1 Meeting records with the supervisor.

Acknowledgement

Mr. Dillum Fernando from the International College of Business and Technology (ICBT) oversaw this project. Therefore, I thank him for offering understanding and knowledge that has significantly helped the research and results and for remarks that have significantly enhanced the report.

I thank Ms. Lanshiya Pearlien for assisting me with the project management aspect of the project. I thank Mr. Shukri Razick for assistance with designing a system to control the speed of the vehicle.

I would like to say thanks to two anonymous reviewers for giving me insights on this project. I am also immensely grateful to Dr. Kallum Kathriarrachhi, Mrs. Aravindy Weerasinghe and Mr. Sanka Gunawardena 's remarks on a previous version of the study, although any mistakes are my own and should not undermine the reputation of these esteemed professionals.

Abstract

When a driver doesn’t get proper rest, they fall asleep while driving and this leads to fatal accidents. This particular issue demands a solution in the form of a system that is capable of detecting drowsiness and to take necessary actions to avoid accidents.

This report explains the final project, driver drowsiness detection system. The detection is achieved with three main steps, it begins with face detection and facial feature detection using the famous Viola Jones algorithm followed by eye tracking. By the use of correlation coefficient template matching, the eyes are tracked. Whether the driver is awake or asleep is identified by matching the extracted eye image with the externally fed template (open eyes and closed eyes) based on eyes opening and eyes closing, blinking is recognized. If the driver falling asleep state remains above a specific time (the threshold time) the vehicles stops and an alarm is activated by the use of a specific microcontroller, in this prototype an Arduino is used.

Abbreviation

Abbildung in dieser Leseprobe nicht enthalten

Chapter 1: Introduction

1.1 Rationale of the topic

A micro sleep is something we all may have experienced at least once in our life, where we momentarily fall asleep and then immediately awake up. And it is a very common thing among vehicle drivers, they fall asleep and awake up immediately and continue with their previous activity and it will not be an issue as long as it is a straight road and there is no vehicle ahead of them or behind them but at the same time if the driver is driving along a curve or the motor vehicle ahead of the driver is applying a sudden brake then the reflex action by the driver will not be fast enough to take appropriate action and mostly result in an accident.

Accidents due to sleep deprivation is a common problem all around the world. According to a database study conducted by the Australian Transport Bureau in 2002 about fatality crashes [2] it is found that 16.6% of fatal crashes were caused by sleep deprivation and based on the report submitted by Ministry of Transportation of Ontario in 2004 [16], 25.5% of injury causing crashes and 17.8% of crashes with fatality related to sleep deprivation. All of the above statistics is an estimate because in severe cases the drivers will not admit the real cause to the Police.

The Sri Lankan law on driver fatigue is described in Section 169 of the Motor Traffic Act [17] [4], which reads as follows:

1. No individual shall drive, cause or allow any individual to ride any motor tricycle hired by him/her or subject to his/her instructions. Van, truck or motor coach:

- For any stretch of higher than four and a half hours continuously; or
- Consequently, the vehicle driver does not have a minimum 10 continuous hours of sleep in any twenty-four-hour period calculated from the beginning of any driving period.

2. Because of the purposes of the subsection of the first point, any two or more-time period will not be considered as a consecutive period of time, except when the time periods are splited by a break of minimum thirty minutes in which the driver will be able to get some refreshment and rest.

In reality the majority of the people does not follow any of the above law and it is very difficult even for a traffic Police officer to penalize the driver with any of the above law. And this is where technology comes in, there have been a lot of devices that were invented around the world to reduce this cause but most of them are very expensive or not available in the local market. In this report, I discuss about a system that uses computer vison and image processing to control the damage caused by the above stated phenomenon.

1.2 Problem identification

Driver drowsiness is one of the main causes of road accidents around the world. Even though there are many road safety products designed to prevent road accidents due to driver drowsiness, most of them are either very expensive or produce false alarms or are not available easily in Sri Lanka.

1.3 Aim and Objectives

1.3.1 Aim

To design a system that will detect drowsiness and take necessary steps to avoid accidents. The driver drowsiness detection system, being implemented in this project aims at being easily available and can be used with different types of vehicles.

1.3.2 Objectives

- Choosing a suitable software for image processing.
- Design a suitable algorithm for fatigue recognition.
- Design a system to control the brake and activate an alarm via Arduino.
- Design a prototype for driver drowsiness detection system.
- Testing and result analysis.

1.4 Scope and Limitations

1.4.1 Scope

The fatigue detection system in this project runs on MATLAB and is capable of detecting drowsiness on drivers and then based on the current situation will either slow down the vehicle or stop it completely and alert the driver using an alarm.

1.4.2 Limitations

The fatigue detection in this project is done using Image processing and Computer vision techniques.

1. The algorithm that is used to detect the eyes (which plays the major role in this project), produces a high inaccuracy when tested with wearing spectacles, which is a major drawback.
2. Tilting the head downwards also causes inaccuracy in fatigue detection.
3. MATLAB, the software used for image processing in this project consumes a lot time to process real time videos.

In the near future the system will be modified and adjustments will be made in order to cope up with the above mention drawbacks.

1.5 Report Overview

This report contains six chapters. The first chapter will discuss about the background of the problem identified, the gap in the research and al last the scope and limitation of the proposed project.

In chapter 02, the description is about the previous researches carried out on drowsiness detection. It includes Convolutional Neural Network methods, basic theories and other different types of techniques.

In chapter 03, three conceptual designs are discussed in order to select an ideal design for the proposed project. The selected design with software implementation, hardware implementation and calculations are discussed in chapter 04.

In chapter 05 the results of the protype are discussed. Chapter 06 contains the conclusion and the future recommendation for my research.

Chapter 2: Literature Survey

Abbildung in dieser Leseprobe nicht enthalten

Figure 2.1 Literature Mapping

2.1 Types of image processing software.

According to (Machaca Et al, 2018) [12] Image processing is done by converting the image in to a two-dimensional matrix where each pixel of the image is corresponded by each element of the matrix. The image should be prepared in a way so that the algorithms that will be performed on it will be very effective, in other words the contrast and the noise is adjusted and reduced in the image by the use of different types of filters before the main algorithm is performed on to it.

(Essayed & Yousef, 2019) [1] states that, MATLAB is well known for its good support and documentation and also its code is very easy to debug but, on the other hand MATLAB is not an open source and its license is expensive and also MATLAB takes more time to process the code so doing real time image processing would be a challenge when working with MATLAB, however the program can be modified to produce fast output by the use of high-level functions.

According to (Cheng, W.-C., & Xu, Y.-S, 2013) [21] OpenCV is well known for being an open source and it has a larger user community. The OpenCV has more features when it comes to computer vision and image processing compared to MATLAB and it takes a very little processing time. If the library of OpenCV finds the Integrated Performance contents of Intel on the system, it will use it to speed up the process and optimize the performance. But when it comes to filtering, the task becomes difficult compared to that of MATLAB.

(Divya et al, 2016) [18] states that ImageJ is an image processing program for open source java images. The software provides a toolbox that is common for interactive image manipulation and analysis. ImageJ can also be efficiently extended by composing parts of a program using an authentic programming language. This software is mainly independent of the platform, the implementation is fully done in Java. And this fast reversal makes the software an optimal phase to create and develop different methods and algorithms for image processing.

According to (A. Atrish and V. Kumar, 2017) [10] DIP VLabb is mainly intended to encourage users to perform tests and researches without worries about programming difficulties, that enables users to keep up their experimental curiosity and interest rather than execution. LabVIEW is used to create a laboratory that is virtual and offers different characteristics such as flexibility, robustness, scalability and more.

2.2 Techniques to identify fatigue.

According to (Machaca Et al, 2018) [12] the image acquired from the camera is filtered to reduce noise and then adjust contrast following this, the face is detected and then using ROI eyes are isolated and the eyes are tracked and by comparing the eyes with two externally fed templates eye opening and closing pattern is recognized which is used to calculate the rate of eye blinking, if the eye blinking rate is higher than the threshold frequency then the driver is classified as not fit to drive. The entire process is carried out with MATLAB.

(Manu BN, 2016) [13] states that fatigue can be recognized by following three steps. Step one is to detect the face using the Viola Jones algorithm, step two is to track eyes and recognized blinking and then in the third step the yawning is also detected. Once the face image is detected the background is removed by using skin color detection in the face region. Yawning is detected by using a template matching technique with correlation coefficient. Then vector extracted from each of the above step is then linked together by using a vector classifier and classify the result as driver fit to drive or driver not fit to drive, if it is above the threshold time. The entire process is carried out with MATLAB.

According to (Singh, H. et al, 2011) [7] in addition to eye blinking detection, the speed of the vehicle can also be taken in to account as there is a possibility for sudden change in speed (acceleration / deceleration) due to driver fatigue. So, when all three inputs (eye blinking, threshold time and speed of the vehicle) indicates fatigue then the driver is classified as not fit to drive.

According to (Zhao Et al, 2011) [5] a more advanced technique can be used to produce a very high accuracy. In this technique Viola Jones algorithm is used for face detection and Gabor wavelet transformation is used for filtering. With the use of classifiers such as Multilayer Perceptron (MLP), a data set of fatigue expressions developed. The fatigue in driver is classified as awake, moderate fatigue and severe fatigue by the use of expression data set.

(Ma, X., Chau, L.-P., & Yap, K.-H, 2017) [20] states that, Convolution based Neural Network (CNN) can be used with depth video sequences to detect driver fatigue in night time. The method is different from all the above, mentioned techniques because it provides accurate results in less light. The algorithm uses motion vectors. Driver drowsiness is detected by taking yawning and mouth opening only.

(Warwick et al, 2015) [3] states that, drowsiness in a driver can be determined by the use of wireless wearable devices. This can be done by using a biosensor called Bio Harness. This method consists of two stages. In stage one, the biosensor will collect physiological data from the driver and measure and analyze the data to find signs related to drowsiness. In the second and the final stage, an algorithm designed to detect drowsiness uses a mobile app to alert the drivers in necessary times.

(Warwick et al, 2015) [3] also states that taking physiological readings as key parameters to determine drowsiness, produce more accurate results than using readings based on behavior and vehicle.

According to (Zilberg et al, 2007) [6] a noninvasive device could be made using piezofilm motion sensors and can be used to detect different levels of drowsiness, using only the piezofilm reading or in combination with some other physiological readings. In this method, piezofilm motion sensors are integrated with car seat, steering wheel and seat belt. An algorithm is developed that uses regression models and sequential fitting to detect drowsiness. This uses the underlying principle of statistical analysis that suggest during drowsiness the motion reading recorded by the sensors integrated decreases.

(Katyal et al, 2014) [22] states that, drowsiness in vehicle driver can be determined by detecting the lane in addition to eye closure. Hough Transformation is used for lane detecting, it basically detects lines in an image. As in this case it is used to detects the lane markings of the road. Given that, the driver will give a signal, whenever he is going to cross the lane marking. If a change in lane occurs without a signal, it is considered as a parameter for drowsiness detection. If this happens during the extraction of a fatigue sign by the eyes, then the driver is classified as a fatigue and a warning signal is given out.

2.3 Fatigue identification follow ups

(Manu BN, 2016) [13] States that once the driver is classified as not fit to drive, necessary steps must be taken to avoid accidents. This is achieved by using a microcontroller.

2.3.1 Non self-driving mode

According to (Ma, X., Chau, L.-P., & Yap, K.-H, 2017) [20] one way to alert the driver after severe fatigue detection is to generate vibration in the seat belt with frequency of around 200 – 300 Hz.

According to (Machaca et al, 2018) [12] after detecting patterns of drowsiness, an alarm can be used to alert the driver. The alarm is modified to give out sound in relation to the level of drowsiness detected. Therefore, the driver doesn’t get used to the sound and ignore it.

(M. Maiti et al, 2015) [11] states that, if drowsiness pattern is detected then based on the conditions the alarm can be generated and the vehicle is halted by using an auto-braking system.

(M. Maiti et al, 2015) [11] also states that, in order to alert the driver, audio speakers can be used and messages can also be sent to the driver’s phone.

2.3.2 Self-driving mode

According to (A. K. Jain, 2018) [9] a much complex but advanced approach would be to put the vehicle in self driving mode. In this approach, a camera is attached to the top the vehicle which would get the input and then the video input is processed by Raspberry Pi using Convolutional Neural Network which then predicts which direction to go. But this technique would involve a lot of hardware implementation in order to achieve accurate results.

According to (Q. Memon et al, 2016) [14] a self-driving vehicle can be designed by integrating the processor with Google Maps API and suitable proximity sensors. This way the vehicle can detect obstacles, reroute and move without colliding.

According to (Zhang et al, 2010) [23] FNN can be used to design unmanned vehicle. Using FNN the algorithm can learn from training examples fuzzy logic laws and ideal membership features. Using this technique, it is possible to control a hybrid vehicle by training the system with skilled driving techniques.

(W. Lin et al, 2018) [19] states that, a combination of Vision-Based Deep Learning and Feedback Control can be used to develop an unmanned road vehicle. In this method it can be made by developing a smart agent that could imitate human behaviors for driving. Human beings in general use eye vision to recognize the surrounding environment, speed and the position of the vehicle, they drive, similarly the system in this method use deep neural network as a the basis of understanding the position of the vehicle in relation with the road, hence it uses this information as a key element for the feedback control.

Abbildung in dieser Leseprobe nicht enthalten

Figure 2.2 Findings of literature survey

Chapter 3: Methodology

MATLAB is chosen as the software to perform image processing in this project and all three conceptual designs discussed below are developed using it. OpenCV is opensource and its speed is higher in real time video processing than MATLAB. Regardless of this, MATLAB was chosen because of its ability to visualize easily, the language is much simpler, convenient while debugging and furthermore based on a thorough research it was concluded that MATLAB is much more convenient for prototyping whereas software like Visual Studio (OpenCV) and ImageJ are good for production.

3.1 Conceptual Design

3.1.1 Conceptual Design 01

Abbildung in dieser Leseprobe nicht enthalten

Figure 3.1.1 Flow Chart - Conceptual Design 01

In this design, first the driver’s face image is captured by the use of a camera fixed to the vehicle. Following this there are four steps to identify drowsiness in driver, as shown in figure 3.1.1. Firstly, the video input undergoes noise filtering, then the face is detected and cropped out. In the second stage the eye area is detected from the cropped image of the face. Thirdly the mouth region is detected from the face.

These two tasks work together to enhance the outcomes of face detection. Then the removed mouth and eye closure functions are introduced to the removed mouth and eye regions. These functions perform operations on those regions to determine signs of drowsiness, that is closing of the eye and yawning.

Next, the outcomes will be combined and the driver's fatigue will be determined. If both of the parameters show signs of drowsiness at the same time then the driver is classified as not fit to drive, if either one of the parameters doesn’t show signs of drowsiness then the output is not passed on to the next stage of the process, instead the reading is taken again. Finally, if a state of drowsiness is determined, then an alarm is activated and the brake of the vehicle is pressed by the use of a buzzer and a motor attached to an Arduino respectively. The face is then monitored and the operation is repeated in the next captured frames.

3.1.2 Conceptual Design 02

Abbildung in dieser Leseprobe nicht enthalten

Figure 3.1.2 Flow Chart - Conceptual Design 02

This approach contains two subsections and this is basically a fusion of two different methods proposed by (Zhao Et al, 2011) [5] and (Singh, H. et al, 2011) [7]. The system first uses Wavelet Transform and Neural Network (Zhao Et al, 2011) [5] to detect drowsiness pattern in drivers and if drowsiness is detected then the system detect sudden change is velocity by using the speedometer reading (Singh, H. et al, 2011) [7].

[...]