Project Planning and Objectives Estimation

Project Overview

This project aims to create a chatbot that analyzes the user's emotions and conveys empathy and comfort in the way the other person wants. Analysis of the user's emotions is analyzed in two ways: text analysis and facial images analysis. Analysis through text aims not only to capture words representing specific emotions, but to infer the user's emotions by grasping the context. In addition, the method of treating according to emotions allows users to respond in the way they want, such as friends, parents, and lovers.

 

 

Key Features

- Text-based emotional analysis: It analyzes emotions by analyzing text input by the user. 

- Image-based emotional analysis: analyze the facial image image image that the user posted by analyzing the facial image. 

- Provides response to customized comfort: Based on the analyzed emotions, the user responds with the desired type (EX/parents, friends, lovers, etc.).
- Personal custom: Continuous conversation analyzes minor patterns in individual texts, images, etc. to derive more sophisticated responses.

 

 

Model Implementation

 

The text recognition model and the image recognition model are distinguished and implemented respectively. In the initial plan, the text dataset and the image dataset were combined to be implemented as a single dataset, but due to the problem of data size mismatch, the model that recognizes text and images at the same time was not immediately implemented, but after implementing the text recognition model and the image recognition model respectively, it was decided to create a recognition model by combining them.

 

The text recognition model implements NLP (natural language processing), especially after analyzing the morpheme of Korean, grasping the context, and inferring emotions. 

KoNLpy is used for Korean morpheme analysis.
In the preprocessing process, the dataset is divided into training, validation, and test sets and calculated at a ratio of 8:1:1.
The training text dataset, which has been preprocessed through NLP, is applied to the LSTM model to proceed with training, and tested with the test text dataset, which is tested 20 times with epoch=20 and the performance of the model is gradually improved. The performance of the model is judged based on Accuracy.  
The performance of the model is aimed at Accuracy Score 0.90 or higher, and if the baseline score is not met, the model is gradually improved through hyperparameter tuning.

 

The image recognition model is largely divided into a training set and a test set in the entire dataset, and 80% is trained and 20% is prepared as a verification set in the training set. Each training, verification, and test set are calculated at a ratio of 3.2:0.8:1. Each prepared dataset was preprocessed through data augmentation and normalization. 
The preprocessed data is applied to the EfficientNetB0 model to proceed with training, and the test is performed with a test image dataset, and the performance of the model is gradually improved by testing it 20 times with epoch=20. The performance of the model is judged based on Accuracy.

 

Combining the two completed models, we implement one recognition model and test it in CoreML by adding other features.

 

Technical stack and development environment

Programming Language: Python

Text Emotion Analysis Model: KoBERT (Korean BERT) (Context-based Emotion Analysis), LSTM + Word2Vec (Current Neural Network for Emotion Analysis)

Image Emotion Analysis Models: CNN (Face Emotion Analysis), EfficientNet (Face Emotion Prediction)

Text Dataset : aihub Emotional Conversation Dataset

https://aihub.or.kr/aihubdata/data/dwld.do?currMenu=&topMenu=&dataSetSn=270&beforeSn=274&inqrySeCode=&intrstDataAt=N&reloadYn=N&useAt=

Image Dataset: FER2013 (Face expression dataset)
https://www.kaggle.com/datasets/msambare/fer2013

 

Data processing: Numpy (multidimensional array and numerical operations, optimization of vector operations of emotion analysis results), Pandas (storage and analysis of emotion analysis results in data frame format, emotion analysis evaluation and statistics processing), Tensorflow (training and optimization of text emotion analysis models, building CNN models for image emotion analysis)

Development Tools: Jupiter Notebook

 

Expectation Effectiveness

It provides customized comfort services through emotion analysis and can be used for various services dealing with emotions (psychological counseling, etc.).