ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal Regular Issue, Vol. 11 N. 2 (2022), 147-158 eISSN: 2255-2863 DOI: https://doi.org/10.14201/adcaij.27184

Optimization of Window Size for Calculating Semantic Coherence Within an Essay

Kshitiz Srivastava^a, Namrata Dhanda^b and Anurag Shrivastava^c

^a Ph.D. scholar, Dr. APJ Abdul Kalaam technical university, Lucknow, Uttar Pradesh, India

^b Professor, Department of computer science, Amity University, Lucknow, Uttar Pradesh, India

^c Professor, Department of Computer Science & Engineering, Babu Banarasi Das Northern India Institute of Technology, Lucknow, Uttar Pradesh, India
akshitiz.sri@gmail.com, bndhanda@lko.amity.edu, canuragbbdlko@gmail.com

ABSTRACT

KEYWORDS

1. Introduction

In the world of computer science, automated essay assessment is always seen as a challenge. It is both a natural language processing application and a statistics classification challenge. There are numerous automatic essay evaluation systems that have been designed and implemented, Psalmerosi, F. H. (2019). Some are already on the market, while others are still in the works. Ke, Z., and V. Ng (2019). Thousands of pupils in the United States are scored each year using systems like Project essay grader, which was developed in 1966 and is currently a product of Measurement Inc. Systems like C-rater, on the other hand, were unable to influence the marketplace with their performance, Y. Attali, (2011). C-rater is now a standard feature of many other grading systems. With advancements in the field of natural language processing, the need for essay grades that are closer to human scores has grown.

All these systems work on attributes, these attributes denote the different features of the essays, Chandrasekaran, D. et al., (2020). Attributes are broadly classified into three categories: style, content, and semantics of the essay, Khatavkar, V., and Kulkarni, P. (2019). These attributes are subdivided into smaller attributes for example lexical sophistication, Readability measures, Lexical diversity, mechanics, content, etc. These sub-attributes may be a group of other smaller attributes for example lexical sophistication may include all or few of the following sub-attributes: number of characters, number of words, number of long words, number of short words, etc. Zupanc, K., and Bosnic, Z. (2014).

All of the proposed systems have prioritized style and content features, with semantic attributes receiving less attention, Azmi, A. M. et. al. (2019). Measuring the semantic of an essay becomes a huge difficulty due to differences in writing style and content length. A few features were proposed in 2014, and these qualities were integrated as an important component of SAGE Zupanc, K., and Bosnić, Z. (2014), allowing us to quantify the semantic coherence between the portions of the essay. These attributes were able to measure semantic coherence by splitting the essay into smaller sections using various reduction techniques, then placing these portions as points in a high-density semantic space and measuring attributes using various methodologies. In one experiment, a window of around 25% of the essay’s entire size was taken, and a smaller corpus was identified by moving the window by 10 words Zupanc, K., and Bosnic, Z. (2016). In our work, we are trying to identify the appropriate window size that can increase the overall performance of the system. For the experiment, we have taken data from the Kaggle website, A. Mellor, (2011).

2. Ethics

The field of Automated Essay Evaluation is carried out in different problem areas, A. Mellor, (2011). In the following subsection, we have discussed these problem areas.

2.1. Automated Essay Evaluation Systems and their Approaches

Project Essay Grader was the first automated essay evaluation system, proposed by Elis B Page M. D. Shermis and J. Burstein (2003). It was developed in the mid-1960s. According to Page, the system was a better solution to manual grading. Because of the technology of that time his project was not much accepted by the community, as the operational cost was too high. With the advancement in technology, the internet, and text processing software, development in the field of automated essay evaluation systems took a great pace. Many systems and techniques were proposed and implemented. A few of the mentions systems are Intelligent essay assessor Srivastava K., Dhanda N. & Shrivastava A. (2020), Intellimetric M. T. Schultz (2013), Project essay grader, and E-Rater J. Burstein, J. Tetreault, and N. Madnani (2013).

One of the major obstacles in achieving success in this area is the non-availability of any open-source automated essay evaluation system. Most of the systems are either commercially available (PEG, E-rater, Intellimetric, etc.) or under development. The only system whose compiled code along with source code was publicly available was LightSide, developed by Mayfield and Rose Mayfield, E., and Rosé, C. (2010). It was designed to do various these, in which essay grading was one of the major features.

The quality of an essay can be measured by focusing on three basic attribute styles, content, and semantics. Style attribute focuses on the way the essay is written spelling, punctuations, grammar, etc. The content attribute is based on a comparison of the essay with the pre-graded essay. This attribute is used to compares two essays and finding the similarities between them. The semantic attribute is used to verify the correctness of the essay, Chandrasekaran, D., and Mago, V. (2020), D. Higgins, J. Burstein, D. Marcu, and C. Gentile (2004), Darwish, S. M., and Mohamed, S. K. (2019).

Numerous methodologies are used to extract attributes from the essay. Latent semantic analysis is one of the most popular methodologies used. Other methods that are used are pattern matching, sentence similarity networks, Generalized Latent semantic analysis, n-gram approaches, etc. For correctness and consistency semantic networks, ontologies, fuzzy logic, open information extraction, etc. are used Zupanc, K., and Bosnic, Z. (2016); Ferreira-Mello et al. (2019); T. K. Landauer et al.,(1998).

Most of the system uses Machine learning algorithms to create the prediction models like regression modeling, cosine similarity, rule-based expert system, etc, Romero, C., and Ventura, S. (2020). The idea is to use a pre-graded essay to create a prediction model, T. Kakkonen, et al., (2008). This prediction model is then used to grade the rest of the essays, Peng, C., et al. (2020) The performance of such a model is calculated by the comparing score obtained with the actual score, Gao, Y. et al., (2019); P. W. Foltz, et al. (2013); P. W. Foltz, et al. (2007). A brief comparison of a few of the available systems is given in Table 1 Srivastava K., Dhanda N. & Shrivastava A. (2020).

2.2. Semantic Coherence

Coherence is defined as the flow of information from one part of the essay to other parts Mimno, D., Wallach, H., Talley, E., Leenders, M., and McCallum, A. (2011). A highly coherent essay has a low coherence movement whereas a low coherent essay has high movement. The system proposed till now either measures the coherence using supervised or unsupervised approaches, Darwish, S. M., and Mohamed, S. K. (2019).

In unsupervised approaches system usually measures the repetition of words or repetition of phrases assuming that a highly coherent essay has high repeating words and phrases Zupanc, K., and Bosnic, Z. (2016). According to Foltz, P. W., Laham, D., and Landauer, T. K. (1999). a highly coherent essay contains high semantically related words and essay. Hearst Hearst, M. A. (1997) has proposed that an essay can be subdivided into smaller parts and can be used in identifying the repetition of semantically related words and phrases. The most commonly used approach is latent semantic analysis.

In a supervised approach, pre-graded essays are used, D. Higgins,et al., (2004); M. D. Shermis and B. Hamner (2013). According to the centric theory Grosz, B. J., Joshi, A. K., and Weinstein, S. (1995) «The extent a discourse adheres to centering constraints, its coherence will increase and the inference load placed upon the hearer will decrease». This phenomenon helps in locating topic shifts in the essay, Goulart et al., (2018), Foltz, P. W. (2007).

2.3. Semantic Coherence Measurement

The first step towards semantic coherence measurement is to identify the attributes upon which we measure semantic coherence, Injadat, M. et. al. (2020), Janda, H. K. et. al. (2019). Around 72 different attributes (including all types of attributes: style, content, and semantics) are considered by different AES systems Zupanc, K., and Bosnić, Z. (2017), J. Burstein, (2010). We have considered four semantic attributes upon which we will be calculating semantic coherence, Muangkammuen, P., and Fukumoto, F. (2020), D. Higgins et al., (2004). These attributes are attribute-1 (average distance) [8], attribute-2 (Minimum distance), attribute-3 (Maximum distance), and attribute-4 (average nearest neighbor), Romero, C. and Ventura, S. (2020).

To calculate these attributes first data is cleaned by: lowering, stemming, stop-word removal and correcting spelling, Misuraca, M. et al., (2021). Then data obtained is divided into smaller datasets by choosing a window of certain length and moving it by certain words to obtain sequential overlapping parts, J. Burstein, et. al. (2010). For this example, our window consists of 25% of the overall size of data. This window is moved by 10 words to have the next overlapping part Zupanc, K., and Bosnić, Z. (2017).

Each dataset obtained is represented as point in semantic space by first calculating Tf-idf (statistical method). Tf-idf provides the weights to the words in a particular document, Janda, H. K. et al., (2019).