Jurnal Informatika Vol.
No.
January 2023, pp.
ISSN 1978-0524 .
| 2528-6374 .
http://journal.
id/index.
php/JIFO/index Towards a Complete Kurdish NLP Pipeline:
Challenges and Opportunities Dastan Maulud a,1.
Karwan Jacksi b,2,*.
Ismael Ali b,3 Department of Information Technology.
Technical College of Informatics-Akre.
Duhok Polytechnic University.
Duhok.
Kurdistan Region Ae Iraq.
Department of Computer Science.
University of Zakho.
Duhok.
Kurdistan Region Ae Iraq.
mawlud@mhe-krg.
2 karwan.
jacksi@uoz.
3 Ismael.
Ali@uoz.
* Corresponding Author Received 25 October 2022.
accepted 20 December 2022.
published 10 January 2023
ABSTRACT
With the rapid growth of Kurdish language content on the web, there is a high demand for making this information readable and processable by machines.
In order to accomplish this, the Kurdish Natural Language Processing (KNLP) pipeline is required.
Computers that can process human language use the field of Natural Language Processing (NLP).
In its efforts to bridge the communication gap between humans and computers.
NLP draws from a wide range of fields, including computer science and computational linguistics.
There have been some notable efforts made toward creating the KNLP However, it does not support the complete NLP tasks needed to enable semantic web and text mining applications.
This paper surveys the work done in the field of NLP for the Kurdish language, its applications, and linguistic challenges.
KEYWORDS
Text Corpus Annotated Corpus Kurdish Language NLP Semantic Web Text Mining This is an open-access article under the CCAeBY-SA license 1 Introduction The amount of Kurdish text is increasing online, and as any other language, it is gaining a high demand for making this web content readable and processable by machines.
However, in terms of computing there is a need for the Kurdish language to be understood and produced by technology as it is the case for other non-Kurdish text on the web .
The future of NLP in Kurdish language development may be explored by deepening the research on a diverse range of tasks and advances in the NLP pipeline .
, .
Kurdish language is a member of the Indo-Iranian branch of Indo-European languages which is spoken by more than 30 million people in Western Asia, mainly in Iraq.
Turkey.
Iran, and Syria .
, .
The Kurdish language has a variety of dialects and owns its own grammatical system and rich lexicon .
Kurdish has traditionally been written in a variety of scripts, as a result the Kurmanji dialect is predominantly written in Latin, whilst Sorani.
Southern Kurdish, and Laki are predominantly written in modified Arabic alphabet .
, .
This does not only complicate communication between readers and speakers, but also adds the difficulty to the language computing .
, .
The Fig.
1 illustrates the Latin- and Arabicbased Kurdish alphabets with the International Phonetic Alphabet (IPA) that are utilized in all dialects.
26555/jifo.
jifo@uad.
Jurnal Informatika ISSN 2086-8138 .
| 2745-7249 .
Vol.
No.
January 2023, pp.
Fig.
A comparison of the Kurdish alphabets Kurdish is a strongly derivational language, due in part to its abundance of affixes and problems or challenges .
While Sorani lacks gender and grammatical cases, it does contain a complete article marking system for definite, indefinite, and demonstrative nouns in singular and plural forms .
Generally.
Kurdish has a subject-object-verb word order and is a no-subject .
r pro-dro.
language in terms of grammar.
Within dialects and subdialects, the presence of grammatical markers for nominative and oblique cases varies .
Another distinguishing characteristic of the Kurdish language is the morphosyntactic alignment of past-tense transitive verbs.
In these tenses, the intransitive verb's subject behaves like the transitive verb's past patient .
On the other hand the natural language content on the web needs to be readable and process-able by machines .
, .
by the NLP pipeline to help grasp, perceive, and control human language .
, .
NLP involves various tasks, ranging from low-level tasks such as sentence segmentation, to high-level tasks such as semantic annotation and opinion mining.
The Semantic Web is about applying semantics, i.
, context, to data on the Web, to make the web pages easier to process and manage by machines .
, .
This paper surveys the work done in the field of NLP for the Kurdish language.
The rest of this paper is organized as In Section 1, we first briefly introduce the Kurdish language and its two main dialects then underline their differences from a rule-based perspective.
Our methods and procedures for summarizing the articles are discussed in Section 2.
Section 3 describes a KLPT literature review and experimental comparison.
Section 4, provides the KNLP Applications Literature Review.
Kurdish language challenges are discussed in Section 5, the paper's discussion in Section 6, finally, we conclude the paper in Section 7.
2 Method While Kurdish is a language with few online resources and is at the beginning level in the area of NLP, we have attempted to summarize almost all studies that have worked on Kurdish NLP.
There are several dialects of Kurdish, with Sorani and Kurmanji being the most frequently spoken.
Thus, all study has been conducted on these two dialects.
To have a better understanding of the present state of Kurdish NLP and computational linguistics, for this reason, the reviewed articles have been split into two main sections with comprehensive analysis of the reviewed articles' quality and the limitations of the study.
First section present the state of the art for KNLP accordingly for every step in the standard NLP pipeline.
According to this section of the literature review, researchers have used a wide range of methodologies and strategies to address a variety NLP steps, from 2010 Table 1 compares the topics studied works in the literature.
As given in Table, there are different NLP tasks have been handled and methods been utilized such as text cleaning, tokenization.
POS tagging, lemmatization/stemming, and named entity recognition.
In second section we reviewed the publications that provide the software application of the work done in the KNLP.
This section demonstrates different types of algorithms, methods and applications in designing the system for implementing and use of different tasks of KNLP.
As was shown in the literature review, variety research methods and approaches have been employed to solve a wide range of issues.
The following research used Kurdish NLP applications, ranging from .
to the present .
A summary of the Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika Vol.
No.
January 2023, pp.
ISSN 2086-8138 .
| 2745-7249 .
latest applications of Kurdish language NLP studies is shown in Table 2.
As outlined in Table, a series of activities may be addressed using a variety of techniques, including machine learning models.
N-gram models, rule-based machine translation, and classification approaches for both Kurdish dialect Sorani and Kurmanji, for the purpose of evaluating the approach performance, the usage of multiple datasets based on the aim is required.
Overall, the author concentrated on three distinct fields: dialectology, speech recognition, and machine translation.
The development of KNLP appears promising, and interesting results have been presented for a limited number of NLP steps, but it is not without challenges.
As a result, we attempted to identify and classify KNLP's notable limitations and difficulties.
3 KNLP: Literature Review The Kurdish NLP pipeline is a collection of tools and algorithms created expressly to process and examine text written in the Kurdish language.
Various natural language processing operations are included in the pipeline that are important for allowing semantic web and text mining applications.
Some of the NLP tasks that the Kurdish NLP pipeline supports: Tokenization.
Part-of-speech tagging.
Named entity recognition.
Dependency parsing, and Sentiment analysis.
Applications for the semantic web and text mining can all benefit from these NLP tasks.
For instance, structured data from unstructured text can be extracted using named entity recognition and dependency parsing, and this data can subsequently be utilized to fill databases or knowledge graphs.
Customer feedback or social media data can be analyzed for patterns using sentiment analysis.
In conclusion, the Kurdish NLP pipeline offers a number of NLP operations necessary to enable semantic web and text mining applications.
These include dependency parsing, named entity recognition, part-ofspeech tagging, sentiment analysis, tokenization including machine translation and speech In this section we present the state of the art for KNLP accordingly for every step in the standard NLP pipeline as presented in the Fig.
Fig.
NLP Pipeline .
1 Text Cleaning The very early step in NLP and text mining is to exclude any non-semantic parts of the text, such as removing punctuation marks and stop-words.
Ahmadi in .
the two-function preprocessing module that was given to the Sorani and Kurmanji dialects of the Kurdish language normalized encoding tasks by unifying characters so that each grapheme uses only one unique encoding, and also normalized the orthography of the text.
The method unify-numeral is also given to help convert numbers, particularly in Farsi (A).
Eastern Arabic .
and Western Arabic .
The author developed a basic system for all scripts.
however, users will have the option to customize the numbers in the Kurdish regions.
Within the preprocess method, all three of these functions are called upon, with the text normalized, standardized, and unified according to the provided inputs.
Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
ISSN 2086-8138 .
| 2745-7249 .
Jurnal Informatika Vol.
No.
January 2023, pp.
Mustafa et.
given a list of stop words for Kurdish Sorani words that are removed from a manuscript once the stemming process begins.
A preset list can be created to include these terms that are not necessary for information retrieval but are often used in Kurdish writings.
A table lists about 240 stop terms, and for two major reasons the list of stop words is developed: .
Words that match the phrase and the document must be It depends heavily on the words which have extreme significance.
Noise words should thus be deleted.
Documents using phrases such as AuAAyIA,AyboAy meaning to AuwhyAy.
AuAAyaOiA.
AuyweAy meaning AuyouAy and AuAAyIaOA.
AunywAy meaning AuinAy the same request must not provide a relevant understanding.
These sound words are irrelevant and can harm the efficiency of the retrieval, as they do not differentiate between the relevant documents and those which are not relevant.
In addition, in Kurdish Sorani, the richness of stop words increased the characteristic vector size.
Two sources have collected the data: A compilation of information from both television sites, containing 1960 pieces of text information and a total of 43594 words, was compiled by Rudaw.
NRT websites.
The provided technique is aimed at lowering file size by 35% to 50% by eliminating stop words.
In the best of our understanding and study we found that the stop words list was not adequate and that they were not large enough to contain all the stop words in Kurdish.
2 Tokenization and Sentence Splitting The input raw text can be segmented into words or sentences to make it ready for further analysis and processing.
The main two segmentation techniques are word-based tokenization and text splitting for the sentence detection task for sentence-based Ahmadi in .
a lexicon and a morphological analyzer are used to tokenize the Sorani and Kurmanji dialects of Kurdish.
The authors illustrate how tokenization may effectively solve the languageAos morphological complexity and absence of a consistent spelling.
They use the WordPunct tokenizer of NLTK to construct a baseline model.
The suggested four-step approach, which includes text preprocessing, compound word tokenization, word tokenization, and morphological analysis, tokenized the text into a series of alphabetic and non-alphabetic letters using a regular expression.
The authors annotate 100 sentences from the Kurdish Textbooks Corpus (KTC) .
for Sorani and 100 sentences from the Pewan corpus .
to produce a gold-standard dataset to evaluate the effectiveness of the proposed tokenization method.
The outcome indicates that tokenization of individual words had a 44% accuracy, but the accuracy of compound phrases was either 100% or 0% depending on whether a whitespace was inserted between component parts 100% precision is achieved without white space.
The drawback of the study is that compound verb tokenization, along with tense, aspect, person, and mood, verbs are inflected according to the patient, or object of transitive verbs, and may include additional suffixes such as A.
OiA-ew.
to indicate repetition and AA/A=( UAy/=) to indicate emphasis.
3 Part Of Speech Tagging Part-of-Speech (POS) tagging is the process of associating words with their respective parts of speech, for example, noun, verb, and adjective.
POS tagging is a pre-requisite for more advances NLP tasks such as syntactic analysis.
Walther et.
based on three sources of data: lexical information, a nonformalized reference grammar, and raw corpora.
The authors have created a morphological lexicon and a POS tagger for Kurmanji Kurdish (KurLe.
, as the following actions were involved: .
A list of Kurmanji categories is compiled using data from the reference Simultaneously, they formalized Kurmanji morphology using the lexical formalism.
Alexina.
The Kurmanji Kurdish Morphological Lexicon (KurLe.
was created by extracting lemmas from several lexical information sources and inflected them.
A 36tag POS tagset is designed after narrowing the categories list.
Using a variety of simple statistical techniques and heuristics, authors developed different models, then used automatic POS annotation to create a POS-annotated corpus using just lexical information.
Finally, this corpus and KurLex were used to train the Maximum-Entropy Lexiconenriched Tagger (Mel.
PrefHeuristics is the most accurate model, with an Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika Vol.
No.
January 2023, pp.
ISSN 2086-8138 .
| 2745-7249 .
accuracy of 87.
5 percent.
ItAos worth mentioning that the limitation of this study is that it is an insufficient research corpus from which to make reliable conclusions.
Ahmadi et.
analyzed the morphology of the Kurdish language (Sorani dialec.
using computational methods to create finite state transducers (FST.
that were used to generate and interpret words.
The described method has four categories, including verbs, nouns, adjectives, and adverbs, of Kurdish morphology.
The drawback of the study is the absence of syntactical characteristics that may be used to change morphological forms inside a phrase.
4 Lemmatization and Stemming Text normalization techniques like stemming and lemmatization turn different forms of a word into a single token.
This makes the text ready for further word-based processing and Saeed et.
the AuReberAy technique is suggested for Sorani dialects of Kurdish to reduce lengthy affixes and prefixes regardless of their sequence.
The approach covers three steps: Step one employs a Aufor loopAy that iterates three times, with each loop removing one After that, three array lists are employed for this purpose, with each resulting string then being compared within the array list to identify the smallest one.
The second phase involves removing suffixes using one Aufor loopAy made up of four iterations, with each iteration removing one suffix.
This method was developed using Java programming and was used for the KDC-4007 dataset .
dataset with eight classe.
For classification.
Support Vector Machine (SVM) and Decision Tree (DT) are used.
A comparison has been made between the suggested stemmer and the Longest-Match stemmer approach.
The F-measure of the Reber stemmer and the longest-match technique in SVM are greater than DT.
SVM stemmer with Reber were the best in F-measure, but all other stemmers were lower in Longest-Match.
stemmers for classes .
eligion, athletics, and ar.
using Reber exhibited a greater F-measure, whereas for the remaining classes.
F-measure was lower.
Even if the authors have gotten promising results, it will be worth trying to replace standard classification methods like neural networks with more advanced ones like deep neural networks on the same dataset, unfortunately, the F score is not written.
Salavati et.
Jedar, a rule-based stemmer for Sorani Kurdish and Kurmanji Kurdish, was introduced.
The authors also employed a state-of-the-art statistical stemming approach.
Graph-based RAS (GRAS), and applied it to both of the Kurdish languages.
Complete experimental research was then undertaken to assess the efficiency of these Jedar manages nested prefixes by using a recursive technique.
In addition, the authors have developed two strategies to address JeradAos over-stemming problem.
The first strategy used from .
is to avoid excessive stemming by specifying a minimum stem length The second strategy is to make use of the Kurdish languageAos natural suffixing Research results .
suggest that stemming can boost the retrieval of Kurdish papers by up to 35%.
They also say that the benefits of the rule-based and statistical methods are equal.
In the study, there are no suggestions for how to fix some systemic stemming mistakes, such as over-stemming and bad handling of named entities.
Salavati et.
proposed a fundamental language processing tool for the Sorani Kurdish language based on the morphological rules and an n-gram language model called Peyv, to extract lemmas of words.
These rules have been extracted for different parts of speech, such as nouns, verbs, and adjectives.
The authors implemented the Peyv lemmatizer for nouns and verbs, respectively.
In the case of nouns, a pruning method is used to find the On the other hand, a bottom-up method is used for verbs that have more complex structures than nouns.
To evaluate the performance of the proposed tool, the authors analyzed 18M words from 115K news articles from the Pewan text corpus.
The accuracy of the Peyv lemmatizer was 86.
One of the conclusions that can be drawn from this study is that writing mistakes have previously caused researchers to discover incorrect roots for certain words.
Therefore, for developing a more accurate lemmatizer, large number of Kurdish lemmas should be utilized.
Mustafa et.
a Kurdish stemming-step was developed, which is employed for eliminating affixes in the Sorani dialect of the Kurdish language.
This is a step-based Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
ISSN 2086-8138 .
| 2745-7249 .
Jurnal Informatika Vol.
No.
January 2023, pp.
method to explain how words go through stages before arriving at the extracted root.
The Kurdish stemming-step module aims to catch possible roots by stripping prefixes, suffixes, and postfixes from the input word.
The provided word will be verified through all of the Kurdish stemming processes in order to map the string of letters at the beginning or end of the wordAos root.
This technique relied on the creation of sets of possible prefixes and suffixes, which are often used in Kurdish text documents.
The proposed approach does not rely on a dictionary for root word checking.
For example, (A )EiNiIOiIUIAbecomes (AoA AoNiIOAmeans AophaseA.
in step 1, which drops prefixes (A )EiAand then does further mapping through several steps, ending with (A)NiIOiIA.
The strategy uses the suffixes in the step it follows to identify a match, then it eliminates that suffix from the term at step 10.
The prefix (A )IAis discarded as a result.
After that, the final stage sees the final suffix (A )NAmatched and deleted.
This method is also employed in languages other than English, and not only is it used to remove affixes from nouns and verbs, but it also removes affixes from common stop words used in Sorani Kurdish.
A Kurd-specific stemming module was used for the papers gathered, which include a 1960-word text database.
Kurdish stemming-step module F measures are close to 1.
0, with an average recall of 93 percent.
The studyAos primary limitation is that it cannot manage modest under-and over-stemming mistakes and the precision figure should be given too.
doubt high F measure if recall is 93 percent.
ahmadi, in .
developed morphological rules using Kurdish morphemes in the Sorani and Kurmanji dialects, as well as an annotated lexicon with stem and part-of-speech tags.
The proposed module has two classes: stem and spellcheck.
While these two groups of classes are each dedicated to separate objectives, they are nevertheless supplied by the same module.
The Stem class has four primary functions: stem for extracting words from their stems.
lemmatize for lemmatization.
analyze for morphological analysis.
and suffix_suggest for returning all potential suffixes for a given lexeme.
The proposed approach is implemented in Python.
The proposed approach is rule-based, which is in turn based on the used dictionaries and corpus, whereas a more generic statistical model is going to be more efficient, at least 5 Named Entity Recognition Named entities are definite noun phrases that talk about specific people, places, or ideas, like organizations, people, and dates.
Extracting named entities can help a wide range of applications in text mining, information retrieval and ontology engineering.
Hassani in .
a technique has been developed for identifying proper nouns in Sorani and Kurmanji Kurdish texts, by use of rule-based approaches applied to a pair of name dictionaries, a gazetteer, a set of trigrams1 taken from an untagged corpus, and a limited set of hand-crafted rules.
The author proposed a tripartite architecture.
Name lists consist of a gazetteer .
hich includes a subset of proper names as in most existing NER method.
, a dictionary of Kurdish names .
hich includes human names with mostly Kurdish origin.
, and an Arabic name list .
hich includes Arabic names that may be recognized in Kurdish .
A collection of hand-crafted criteria that include phrases that may occur before or after a proper noun, increasing the likelihood of the candidate names being taken as proper For instance, if a candidate name is followed by any member of the subset (AuagaAy.
AuaxaAy.
AubegAy.
AuxatynAy.
AuxanimA.
, then the candidateAos name is deemed to be a discovered .
A collection of methods for the recognition of Kurdish names.
The proposed approach was tested on 15 documents of various sizes.
8 of them were in Kurmanji and 7 were in Sorani.
The precision of the approach they applied was better than 95 percent, and the recall ranged from 40 percent to 80 percent, with the F-measure falling between 60 percent and 80 percent.
Due to a lack of name lists, their recall precision was low.
The studyAos limitations include that the name lists were not comprehensive enough to cover the great A POS string that is also a non-proper-name character.
Kurdish Latin script can signify AuA( AyIEAcema.
and can be translated as AubeautyAy or Auface,Ay depending on the context.
The suggested method included such names in the Kurdish names dictionary, allowing the algorithm to apply rules depending on the surrounding terms .
Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika Vol.
No.
January 2023, pp.
ISSN 2086-8138 .
| 2745-7249 .
majority of Kurdish names, and that geographical and place names were not included in the evaluation procedure.
Littell et.
as part of a pilot research on linguistic rapid response to emergency humanitarian assistance circumstances, it discusses the development of a named-entity recognition (NER) system for the Kurdish and Tajik languages, a dialect of Kurdish.
People, places, and organizations were identified in the text.
The framework is a conditional random field (CRF) based system that uses L1-regularization to disambiguate ambiguous Sorani forms .
lso known as Lasso regressio.
The performance of the proposed technique was evaluated by utilizing the annotated NER data included in the less commonly taught languages (LCTL) language pack .
Their results reveal that when adding features, the methodAos precision is above 74%, recall is 41%-51%, and F-measure is near 51% to 60%.
For inferred morphological analyses, authors used input from human linguists, and as we can see, systems perform badly, while adding features the experimental results are low.
6 Syntactic Analysis (Relation Recognitio.
Syntactic analysis analyzes the sentence grammatically.
Modern parsing methodology called dependency parsing (DP) is commonly utilized.
The basic principle of DP is that each word is related to every other word via a directed connection.
In linguistics, these linkages are termed dependencies.
There are a notable efforts performed in the contemporary parsing community .
To the best of our knowledge and investigated literature, there has been no work done on syntactic parsing of Kurdish text.
The reasons for this can possibly be referred to the challenges mentioned in the section .
7 Word-Sense-Disambiguation Word sense disambiguation (WSD) is the task of finding out which AusenseAy .
of a word is triggered by its use in a certain context, and it appears to happen without people being aware of it .
, such as the work bank (I deposited my money in the bank close to the bank of the city rive.
which means the financial institution in the first mention and riverside in the second one.
The WSD is an issue of assigning words to their correct meanings.
this case, each word is defined by a dictionary and is assigned to its most accurate sense.
The context .
uch as surrounding word.
can be used to classify words .
An overview of the most recent Kurdish language NLP papers.
Table 1.
Approach Kurdish Dialect Application (Fiel.
Text Cleaning Orthographic Sorani Kurmanji .
Text Cleaning Predefined list of stop words Sorani .
Tokenization Lexicon.
Sorani Kurmanji Information Information retrieval, text Information retrieval, text POS Tagging Finite state POS Tagging POS tagging .
Lemmatization and stemming .
Ref NLP Task Dataset Results Accuracy Limitations Pewan Not Mentioned Rule-based approach reduces Lack of Kurdish stop-words.
Inaccurate singleword tokenization Rudaw.
NRT
KTC,
Pewan Sorani Morphologica.
Wergor Not Mentioned Syntactical absences can modify phrase Kurmanji Morphologica.
Raw FPrecision Small sized corpus.
Step-based Sorani Text mining, 1960 pieces of text data Not Mentioned Lemmatization and stemming Loops.
SVM
and DT
Sorani Information retrieval, text
KDC-4007
Not Mentioned Lemmatization and stemming Rule-based, the Sorani Kurmanji Morphologica.
Pewan Statistical inference model to replace the generic rulebased method.
Worthy to attempt deep neural Small sized corpus Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika ISSN 2086-8138 .
| 2745-7249 .
Vol.
No.
January 2023, pp.
Lemmatization and stemming Morphological Sorani Information retrieval, text Pewan Precision Recall F-measure .
Lemmatization and stemming Rule-based stemmer.
GRAS
Sorani Kurmanji Information retrieval, text Pewan Not Mentioned Named Entity Recognition Rule based, hand-crafted Sorani Kurmanji Information retrieval, text Named Entity Recognition CRF.
Lasso
Sorani Information retrieval, text
LCTL
Precision Recall 60% F-measure Precision Recall 69%-74%.
F-measure 51%-60%.
Unable to manage under-and overstemming mistakes.
Over-stemming and not clear handling of named entities.
Not examining the most of Kurdish names and geographical and place names.
Features improve human linguisticsbased inferred According to the literature review above, researchers have used a wide range of methodologies and strategies to address a variety NLP steps, from 2010 to 2020.
Table 1 compares the topics studied works in the literature.
As given in Table, there are different NLP tasks have been handled and methods been utilized such as text cleaning, tokenization.
POS tagging, lemmatization/stemming, and named entity recognition.
All papers focused on two most widely spoken dialects.
Sorani and Kurmanji.
Papers .
, .
used various methods for performing text cleaning and evaluated the effectiveness of those approaches by applying them on diverse datasets.
the output of the approached module in paper .
may be entered by the user into other modules as an input.
The author in .
have used Lexicon and a morphological analyzer to tokenize words and for evaluating the approach the Kurdish textbook corpora (KTC) and Pewan corpora are It is worth mentioning that, the compound words are not addressed efficiently.
whereas the accuracy for compound words is either 100% or 0% depending on whether a whitespace was inserted between component parts.
The authors in .
, .
have used finite state transducers and a list of POS tagging categories for implementing POS taggers in the field of morphological and syntactic analysis using different datasets for evaluating approaches, and both the suggested methods will enable the assessment of future machine transliteration and machine translation systems.
The authors in .
, .
, .
, .
have evaluated several techniques for lemmatization and stemming on diverse datasets utilizing many methodologies.
One of the primary distinctions between Kurdish and other languages is the fact that Kurdish has several prefixes and suffixes following each other.
Some techniques are helpful for various kinds of tasks, including spell checking in text editors.
Based on our investigation of the literature, we determined that Kurdish morphology stemming algorithms that are based on the most frequent affixes in the language will be more successful than those based on the most frequent n-grams in each set.
The authors in .
, .
have worked on named entity recognition tasks for information retrieval and text mining using a variety of methods, including rule-based methods and conditional random fields, while the authors of the paper .
concentrated on noun identification alone, while the authors of the paper .
concentrated on individualsAo names, locations, and organizations in textual data.
The experiment shown that the result obtained in paper .
is superior than that obtained in paper .
because it employs rule-based techniques and concentrates on a single NER category.
4 Applications of KNLP: Literature Review To have a better understanding of the present state of Kurdish NLP and computational linguistics, we reviewed the publications that provide the software application of the work Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika Vol.
No.
January 2023, pp.
ISSN 2086-8138 .
| 2745-7249 .
done in the KNLP.
This section demonstrates different types of algorithms, methods and applications in designing the system for implementing and use of different tasks of KNLP.
Azad et.
developed Kurdish fake news corpus which contains two different sets .
rawled fake news from illegitimate sources and manipulated tex.
The authors applied five machine learning models on the corpus after using Term Frequency-Inverse Document Frequency (TD-IDF) vectorizer as feature selection.
The outcomes indicated that the accuracy of the Support Vector Machine (SVM) for set 1 was 88.
71% and the Random Forest 08% for set 2.
Although this work showed promising results but due to the limitations in a low-resourced language such as Kurdish there are still many unanswered questions and gaps in other categories.
Al-Talabani et.
developed a technique for identifying dialects and languages by using phonetic and stylistic characteristics.
Authors recommended a onedimensional local binary pattern (LBP) for use in the features: Their research shown that the suggested LBP feature set of the dialect and language recognition systems is effective.
The main purpose of this study is to determine how closely related each Kurdish dialect is to its Three Kurdish dialects (Sorani.
Kurmanji, and Hawram.
and three neighboring languages were used in this study (Arabic.
Persian, and Turkis.
The conclusion is that the Sorani and Kurmanji dialects are more closely related to each other than the Hawrami dialect is to either, and the closest language to them is Arabic.
While Persian is the closest language to the Hawrami dialect.
The studyAos primary limitation is that was no statement of the degree of similarity between dialects and another limitation it could be to the type of similarity used for comparison.
Since Kurdish and Arabic belong to very different language family.
Hassani et.
In .
used a supervised machine learning SVM classification approach to detect the dialects of the Kurdish texts.
The research has been done on the Kurmanji and Sorani dialects of Kurdish.
The modified approach was used in multiple steps: data collecting, transliteration, and creation of weighting lists.
The research showed that when a suitable vocabulary list is utilized to train the system, the dialect of the text can be accurately The research also concluded that there is a limited vocabulary that serves an important function in the creation of each dialectAos context.
There are some limitations to the research that were not addressed, including the use of a stemmer while generating the weighted list and during classification.
Kaka-Khan in .
presented a rule-based machine translation system developed to translate simple English sentences to Kurdish based on the Apertuim free open-source engine .
, which is a rule-based translation tool.
The proposed system is considered a good step toward more advanced machine translation tools and technologies.
however, the proposed work can only be used to translate some simple sentences, compound sentences, phrases, and idioms from English to Kurdish.
Mohammed et.
In .
developed a model based on n-grams for categorizing Kurdish Sorani text with many n-gram levels.
The dice method was used to categorize the texts, and performance was measured using recall and accuracy.
Their results indicated that level 5 ngrams surpass the other n-gram levels in categorization.
Despite the promising results of this study, the authorsAo evaluation of the text classification system is limited to a small number of documents, since they have not stemmed the terms in this work.
Hassani in .
Intralingual machine translation was used to translate documents from Kurmanji to Sorani using word-for-word translation .
iteral or direct translatio.
across the The author employed a modified technique in which they developed a word collocation list as proposed by Zhang .
The translated texts were found to be comprehensible in 71% and 79% of cases, for Kurmanji and Sorani, respectively.
They are understood in 29 percent of Kurmanji instances and 21 percent of Sorani cases.
In order to provide a better translation between the two dialects, the author employed bi-dialectal dictionaries instead of a parallel corpus.
Malmasi in .
Classification methods employed in Kurdish texts written in different areas, notably Iran and Iraq, for finding sub-dialect variations in Sorani.
Utilizing surface characteristics n-grams of the character and n-grams of the word.
Their findings indicated that sentences from Iraqi and Iranian news sources may be distinguished with 96% accuracy.
Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika ISSN 2086-8138 .
| 2745-7249 .
Vol.
No.
January 2023, pp.
This work resulted in novel results however it was limited to the Sorani dialect rather than covering other Kurdish dialects as Kurmanji.
Hassani et.
developed a Kurdish Text-to-Speech software based on Concatenative Synthesis method.
The Concatenative method uses diphone units to have a better transition between the phonemes and to have a more natural speech.
The system has been tested by a group of volunteer listeners, and the result of the test showed that the produced speech has a good score of intelligibility, however the work on the reverse process of Kurdish Speech-to-Text has to be addressed by the KNLP research community due to its wide range of applications in high-tech applications as in Internet of Things.
Hamarashid et.
proposed a new word recommendation method for Sorani and Kurmanji Kurdish languages.
The Stupid-Back Off method was implemented in the suggested system, and the N-gram model was employed.
The proposed system includes the following steps: data collecting, reading the Kurdish text corpus in R-studio, encoding the text corpus, data cleaning, producing N-grams, saving the N-grams, changing characters or letters, reading the stored N-grams, and predicting the next words.
The accuracy of the proposed model was high as 96.
3% on the used datasets.
This work can be more valuable if testing it on a variety of texts in the terms of domains of knowledge such as texts from sport, economics, technology and medicine.
Qader et.
the Basic Dataset for Sorani Kurdish Automatic Speech Recognition (BD-4SK-ASR) is used to create an automated speech recognition system for Sorani Kurdish.
The projectAos aim was to build a system capable of automatically recognizing basic phrases based on the language used in grades one through three of elementary schools in IraqAos Kurdistan Region.
The authors utilized CMUSphinx to train the system and created a (BD4SK-ASR) dataset.
In addition to the studyAos shortcomings, a lot more work is required to create Kurdish ASR.
There are many areas of study that require investigation, and a lot of resources must be acquired and utilized.
One difficulty in expanding the use of the Kurdish ASR, or language resources, is a need for a bigger corpus of data, along with study on the languages of other Kurdish dialects and more settings in which ASR may be applied.
Table 2.
An overview of the most recent applications of KNLP.
Results Accuracy SVM
88,71%
Random Forest Ref Objective Technique.
Kurdish Dialect Application (Fiel.
Dataset Tools .
Fake news Machine models and using TD-IDF Sorani Dialectology KurdFake Next word.
N-gram
StupidBackOff
Sorani Kurmanji Dialectology websites and PDF books Rprogram
R-studio
Untested on different corpus
Recognize CMUSphinx
Sorani Speech
BD-4SKASR
Java Not Mentioned Relatively a small sized Apertium Not Mentioned Translating simple phrases Not Mentioned Dialects were not compared.
71% for Kurmanji 79% for Sorani Used bidialectal instead of a parallel corpus.
Translation Rule-based Machine Sorani Machine Simple Complex Idioms and Phrases Dialect Phonetic and a style-based Sorani.
Kurmanj Dialectology Kurdish TV Transliterate Word bi-dialectal Sorani Kurmanj Machine Kurdish media and Limitations Detected and answered a few Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika Vol.
No.
January 2023, pp.
ISSN 2086-8138 .
| 2745-7249 .
Detecting sub Classification methods and N-gram Sorani Identify Supervised (SVM) Sorani Kurmanji Text N-gram Sorani Dialectology Text-toSpeech Concatenative Synthesis Sorani Speech Dialectology Kurdish Dialectology Several Kurdish Government and Kurdish Group of Java Java Not Mentioned Java Not Mentioned Java Not Mentioned Lack of Kurmanji Kurdish dialects.
Better used stemmer for weighted list Use of few documents as they did not match the terms.
Speech-to-Text is not addressed As was shown in the literature review above, variety research methods and approaches have been employed to solve a wide range of issues.
The following research used Kurdish NLP applications, ranging from .
to the present .
A summary of the latest applications of Kurdish language NLP studies is shown in Table 2.
As outlined in Table, a series of activities may be addressed using a variety of techniques, including machine learning models.
N-gram models, rule-based machine translation, and classification approaches for both Kurdish dialect Sorani and Kurmanji, for the purpose of evaluating the approach performance, the usage of multiple datasets based on the aim is required.
Overall, the author concentrated on three distinct fields: dialectology, speech recognition, and machine translation.
The authors in .
, .
, .
, .
, .
have utilized an N-gram model for dialectology and a verity dataset, and they have several shortcomings, such as: in article .
there are still many open questions and gaps that have not been addressed.
In article .
it is not tested on a variety of sports, economics, technology, and medicine-related books.
There was no mention of the degree of resemblance across dialects in article .
In article .
it covered just a few Sorani dialects as opposed to all Kurdish dialects such as Kurmanj.
few materials were utilized since they did not stem the phrases.
The authors of .
, .
used Rule-based and dictionary-based translation, both of which have a variety of The proposed approach in article .
can only be used to translate a few short lines and also Instead of a parallel corpus, bi-dialectal dictionaries were used in article .
The authors of .
, .
employed concatenative Synthesis and CMUSphinx for speech recognition with many constraints, such as the absence of a Kurdish Speech-to-Text reverse technique and the absence of a large corpus for ASR.
5 Kurdish Language Challenges Developing KNLP seems promising and interesting results have been proposed for a limited number of NLP steps, however it is not free of challenges.
The literature has already listed some notable limitations in the way of KNLP, challenges categorized into five groups.
While the first two groups are concerned with the diversity aspect of the Kurdish language, the third and fourth highlight the processing difficulties and the last one examines the depth of resource-scarcity for Kurdish.
The limitations and challenges in KNLP are .
1 Dialect Diversity The primary difficulty in analyzing Kurdish texts is the dialect variety.
In this study, we focus on the two most prominent Kurdish dialects.
Kurmanji and Sorani, with respect to their number of speakers and level of standardization.
Nearly three-quarters of native Kurdish speakers .
are accounted for by this combination.
Some of the major variations in morphology .
, .
A In regards to gender assignment, the Kurmanji dialect is more restrictive than the Sorani dialect, despite the fact that there are exceptions even in the Kurmanji dialect.
Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika ISSN 2086-8138 .
| 2745-7249 .
Vol.
No.
January 2023, pp.
A Unlike Kurmanji, which has a strong conservatism of case opposition .
blique and absolut.
, the Sorani dialect does not.
There are complete ergative transitive verbs in the past tense of Kurmanji, but Sorani does not have them.
A Passive and causative are generated differently in each dialect as well.
in Sorani, the verb undergoes a morphological change to create them, but in Kurmanji, they are formed by helper verbs.
2 Script Diversity Due to geopolitical considerations .
the dialects of Sorani and Kurmanji are written in two completely different alphabet systems and lack of standardization of Kurdish writing.
The common alphabets are shown in Fig.
3 with the mappings among them.
Arabic based Latin based /a AOA
AUA
ANA
U/W
Y/y E/H Arabic based Latin based AiA
a AA
(RR)
(E)
(X)
(H)
One-to-Two Mappings .
One-to-Zero Mappings Arabic based Latin based Arabic based AA
AIA
a a AA
AoA
AEA
AIA
AIA
AIA
ACA
AOOA
One-to-one Mapping Latin based Fig.
, b, .
: The two standard Kurdish alphabets 3 Per-processing The process of pre-processing in Kurdish is extremely challenging for a variety of reasons, some of which will be mentioned below.
A A variety of Kurdish Unicode keyboards make use of characters that aren't native to Kurdish Unicode .
This is due to the fact that some Kurdish letters share their typography with letters in other Arabic-based scripts, such as Persian and Arabic as shown in Fig.
A Conjunctive AoAAo OAmeaning AuandAy as an individual word.
Kurdish language standards define the conjunctive AoAAo OAas a separate word.
However, some Kurdish publishers and academicians are unaware of this, thus they add conjunctive AoAAo OAto a proceeding word, generally one that ends in a vowel .
, .
A There are two distinct AorAos that are both written and spoken differently.
Many Kurdish scholars make the mistake of writing AoA AoAat the beginning of words when they should be writing AoAAoiA, in Sorani Kurdish, the letter AoA AoAnever comes as the first letter of words .
Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika Vol.
No.
January 2023, pp.
Unicode Value Latin Char.
Arabic Char.
Unicode Value ANA AOOA a ISSN 2086-8138 .
| 2745-7249 .
Unicode Value Arabic Char.
AOA
AOA
AUA
From Latin-based to Arabicbased ANA
AOA
Latin Char.
Unicode Value .
From Arabic-based to Latinbased Unicode Value Arabic Char.
Latin Char.
a AA AiA a Latin Char.
R R
L L
Latin Char.
From Arabic-based to Latin-based (Approx.
Fig.
, b, .
: Simple mappings between Arabic-based and Latin-based Kurdish Alphabets 4 Segmentation and Tokenization The technique of finding the boundaries of text components like sentences, phrases, and words is known as segmentation.
Because short vowels are clear expressed in Kurdish writing systems .
, this procedure is simpler than in Persian and Arabic.
Short vowels have a large role in creating ambiguity in the Arabic language, thus word meaning disambiguation, homograph resolution, and part-of-speech identification become quite challenging .
Despite the addition of short vowels, the Arabic-based Kurdish script has two shortcomings acquired from the Arabic writing system .
A Due to the absence of capitalization in the Arabic script, sentence boundaries and Named Entities are harder to distinguish.
A Space is not a deterministic boundary delimiter or sign .
It can be found in a word or between words, or it may come in the middle of two words.
In Persian .
and Urdu .
, there are several ideas for resolving this issue.
5 Lack of Annotated corpus The Kurdish language is an under-resourced language on the web with just raw text available as a linguistic resource.
Larger-scale and more dependable corpora do not yet exist for Kurdish.
Even more challenging is that there is no gold-standard dataset to assist in processing Kurdish.
The variety of dictionaries accessible for the Kurdish annotated corpus and the lack of big datasets are two shortcomings .
Developing a huge corpus of raw text from data that is readily available on the Internet may be utilized for information retrieval applications, for example: While the data obtained from the Internet does present certain issues, the ambiguity of characters must be resolved in the course of normalizing raw text .
The majority of Kurdish written materials have not been digitized, and the ones that are available are either not accessible or not completely convertible.
The lack of a uniform orthography and the inability to use Unicode keyboards contribute to a lack of standardization in text processing in Kurdish .
6 Discussion After presenting the challenges related to the nature of the Kurdish language, this section presents the challenges and drawbacks related to the proposed approaches and used techniques along with the challenges that appear when applying KNLP into different types of applications.
As far as tokenization, it is quite challenging to detect compound verbs .
, besides tense, aspect, person, and mood, verbs are also inflected according to the patient.
Also, the proposed approaches in this regard are mostly rule-based and relying on Punkt Sentence Tokenizer.
For the POS tagging step of KNLP, despite promising results the proposed literature either used a relatively small-size corpus for training the proposed models .
or did ignore the syntactical features which may modify morphological forms in a sentence.
Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
ISSN 2086-8138 .
| 2745-7249 .
Jurnal Informatika Vol.
No.
January 2023, pp.
The literature on lemmatization and stemming needs a trial of advanced ones such as deep neural networks will be worth attempting for the same used dataset instead of use of traditional classification and rule-based algorithms with more .
, .
, .
Also the lack of solving the problem with some of the systematic stemming errors .
, over-stemming and mishandling of named entitie.
, .
and the lack of a significant number of Kurdish lemmas will be required .
To address the NER recognition problem, the suggested experiments employed condensed name and feature lists that excluded the great majority of Kurdish names, yielding less than encouraging results .
As far as applying KNLP in real-world applications, the literature used KNLP tools with classical text mining techniques such as TF-IDF.
LBP and n-gram rather than more advanced ones .
, .
, .
or not fully using KNLP phases .
Also the applications have a narrow output such as only translating simple sentences, compound sentences, phrases, and idioms from English to Kurdish .
, .
, .
or use of a small-size input corpus .
, .
, .
According to our survey.
Sorani Kurdish POS-tagging will be the future direction for Kurdish natural language processing community.
While some progress has been made in KNLP's POS tagging phase, it has only been achieved with a limited number papers despite promising results in .
, .
The proposed literature for the Sorani dialect either used a rule-based approach with only four tag categories as a base .
here are only four categories:
verbs, nouns, adjectives and adverb.
or ignored the syntactical features that may modify morphological forms in a sentence .
The suggested models for the Kurmanji dialect were trained using a corpus of relatively limited size for the Kurmanji dialect .
7 Conclusion The increase in the volume of machine-readable Kurdish web content has notably caused a rising need for the existence of the Kurdish Natural Language Processing (KNLP) pipeline.
In the field of artificial intelligence.
NLP has helped in developing intelligent applications which are capable of both human languages understanding and producing, to a good degree, by incorporating multiple fields such as computer science and computational linguistics.
the other hand, the new field of Kurdish NLP (KNLP) is presenting noteworthy projects, but they are not enough to support industry level semantic web, text mining and information retrieval applications for Kurdish language text.
This review paper addressed the KNLP work in the literature and its applications and listed the state-of-the-art progress in the field of KNLP and its applications beside the language-related challenges, and following is the A The KNLP is in its middle stage of development, so researchers can focus now on its NLP pipeline from the step of POS tagging and on.
A As far as applications of KNLP are concerned, since the KNLP pipeline is immature, its applications have also not yet enabled commercial-level text mining and semantic web A From a linguistics perspective, the Kurdish language has its own challenges when it comes to developing machine models for understanding and producing it through an NLP Such challenges are dialect diversity, script diversity, and a lack of rich annotated However, these challenges can be overcome by the development of a standard Kurdish language and dialect for the Kurdish language content on the web, for example.
Also populating the awareness of research on developing a fully KNLP pipeline toward enabling efficient text mining and semantic web applications for the Kurdish content on the web.
Karwan Jacksi et al.
(Towards a Complete Kurdish NLP Pipeline: Challenges and Opportunitie.
Jurnal Informatika Vol.
No.
January 2023, pp.
ISSN 2086-8138 .
| 2745-7249 .
References