---
title: "TextPrep"
subtitle: "Comparing Tools and Workflows for Data Quality in Basic Text Preprocessing with R"
author:
  - name: "Yannik Peters"
  - name: "Kunjan Shah"
format: html
link-external-newwindow: true
image: "image/icon_textprep.png"
editor: visual
bibliography: references.bib
---

# **At a Glance**

In this tutorial you will learn:

1.  **Enhancing data quality through preprocessing:** The tutorial provides a practical guide on how preprocessing methods, such as automated translation, minor text operations and stopword removal, can significantly improve the quality of social media data depended on use case, data types and methods.
2.  **Comparison of tools, packages and strategies:** By systematically evaluating and comparing different approaches (e.g. different stopword lists), it is highlighted how they can alter textual content and impact data interpretation and quality.
3.  **Creation and analysis of preprocessing levels:** Defining four preprocessing levels offers a structured framework to analyze text data at varying degrees of preparation, helping to understand how preprocessing affects analytical outcomes.
4.  **Applying certain metrics to assess data quality:** Text similarity measures such as word count or cosine similarity are used to document differences between the various preprocessing strategies and packages. Also Structural Topic Modeling is used to compare different preprocessing stages using semantic coherence and exclusivity.

# 1. Introduction

The digitalisation has led to innovations of research objects and research methods. While statistical methods for analyzing numerical data have a long tradition, it is especially the automated analysis of text data that has seen a significant advancements in recent years. Automated text analysis methods are applied to various data sources, including social media data, news paper articles, parliamentary speeches, historical texts or literature. In this introduction, we want to focus on an important, necessary and often times challenging aspect related to data quality in text data: text preprocessing. Text preprocessing can be defined as all changes made to the text data after the data collection and before the data analysis. Its main purpose is to bring the raw data in a form that is then suitable for applying specific research methods, but also to reduce the likelihood of errors. In this sense, text processing is closely linked to the measurement dimension of data quality. On the one hand, text processing can help to reduce measurement errors, by increasing consistency or accuracy. On the other hand, text processing itself can become a source for potential errors. In the TED-On, the "Total Error Framework for Digital Traces of Human Behavior On Online Platforms" [@sen2021] these errors are referred to as `trace reduction errors`. According to Sen et al. an example for this error would be: "Researchers might decide to use a classifier that removes spam, but has a high false positive rate, inadvertently removing non-spam tweets. They might likewise discard tweets without textual content, thereby ignoring relevant statements made through hyper-links or embedded pictures/videos" (p. 413).

In this tutorial we aim to explore different levels of text preprocessing and compare as well as recommend various packages, tools and functions to use. @churchill2021 distinguish between four levels of text preprocessing.

| Preprocessing level | Preprocessing operations |
|------------------------------------|------------------------------------|
| Elementary pattern-based preprocessing | e.g. removal of punctuaction, special characters, symbols, numbers etc. |
| Dictionary-based preprocessing | e.g. removal of stopwords |
| Natural language preprocessing | e.g. stemming, lemmatization |
| Statistical preprocessing | e.g. removal of tokens with high and low frequency |

We will apply operations from the first three preprocessing steps and evaluate how different choices for these steps impact our data. In this respect, data quality is not only a fixed condition but also a set of practices: "doing data quality". In this tutorial, we emphasize the practice of comparison, i.e. determining which options of preprocessing influence the data (quality) and how. The types and sequence of text preprocessing steps depend on the nature of the data being analyzed and the methods applied. For instance, social media data tends to be noisier compared to more structured sources like newspaper articles or scientific papers. Additionally, the chosen analytical method significantly influences preprocessing requirements. Methods like LDA demand extensive preprocessing, while others, such as BERTopic, function effectively with minimal or no text modifications. In the final step of this tutorial, we will use differently preprocessed text data to apply Structural Topic Modeling (STM) [@roberts2019] and compare differences in the analysis. For this purpose, we created a small social media data set with posts about the Olympic summer games in Paris 2024, supposedly "collected" from the #Olympics2024. For copyright reasons, we have constructed an artificial data set that does not contain any real content. The Olympic summer games can be considered a transnational media event [@hepp2015], which today is, of course, not only covered by traditional media but is communicatively accompanied on social media.

# 2. Set up

First, we will load all relevant libraries. Please ensure to have all relevant packages installed using the `install.packages()` command. We will use and also compare functions from some of the most important text preprocessing and analysis R packages such as `quanteda`, `stringr`, `textclean` or `textTinyR`. Additionally, we will incorporate specific packages like `skimr`, `polyglotr` or `deeplr` for special purposes.

```{r message = FALSE, warning = FALSE}
library(tidyverse)
library(quanteda)
library(textclean)
library(textTinyR)
library(tidytext)
library(skimr)
library(polyglotr)
library(deeplr)
library(stopwords)
library(stm)
```

Finally, we will then load our artificial dataset from the Olympic summer games.

```{r message = FALSE, warning = FALSE}
url <- "https://raw.githubusercontent.com/YannikPeters/DQ_Tool_TextPreprocessing/main/data/Olympics_Summer_1_2024.csv"
olympics_data <- readr::read_csv(url, locale = locale(encoding = "Latin1"))
olympics_data
```

# 3. Application of tools and use case

### 3.1 Basic data (quality) checks

Let's start with checking out the basic structure of our data set.

```{r message = FALSE, warning = FALSE}
str(olympics_data)
```

Here we find 13 variables and 150 observations.

```{r message = FALSE, warning=FALSE}
summary(olympics_data)
```

When running the `summary()` function, the overview might become difficult to interpret depending on the size of the dataframe. Therefore, we recommend R tools that provide basic data quality reports and present the results in a clearer format, such as the `skimr` package. With skimr, you can obtain an overview of the various variables of our data set, including descriptive statistics and missing values.

```{r warning=FALSE}
skimr::skim(olympics_data)
```

### 3.2 Handling multilingual text data

As the Olympic games are a transnational media event, it is not surprising to receive a multilingual data set. The primary challgenge with multilingual data sets is that common CSS research methods, such as topic modeling or sentiment analysis typically expect text data to be in a single language.

::: callout-note
If you use this tutorial on your own data set and it is not a multilingual one, you can skip this part and continue with 3.3.
:::

There are various strategies to handle multilingual data sets. The chosen strategy should depend on the specific context, the methods applied and the research design and questions. We can distinguish three or four main strategies for working with multilingual data sets (see @haukelicht2023, @lind2021a). For this short version, we will discuss three primary strategies: 1) selecting cases, 2) working with multiple language data sets and 3) using machine translation.

::: callout-important
An alternative, fourth way would be to apply methods which are able to analyze multilingual text data. These methods are usually based on multilingual word or sentence embedding. Examples for these strategies can be found in @licht2023a or @chan2020.
:::

1\) Selecting cases: a single language

This approach involves selecting cases that contain documents in only one language. For our Twitter/X data set, we could for instance remove all postings that are not in English.

```{r warning=FALSE, message=FALSE}
olympics_data_en <- olympics_data %>% dplyr::filter(language == "en")
table(olympics_data_en$language)
```

Of course, this strategy might lead to a representation error as specific content is systematically excluded from analysis (in our case twenty tweets). Therefore, let's explore the other strategies.

2\) Multiple single language data sets

Another way of dealing with multilingual data sets is to create language-specific subsamples of our data. The main advantage of this strategy is, that no content is lost due to exclusion or translation errors. However, compared to the other methods there are more validation steps required for each single language data set (for detailed information see @haukelicht2023, @lind2021a). As we have already created a data set which only contains English tweets, we will create two additional dataframes for German and French tweets.

```{r warning=FALSE, message=FALSE}
olympics_data_de <- olympics_data %>% dplyr::filter(language == "de")
table(olympics_data_de$language)

olympics_data_fr <- olympics_data %>% dplyr::filter(language == "fr")
table(olympics_data_fr$language)
```

We only find a few documents that are not in English resulting in very small language subsets. Therefore, this strategy might not be the best for our example data.

3\) (Machine) Translation

The third option of dealing with multilingual datasets is to translate the non-English speaking tweets into English. Since this is a just a small, artificial, sample data set, we could actually translate the few tweets manually. In a real case scenario however, analyzing a data set of millions of tweets, you would typically use an automated translation algorithm or method. The main advantage of the translation method is to generate one singular data set, which can then be analyzed using one model. This approach also requires fewer resources. The main disadvantage lies in the potential for translation errors. It is therefore necessary to evaluate the translation method used. For this purpose, let's translate all non-English tweets with both tools in order to compare the results. The most common translation tools are `Google Translator` and `DeepL`. First, we will use the `polyglotr` and `deeplr` packages to translate the German text.

```{r message=FALSE, warning=FALSE}
#Translation of German posts and creation of translated dataframe using Google Translate
translation_google_de <- polyglotr::google_translate(olympics_data_de$tweet_text, target_language = "en", source_language = "de")
translation_google_de <- sapply(translation_google_de, function(x) x[[1]])
olympics_data_de_google <- olympics_data_de
olympics_data_de_google$tweet_text <- translation_google_de
```

To access the DeepL API, you need a developer account. You can use this [link](https://www.deepl.com/en/pro/change-plan#developer) to reach the registration page. A free account allows you to translate up to 500,000 characters per month and provides access to the DeepL Rest API. To translate text data using DeepL in R, you first need the API-key. When signing up for a developer account, you will automatically receive this key.

```{r message=FALSE, warning=FALSE, echo=FALSE}
my_key <- "37cbfea5-12e9-4062-bdd4-04bdca6aef75:fx"
```

```{r warning=FALSE, message=FALSE, results='hide'}
#Translation of German posts and creation of translated dataframe using DeepL
translation_deepl_de <- deeplr::translate2(olympics_data_de$tweet_text, target_lang = "EN", auth_key = my_key)
olympics_data_de_deepl <- olympics_data_de
olympics_data_de_deepl$tweet_text <- translation_deepl_de
```

::: callout-note
For this code to work, make sure that you create a my_key object containing your API key.

my_key \<- "Your key"
:::

Let's compare the results for the German tweets.

```{r message=FALSE, warning=FALSE}
head(olympics_data_de_google$tweet_text)
head(olympics_data_de_deepl$tweet_text)
```

A quick comparison shows that the translations seem pretty similar. We can also use certain metrics to determine the degree of similarity. For this example, we will apply [cosine similarity](https://towardsdatascience.com/overview-of-text-similarity-metrics-3397c4601f50). First, we need to `unlist` our text data, as the `COS_TEXT` function from the `textTinyR` package requires a vector as an input.

```{r message=FALSE, warning=FALSE}
google_translation_de <- unlist(translation_google_de)
deepl_translation_de <- unlist(translation_deepl_de)


cosine_similarities_de <- textTinyR::COS_TEXT(google_translation_de, deepl_translation_de, separator = " ")

cosine_similarities_de
mean(cosine_similarities_de)
```

With a mean cosine similarity of 0.90 on a scale from 0 to 1, the translations from Google Translate and DeepL are indeed very similar. We can apply the same process for the French postings.

```{r message=FALSE, warning=FALSE}
#Translation of French posts and creation of translated dataframe using Google Translate
translation_google_fr <- polyglotr::google_translate(olympics_data_fr$tweet_text, target_language = "en", source_language = "fr")
translation_google_fr <- sapply(translation_google_fr, function(x) x[[1]])
olympics_data_fr_google <- olympics_data_fr
olympics_data_fr_google$tweet_text <- translation_google_fr

#Translation of French posts and creation of translated dataframe using DeepL
translation_deepl_fr <- deeplr::translate2(olympics_data_fr$tweet_text, target_lang = "EN", auth_key = my_key)
olympics_data_fr_deepl <- olympics_data_fr
olympics_data_fr_deepl$tweet_text <- translation_deepl_fr

#compare Google Translate und Deepl translation manually
olympics_data_fr_google$tweet_text
olympics_data_fr_deepl$tweet_text

#unlist french translation data
google_translation_fr <- unlist(translation_google_fr)
deepl_translation_fr <- unlist(translation_deepl_fr)

#calculate cosine similarities for French translation data
cosine_similarities_fr <- textTinyR::COS_TEXT(google_translation_fr, deepl_translation_fr, separator = " ")

cosine_similarities_fr
mean(cosine_similarities_fr)
```

With an average cosine similarity of 0.85, the French translation is slightly less similar than the German one.

If we combine the translated data sets with the original English language data set and then compare the cosine similarity, it is of course higher with 0,98. So in this case, the difference of our combined data set is minimal. This would, of course, be different if the amount of non-English content were higher.

```{r message=FALSE, warning=FALSE}
olympics_data_en_deepl_full <- rbind(olympics_data_en, olympics_data_de_deepl, olympics_data_fr_deepl)
olympics_data_en_google_full <- rbind(olympics_data_en, olympics_data_de_google, olympics_data_fr_google)

cosine_similarities_full <- textTinyR::COS_TEXT(olympics_data_en_deepl_full$tweet_text, olympics_data_en_google_full$tweet_text, separator = " ")
mean(cosine_similarities_full)
```

The key question for us is now: Should we use the translation of DeepL or Google Translate? Generally, DeepL is considered to be more accurate than Google Translate (e.g. [here](https://www.geeksforgeeks.org/deepl-vs-google/)). Still, Google Translate has proven to be highly accurate for calculating topic models [@devries2018]. In current research, both tools are considered suitable. For the translation of Spanish idiomatic expressions into English, @hidalgo-ternero2021 found DeepL to perform slightly better, with an average accuracy rate of 89%, than Google Translate at 86%. @sebo2024 did not find significant differences in the accuracy of the two tools. One of the major advantages of Google Translate is that it can be applied to significantly more languages than DeepL. As we had to translate only two languages in our example data, we will use DeepL for this example.

```{r message=FALSE, warning=FALSE}
url_2 <- "https://raw.githubusercontent.com/YannikPeters/DQ_Tool_TextPreprocessing/main/data/olympics_data_en_deepl_full.csv"
olympics_data_en_full <- readr::read_csv(url_2, locale = locale(encoding = "Latin1"))
olympics_data_en_full

```

::: callout-important
The translations of DeepL and Google Translate may differ slightly with each new API request due to coincidence or updates to the model. Especially when using Bag of Word models, this can lead to deviations depending on the proportion of translated text. Recent literature also highlighted the importance of using open source models for extrinsic data quality values like reproducibility [@chan2020], especially since they perform only slightly less accurate compared to the commercial ones [@licht2024a]. In R, the authors have currently not found any convincing open source alternative integrated in packages. Open source models like OpusMT can be used in Python via [Hugging Face](https://huggingface.co/models?pipeline_tag=translation&sort=trending). In R, these models can be accessed via the [`reticulate`](https://rstudio.github.io/reticulate/) package, which offers a connection to Python applications (see [here](https://rpubs.com/eR_ic/transfoRmers)). In future, a another option might be the [`text`](https://rpubs.com/eR_ic/transfoRmers) package, in which the translation function still has an experimental status.
:::

### 3.3 Minor text operations

Minor text operations and removing stopwords in text preprocessing are highly dependent on two factors:

a\) The text data type

Compared to other types of text data, such as newspaper articles or literature, social media data often features a rather informal communication style and specific characteristics. For instance, the proportion of abbreviations, slang, spelling mistakes, or emojis is usually high, which can be considered as noise (not errors) in the data.

b\) The specific method to be applied

Different methods in Computational Social Science require varying strategies of text preprocessing. Methods that use a bag of words approach (BoW), for example, tend to remove a rather high number of different special characters, as these are not regarded as meaningful for interpretation, but rather as disruptive. In contrast, approaches that incorporate context and semantics, such as modern transformer-based models tend to retain characteristics like punctuation marks and generally require less preprocessing steps.

In our case, we will later apply STM, which is based on a bag of word approach. Therefore, we will apply some operations like removing hashtags, punctuaction or URLs. Before we do so, let's check the length of our documents.

```{r message=FALSE, warning=FALSE}
sum(stringr::str_count(olympics_data_en_full$tweet_text, '\\w+'))
```

First, we will remove all hashtags from the original text and save them in a separate column. After compareing functions from multiple packages, we decided to use the one from `textclean`, because the ones from the other packages performed slightly less accurate (e.g. some were unable to remove punctuation at the end of an hashtag within a sentence). Removing hashtags from the text is advisable in our case, as we are analyzing tweets that all contain #Olympics2024. In a topic model, #Olympics2024 would likely be closely associated to every topic and would not add significant value to the interpretation.

```{r message=FALSE, warning=FALSE}
olympics_data_en_full_rem <- olympics_data_en_full %>%
 dplyr::mutate(
        	# Extract hashtags
              	Hashtags = sapply(str_extract_all(`tweet_text`, "#\\w+"), paste, collapse = " "),

              	# Remove hashtags from the original text
            `tweet_text` = textclean::replace_hash(`tweet_text`, replacement = "")
      	) %>%
  	# Clean up any extra whitespace left after removing hashtags
      mutate(`tweet_text` = stringr::str_squish(`tweet_text`))

sum(stringr::str_count(olympics_data_en_full_rem$tweet_text, '\\w+'))
```

Removing the hashtags has reduced our total word count by more than 400 words. In addition to the hashtags, we also want to remove special characters such as URLs, punctuation or usernames as they do not add relevant information in BoW models like LDA or STM. Instead, they rather increase the number of words (tokens) to analyze and therefore the calculation time. Again, we will store the \@-mentions in a separate column to preserve information. Finally, as a last step of minor text operations, we will convert the entire text to lowercase in order to ensure the same word is not treated differently due to capitalization.

```{r message=FALSE, warning=FALSE}
# Create a new column for usernames and then clean the text
olympics_data_en_full_rem <- olympics_data_en_full_rem %>%
  dplyr::mutate(
    #Store the usernames in a new column
    user_mentions = stringr::str_extract_all(tweet_text, "@\\w+") %>%
      sapply(paste, collapse = ", "),  # Extract usernames with '@'

    # Clean the text
    tweet_text = gsub("(RT|via)((?:\\b\\W*@\\w+)+)", "", tweet_text) %>% #Remove retweets
      gsub("https?://\\S+", "", .) %>%  # Remove URLs
      gsub("@\\w+", "", .) %>%          # Remove @usernames from the text
      gsub("[\r\n]", " ", .) %>%        # Remove line breaks
      gsub("[[:punct:]]+", " ", .) %>%  # Remove punctuation
      gsub("\\s+", " ", .) %>%          # Reduce multiple spaces to a single space
      trimws(.) %>%                     # Trim whitespace from the beginning and end
      tolower()                         # Convert text to lowercase
  )

head(olympics_data_en_full_rem$tweet_text)

sum(stringr::str_count(olympics_data_en_full_rem$tweet_text, '\\w+'))

```

### 3.4 Removing stopwords

After doing the minor steps of text preprocessing, we now want to focus on removing stopwords as they can highly impact the outcome of certain models. Stopwords are commonly understood to as frequently used words that add only little or no meaning for interpretation. Many popular text mining packages in R like `quanteda` offer predefined stopword lists. These lists are often applied by default, without considering their specifics and peculiarities. For example, [@nothman2018]identified numerous "surprising omissions (e.g. *hasn't* but no *hadn't*) and inclusions (e.g. *computer*)". Also @hvitfeldt2021g found out inconsistencies in specific stopword lists. For example, the SMART stopword list includes "he's" but not "she's". Therefore, it is essential to evaluate the impact of stopword lists in relation to the specific text data. In this tutorial, we will compare three commonly used and general stopword lists of the `stopwords` package: NLTK, SMART and Stopwords ISO. Altough they overlap and can be integrated, `stopwords` offers a broader selection of different lists compared to general text analysis packages with build-in stopword lists. Let's compare now the three lists following first the procedure of @hvitfeldt2021g.

```{r message=FALSE, warning=FALSE}
length(stopwords(source = "nltk"))
length(stopwords(source = "smart"))
length(stopwords(source = "stopwords-iso"))
```

It becomes clear that the lists differ significantly in length. While NLTK is comparatively short, Stopwords ISO contains more than seven times as many words. Let's now examine which words the three lists would exclude to highlight the differences.

```{r message=FALSE, warning=FALSE}
olympics_data_en_full_nltk <- olympics_data_en_full_rem
olympics_data_en_full_smart <- olympics_data_en_full_rem
olympics_data_en_full_iso <- olympics_data_en_full_rem

#function to extract stopwords
extract_stopwords <- function(text, source) {
     # Get stopwords for the specified source
     stops <- stopwords::stopwords(language = "en", source = source)
     # Split into words
     words <- text %>%
         strsplit("\\s+") %>%
         unlist()
     # Find intersection with stopwords
     found_stops <- intersect(words, stops)
     # Return as string
     paste(found_stops, collapse = ", ")
 }
# Apply extraction for each source
olympics_data_en_full_nltk <- olympics_data_en_full_nltk %>%
     dplyr::mutate(
         nltk_stopwords = sapply(tweet_text, extract_stopwords, source = "nltk"))

olympics_data_en_full_smart <- olympics_data_en_full_smart %>%
     dplyr::mutate(
         smart_stopwords = sapply(tweet_text, extract_stopwords, source = "smart"))

olympics_data_en_full_iso <- olympics_data_en_full_iso %>%
     dplyr::mutate(
         iso_stopwords = sapply(tweet_text, extract_stopwords, source = "stopwords-iso"))

#number of stopwords (unique stopwords per row)
sum(stringr::str_count(olympics_data_en_full_nltk$nltk_stopwords, '\\w+'))
sum(stringr::str_count(olympics_data_en_full_smart$smart_stopwords, '\\w+'))
sum(stringr::str_count(olympics_data_en_full_iso$iso_stopwords, '\\w+'))
```

As expected, for our artificial data set, the Stopwords ISO list would remove the highest number of words from our text. Even though it contains more than twice as many words as the SMART list, the difference between the two lists is only 83 words. The larger gap is observed between SMART and NLTK. This is likely due to the fact that certain shared stopwords are highly represented across many tweets, while some of the unique words of the respective lists occur relatively rarely. Let's now apply the three lists and compare the similarity between the procrossed text columns.

```{r message=FALSE, warning=FALSE}

remove_stopwords <- function(data, text_column, stopword_source) {
  # Get stopwords from the stopwords package
  stops <- stopwords::stopwords(language = "en", source = stopword_source)

  # Remove stopwords from the specified text column
  data[[text_column]] <- sapply(data[[text_column]], function(text) {
    words <- strsplit(text, "\\s+")[[1]]       # Split text into words
    filtered_words <- setdiff(words, stops)     # Remove stopwords
    paste(filtered_words, collapse = ", ")       # Reassemble text without stopwords
  })

  return(data)
}

# Example usage
olympics_data_en_full_nltk <- remove_stopwords(olympics_data_en_full_nltk, "tweet_text", "nltk")
olympics_data_en_full_smart <- remove_stopwords(olympics_data_en_full_smart, "tweet_text", "smart")
olympics_data_en_full_iso <- remove_stopwords(olympics_data_en_full_iso, "tweet_text", "stopwords-iso")

# Calculating cosine similarity
cosine_similarities_nltk_smart <- textTinyR::COS_TEXT(olympics_data_en_full_nltk$tweet_text, olympics_data_en_full_smart$tweet_text, separator = " ")
cosine_similarities_nltk_iso <- textTinyR::COS_TEXT(olympics_data_en_full_nltk$tweet_text, olympics_data_en_full_iso$tweet_text, separator = " ")
cosine_similarities_smart_iso <- textTinyR::COS_TEXT(olympics_data_en_full_smart$tweet_text, olympics_data_en_full_iso$tweet_text, separator = " ")

mean(cosine_similarities_nltk_smart)
mean(cosine_similarities_nltk_iso)
mean(cosine_similarities_smart_iso)

```

As expected, the similarity between NLTK and ISO is the lowest, while it is highest between SMART and ISO. Even though the similarities are relatively high, it becomes evident that using specific stopword lists leads to changes in your corpus and subsequently affects your analysis. Therefore, it is crucial to evaluate which lists work best for our data set, use case and the method to be applied. For this reason, we will identify the words that have been removed exclusively by one specific list.

```{r message=FALSE, warning=FALSE}

# Combine the three stopword variables into one data frame
combined_df <- data.frame(nltk_stopwords = olympics_data_en_full_nltk$nltk_stopwords,
  smart_stopwords = olympics_data_en_full_smart$smart_stopwords,
  iso_stopwords = olympics_data_en_full_iso$iso_stopwords,
  stringsAsFactors = FALSE)

# Function to create a unique word list from a string
get_unique_words <- function(column) {
  unique(unlist(stringr::str_split(column, ",\\s*")))  # Split by commas and remove spaces
}

# Create sets of unique words for each column
words_nltk <- get_unique_words(combined_df$nltk_stopwords)
words_smart <- get_unique_words(combined_df$smart_stopwords)
words_iso <- get_unique_words(combined_df$iso_stopwords)

# Words only in column nltk_stopwords
unique_nltk <- setdiff(words_nltk, union(words_smart, words_iso))

# Words only in column smart_stopwords
unique_smart <- setdiff(words_smart, union(words_nltk, words_iso))

# Words only in column iso_stopwords
unique_iso <- setdiff(words_iso, union(words_nltk, words_smart))

# Display the results
cat("Words only in nltk_stopwords:", unique_nltk, "\n")
cat("Words only in smart_stopwords:", unique_smart, "\n")
cat("Words only in iso_stopwords:", unique_iso, "\n")

```

It appears that only the ISO list removed unique words. Among these words, we find terms that hold interpretative meaning for our #olympics24 data set such as "results", "world", "top", "proud", "home", "show", "test" or "beginning". This indicates that Stopwords ISO is not suitable for our use case. Let's now compare now only NLTK and SMART in order to determine the best option for our analysis. First, we will identify the unique words removed by NLTK but not by SMART, and vice versa.

```{r message=FALSE, warning=FALSE}
get_unique_words_2 <- function(column) {
  unique(unlist(stringr::str_split(column, ",\\s*")))  # Split by commas and remove spaces Leerzeichen
}

# Create sets of unique words for each column
words_nltk <- get_unique_words(combined_df$nltk_stopwords)
words_smart <- get_unique_words(combined_df$smart_stopwords)

# Words only in column nltk_stopwords
unique_nltk <- setdiff(words_nltk, words_smart)

# Words only in column smart_stopwords
unique_smart <- setdiff(words_smart, words_nltk)

# Display the results
cat("Words only in nltk_stopwords:", unique_nltk, "\n")
cat("Words only in smart_stopwords:", unique_smart, "\n")

```

It becomes clear that SMART removes more words without any substantial meaning. However, it still includes some words that could be meaningful for use case, such as first, last, best or together. Additionally, the token "ve" is removed by NLTK but not by SMART. This difference arises because SMART and NLTK use different strategies for handling contradictions.

```{r message=FALSE, warning=FALSE}
setdiff(stopwords::stopwords(source = "nltk"),
        stopwords::stopwords(source = "smart"))
```

NLTK also retains word fragments after removing punctuation like "don", "ll" or "ve". Since we have already removed punctuation, we do need to include these forms. A good strategy could therefore be to construct a customized stopword list based on both lists. This would involve incorparating the relevant, unique NLTK words in SMART while excluding words from SMART thate are meaningful for our subject.

```{r message=FALSE, warning=FALSE}
#safe SMART stopword list as an vector
stopwords_smart <- stopwords::stopwords("en", source = "smart")

#create additional word lists
additional_words <- c("don", "ll", "ve", "ain", "aren", "couldn", "didn", "doesn", "hadn", "hasn", "haven", "isn", "ma", "mightn", "mustn", "needn", "shan", "shouldn", "wasn", "weren", "won", "wouldn")

#create a list of word to be removed from SMART
words_to_remove <- c("against", "best", "better", "first", "fourth", "greetings", "last", "second", "third", "welcome")

#adds additional words
stopwords_smart_adapted <- unique(c(stopwords_smart, additional_words))

#calculates word counts of SMART stopword list and adapted SMART list after adding words from NLTK
length(stopwords_smart)
length(stopwords_smart_adapted)

#removes words from the words_to_remove list
stopwords_smart_adapted <- setdiff(stopwords_smart_adapted, words_to_remove)

#calculates word counts of adapted SMART list after adding and removing specific words
length(stopwords_smart_adapted)

```

Let's now apply our case-specific stopword list to the text.

```{r message=FALSE, warning=FALSE}

olympics_data_en_full_smart_adapted <- olympics_data_en_full_rem

remove_stopwords_2 <- function(data, text_column, new_column_removed = "stopwords_smart_adapted") {
  # Stopwort-Liste (diese muss zuvor erstellt werden)
  stops <- stopwords_smart_adapted

  # new column for removed stopwords
  data[[new_column_removed]] <- NA

  # remove stopwords and save results
  results <- lapply(data[[text_column]], function(text) {
    words <- strsplit(text, "\\s+")[[1]]       # split text in words
    filtered_words <- words[words %in% stops] # found stopwords
    cleaned_words <- setdiff(words, stops)    # text without stopwords
    list(
      cleaned_text = paste(cleaned_words, collapse = " "),  # found stopwords as list
      filtered_words = paste(filtered_words, collapse = " ") # removed stopwords as list
    )
  })

  # save results in column
  data[[text_column]] <- sapply(results, function(res) res$cleaned_text)
  data[[new_column_removed]] <- sapply(results, function(res) res$filtered_words)

  return(data)
}

olympics_data_en_full_smart_adapted <- remove_stopwords_2(olympics_data_en_full_rem, "tweet_text")


sum(stringr::str_count(olympics_data_en_full_nltk$tweet_text, '\\w+'))
sum(stringr::str_count(olympics_data_en_full_smart_adapted$tweet_text, '\\w+'))
sum(stringr::str_count(olympics_data_en_full_smart$tweet_text, '\\w+'))
sum(stringr::str_count(olympics_data_en_full_iso$tweet_text, '\\w+'))


```

As we can see, our adapted SMART list removed slightly fewer stop words than the original version, despite adding more words than we removed.

::: callout-important
In addition to modifying existing stopword lists, it is also possible to create your own stopword lists based on the specific data set [@hvitfeldt2021g]. Words with a very high frequency (and possibly also those with a very low frequency) are often selected for this purpose. The advantage of this strategy is that the stopword list is created from the use case. However, the determination of threshold values and the inclusion of meaningful words also pose challenges here. In addition to creating corpus-specific stopword lists, some researcher also recommended to perform this step not before but after modeling [@schofield2017].
:::

What has already been true for minor text operations also applies to stopwords: the choice of a specific stopword list should depend on the data type, the use case and method applied. For example, @hvitfeldt2021d found with regard to their particular data set and their supervised approach,"the results for all stop word lexicons are worse than removing no stop words at all". It is also not recommended to use these stopword lists for sentiment analyses, as negations (e.g. not, don't etc.) are also removed. For STM however, removing stopwords is crucial to increase the model's interpretability.

### 3.6 Creating a DFM: tokenization and lemmatization

Creating a `document-feature-matrix` (DFM), or more specifically a `document-term-matrix`, is a common way to structure text data before analysis. The matrix consists of documents in rows and words in columns displaying the frequency of each word for each document. In order to do so, we first have to tokenize our text data, breaking it into words as smaller sub unit. Before creating such a matrix of tokenized words, it is advisable to lemmatize the words first. Lemmatization refers to the process of merging inflected words into their root form, known as the lemma. In contrast to stemming, which simply removes common suffixes from words, lemmatization results in normalized words. For lemmatization, we will first use the lemma_en.csv list stored in the Git repository. To use the csv file, you have to save it in our local working environment.

```{r message=FALSE, warning=FALSE}
url_3 <- "https://raw.githubusercontent.com/YannikPeters/DQ_Tool_TextPreprocessing/main/lemma_en.csv"
lemma_en <- readr::read_csv(url_3)

olympics_en_dfm_level3 <- quanteda::corpus(olympics_data_en_full_smart_adapted$tweet_text, docnames = olympics_data_en_full_smart_adapted$no) %>%
  # tokenize and remove numbers and symbols
  quanteda::tokens(.,remove_numbers=TRUE, remove_symbols = TRUE) %>%
  # lemmatize
  quanteda::tokens_replace(lemma_en$inflected_form, lemma_en$lemma,
                 valuetype = "fixed") %>%
  # convert to document-feature-matrix
  quanteda::dfm() %>%
  # remove texts that are empty after pre-processing
  quanteda::dfm_subset(., ntoken(.) > 0)

head(olympics_en_dfm_level3)
```

We have now created a DFM for our preprocessed data set. However, it would be valuable to evaluate how the analyses differ when using different levels of preprocessing. In our pipeline so far, we have used approaches of three categories of text preprocessing as defined by @churchill2021: elementary pattern-based preprocessing (e.g. removal of punctuation), dictionary-based preprocessing (e.g. stopword removal), natural language preprocessing (e.g. lemmatization). We will therefore consider it as level 3 preprocessed data. Let's define three additional preprocessing levels.

| Levels | Preprocessing steps |
|------------------------------------|------------------------------------|
| level 0 | tokenization, automated translation |
| level 1 | tokenization, automated translation and elementary pattern-based preprocessing (removal of punctuation, symbols, hashtags, numbers etc) |
| level 2 | tokenization, automated translation, elementary pattern-based preprocessing and dictionary-based preprocessing (stopword removal) |
| level 3 | tokenization, automated translation, elementary pattern-based preprocessing, dictionary-based preprocessing and natural language preprocessing (lemmatization) |

Level 0 leaves the original text mostly unmodified and only uses tokenization. Level 1 then includes all pattern-based preprocessing, but no dictionary-based approaches. In addition to pattern-based preprocessing, level 2 also removes stopwords, but does not use lemmatization. For all levels, we use the combined data set of English tweets and the translated German and French ones.

::: callout-important
Be aware that we do not apply all possible preprocessing steps, which might improve topic interpretability as we limited the tutorial to rather "classic" preprocessing. Some researchers, however, also used POS tagging to exclude verbs and some excluded named entities to increase topic interpretability [@tolochko2024].
:::

```{r message=FALSE, warning=FALSE}
#creating level 0 dfm
olympics_en_dfm_level0 <- quanteda::corpus(olympics_data_en_full$tweet_text, docnames = olympics_data_en_full$no) %>%
  # tokenize
  quanteda::tokens(.) %>%
  # convert to document-feature-matrix
  quanteda::dfm() %>%
  # remove texts that are empty after pre-processing
  quanteda::dfm_subset(., ntoken(.) > 0)

#creating level 1 dfm
olympics_en_dfm_level1 <- quanteda::corpus(olympics_data_en_full_rem$tweet_text, docnames = olympics_data_en_full$no) %>%
  # tokenize remove numbers and symbols
  quanteda::tokens(.,remove_numbers=TRUE, remove_symbols=TRUE) %>%
  # convert to document-feature-matrix
  quanteda::dfm()%>%
  # remove texts that are empty after pre-processing
  quanteda::dfm_subset(., ntoken(.) > 0)

#creating level 2 dfm
olympics_en_dfm_level2 <- quanteda::corpus(olympics_data_en_full_smart_adapted$tweet_text, docnames = olympics_data_en_full$no) %>%
  # tokenize and remove numbers and symbols
  quanteda::tokens(.,remove_numbers=TRUE, remove_symbols=TRUE) %>%
  # convert to document-feature-matrix
  quanteda::dfm()%>%
  # remove texts that are empty after pre-processing
  quanteda::dfm_subset(., ntoken(.) > 0)
```

Let's now compare the DFMs with regard to their general descriptive statistics.

```{r message=FALSE, warning=FALSE}

dfms <- list(
  "Level 0" = olympics_en_dfm_level0,
  "Level 1" = olympics_en_dfm_level1,
  "Level 2" = olympics_en_dfm_level2,
  "Level 3" = olympics_en_dfm_level3
)

# creating a dataframe
dfm_summary <- data.frame(
  Model = character(),
  Num_Documents = numeric(),
  Unique_Tokens = numeric(),
  Total_Tokens = numeric(),
  stringsAsFactors = FALSE
)

# loop to calculate metrics
for (level in names(dfms)) {
  dfm <- dfms[[level]]

  # calculating dimensions of the DFM
  num_documents <- dim(dfm)[1]
  unique_tokens <- dim(dfm)[2]

  # calculating total number of tokens
  total_tokens <- sum(colSums(dfm))

  # add data to dataframe
  dfm_summary <- rbind(
    dfm_summary,
    data.frame(
      Model = level,
      Num_Documents = num_documents,
      Unique_Tokens = unique_tokens,
      Total_Tokens = total_tokens
    )
  )
}

# print descriptive statistics
print(dfm_summary)
```

While the number of tweets remains unaffected by preprocessing at 150, the vocabulary in terms of unique words is significantly reduced by about 44% from level 0 to level 3. With regard to the absolute number of tokens, it is even 65%. It is noticeable that level 2 and 3 are identical in the total number of tokens. This is because all words are replaced during lemmatization, while only the number of unique tokens is reduced. It also becomes clear that the number of tokens differs from what we counted at the beginning with `str_count`. This is due to the fact that the `tokens`-function from `quanteda` has a different strategy to identify tokens. For example, punctuation marks or numbers are also considered as individual tokens by `tokens` function. Compared to other packages, this approach is well-suited for our purpose of comparing different degrees of preprocessing. For example, the `unnest_token` function of the `tidytext` package automatically removes punctuation when tokenizing. However, since we also want to include a model without elementary pattern-based preprocessing for our comparison, this is ideal. Given that the token function from `quanteda` allows the user to customize the process and is open for fine-tuning (e.g. by using `remove_punct` etc.), it is preferable for our use case.

### 3.7 Topic Modeling

As mentioned earlier, we will now use STM [@roberts2019] as a topic modeling strategy to analyze our text data. STM can be considered an extensions of LDA [@blei2002]. In contrast to LDA, STM allows the correlation of topics as well as the inclusion of meta-variables in the analysis, which makes it particularly popular in the social sciences. Even though there are newer methods like [BertTopic](https://maartengr.github.io/BERTopic/index.html), they usually do not [require](https://maartengr.github.io/BERTopic/getting_started/tips_and_tricks/tips_and_tricks.html#document-length) classic preprocessing steps like stopword removal in advance, but rather after calculating embeddings and clustering documents. At the same time bag of word models still have proven to be effective for determining the effects of text preprocessing on social media data [@churchill2021] [@applest2021]. Since the focus here is on data quality and not the method itself, we will not delve into the regular steps and best practices for its application.

::: callout-note
A more detailed discussion on how to perform topic modeling in the social sciences can be found in @maier2018.
:::

Usually, an important step in applying STM is to find the optimal number of topics (k) using certain metrics and manual evaluation. As we are calculating different models based on different text data, we may not end up with the same number of topics per model. As our analysis, however, is based on an artificial data set, we were able to address this issue in advance. We have already integrated the following 10 topics into the data set.

```{r message=FALSE, warning=FALSE}
prop.table(table(olympics_data$subtopic)) * 100
```

We can, therefore, directly compare the topics modeled in LDA with the topics we implemented. Because of this, we will use k=10 for all models.

```{r warning=FALSE, message=FALSE, results='hide'}

N=10

dfm_stm0 <- quanteda::convert(olympics_en_dfm_level0, to ="stm")
dfm_stm1 <- quanteda::convert(olympics_en_dfm_level1, to ="stm")
dfm_stm2 <- quanteda::convert(olympics_en_dfm_level2, to ="stm")
dfm_stm3 <- quanteda::convert(olympics_en_dfm_level3, to ="stm")

set.seed(1000)
STM_0 <- stm::stm(dfm_stm0$documents, dfm_stm0$vocab, K = N, data = dfm_stm0$meta, max.em.its = 75, init.type = "LDA")

set.seed(1000)
STM_1 <- stm::stm(dfm_stm1$documents, dfm_stm1$vocab, K = N, data = dfm_stm1$meta, max.em.its = 75, init.type = "LDA")

set.seed(1000)
STM_2 <- stm::stm(dfm_stm2$documents, dfm_stm2$vocab, K = N, data = dfm_stm2$meta, max.em.its = 75, init.type = "LDA")

set.seed(1000)
STM_3 <- stm::stm(dfm_stm3$documents, dfm_stm3$vocab, K = N, data = dfm_stm3$meta, max.em.its = 75, init.type = "LDA")

```

::: callout-important
In this tutorial, we will use "LDA" (Gibbs sampler) as an initialization method, even though spectral intitialization is [often preferred](https://github.com/bstewart/stm/issues/133#issuecomment-357765115). Spectral initialization is a deterministic approach, so that random seeds have no impact. However, spectral initialization might lead to slightly [different results](https://github.com/bstewart/stm/blob/dbabf3405c660452bffc8bd1aaf72f7ea3867319/R/stm.R#L82-L101) depending on the machine used. In order to avoid differences in the tutorial, we will use LDA for initialization to ensure that the results are reproducible. For this data set, we find the topics being modeled with LDA initialization to be even closer to our predefined topics then those modeled via spectral initialization. Still, this does not imply that one should avoid spectral initialization as it is particularly recommended when working with data sets containing a large number of documents. For Stewart it is still the "[first thing](https://github.com/bstewart/stm/issues/133#issuecomment-357766361)" to try for a new data set.
:::

Let's inspect the most important terms per topic.

```{r message=FALSE,warning=FALSE}
STM_0_labels <- as.data.frame(t(stm::labelTopics(STM_0, n = 7)$prob))
STM_1_labels <- as.data.frame(t(stm::labelTopics(STM_1, n = 7)$prob))
STM_2_labels <- as.data.frame(t(stm::labelTopics(STM_2, n = 7)$prob))
STM_3_labels <- as.data.frame(t(stm::labelTopics(STM_3, n = 7)$prob))

STM_0_labels
STM_1_labels
STM_2_labels
STM_3_labels
```

As we can see, our topics become more meaningful the more preprocessing steps are applied. Especially the models at level 0 and level 1 are not clear as they mostly consists of stopwords like "the", "of", "is", "to" or "at". Regarding level 2 and 3 we do find overlapping topics of course, but also some differences. A comparison of the topics implemented in advance with the ones in model 3 reveals a number of similarities. For instance, we find topics on doping, gold medalists, ceremonies, basektball and athletic performance. In order to represent topic interpretability numerically, often coders summarize the top terms and assign topic labels. Statistical agreement coefficients such as Krippendorff's alpha [@krippendorff2004] would then be calculated. For the purpose of this tutorial, we won't calculate topic interpretability with human coders, but only compare key internal metrics like [semantic coherence and exclusivity](https://mallet.cs.umass.edu/diagnostics.php). Semantic coherence refers to the degree of which top terms from the same topic occur in the same document. Exclusivity, on the other hand, refers to the degree of how unique the top terms in each topic are compared to other topics. Usually, the relationship between topic coherence and exclusivity is asymmetrical.

```{r message=FALSE, warning=FALSE}
#calculating model statistics
compute_metrics <- function(stm_model, documents, model_name, num_topics = 10) {
  #Semantic Coherence
  semantic_coherence <- stm::semanticCoherence(stm_model, documents)
  mean_coherence <- sum(semantic_coherence) / num_topics

  #Exclusivity
  exclusivity <- stm::exclusivity(stm_model)
  mean_exclusivity <- sum(exclusivity) / num_topics

  #return values as a list
  return(data.frame(
    Model = model_name,
    Mean_Coherence = mean_coherence,
    Mean_Exclusivity = mean_exclusivity
  ))
}

# results for all for STM models
results <- list(
  compute_metrics(STM_0, dfm_stm0$documents, "STM_0", num_topics = 10),
  compute_metrics(STM_1, dfm_stm1$documents, "STM_1", num_topics = 10),
  compute_metrics(STM_2, dfm_stm2$documents, "STM_2", num_topics = 10),
  compute_metrics(STM_3, dfm_stm3$documents, "STM_3", num_topics = 10)
)

# create a dataframe with results
results_df <- do.call(rbind, results)

# show results
print(results_df)

# value range for plot
x_min <- min(results_df$Mean_Coherence) - 3

x_max <- max(results_df$Mean_Coherence) + 3

y_min <- min(results_df$Mean_Exclusivity) - 0.1

y_max <- max(results_df$Mean_Exclusivity) + 0.1


# creating a Scatterplot
ggplot2::ggplot(results_df, aes(x = Mean_Coherence, y = Mean_Exclusivity, color = Model)) +
  geom_point(size = 5, alpha = 0.8) +
  geom_text(aes(label = Model), vjust = -1.2, hjust = 0.5, color = "black") +
  scale_color_viridis_d(option = "plasma", end = 0.9) +
  labs(
    title = "Comparing STM-models",
    x = "Topic Coherence",
    y = "Topic Exclusivity",
    color = "Model"
  ) +
  xlim(x_min, x_max) +
  ylim(y_min, y_max) +
  theme_minimal(base_size = 14) +
  theme(
    legend.position = "right",
    legend.title = element_text(size = 12),
    legend.text = element_text(size = 10)
  )


```

The comparison of the models highlights the impact of our preprocessing steps on these two metrics. For semantic coherence, it is evident that the more preprocessing steps are applied, the less coherent the models become. This is primarily because for model 0 and 1, topics include tokens occuring very frequently in the data set and appear across multiple topics (e.g. stopwords). In contrast, the exclusivity score increases with greater preprocessing as topics are less dominated by highly frequent tokens. We already find some smaller differences between model 0 and 1, highlighting the impact of minor text operations. A great distance occurs between model 1 and model 2, underscoring the substantial effect of stopword removal on these metrics. Even though it is shown for our data set that preprocessing results in lower coherence values, this does not argue against the procedure itself. Rather, it indicates 1) that in text preprocessing a certain level of exclusivity must be reached to model meaningful topics, 2) that topic coherence in the use case of preprocessing does not correspond to topic interpretality and 3) how important the manual interpretation and validation of the topics are. Still, this approach offers a structured strategy for evaluating the effects of preprocessing on topic modeling. Still, especially for smaller data sets, these metrics are sensitive and can vary even with smaller changes in the data, which have to be considered.

# 4. Discussion

The tutorial has emphasized the importance of evaluating preprocessing steps while suggesting systematic comparison at all levels of the research process to identify optimal solution tailored to the use case. It examined effective approaches within the preprocessing steps, but also compared how different levels of preprocessing affect modeling outcomes. These strategies are essential for making informed decisions during the research process. However, applying specific preprocessing steps depends on the use case, the collected data and the methods applied. For our use case - an artificial data set based on #olympics2024 tweets - we conducted several preprocessing steps including: 1) adressing multilingual content, 2) performing minor text operations, 3) removing stopwords and 4) lemmatization. We compared DeepL and GoogleTranslate as well as different custom stopword lists like NLTK, SMART or Stopwords ISO and discussed multiple strategies for assessing the impact of the different strategies on our text data. However, we did not employ data augmentation strategies like POS tagging and named entity recognition. To evaluate the effects of preprocessing, we compared different stages of preprocessing by modeling four separate STMs. Each model was then manually evaluated for topic interpretability. It is important to note that the deviations from the incorporated topics could also be influenced by the small size of the data set, as modeling 10 topics for such limited data is relatively ambitious. We then calculated different internal metrics like semantic coherence and exclusivity to compare the four levels of preprocessing. This analysis allowed us to pinpoint the effects of every text preprocessing step on the STM models. Our finding revealed that semantic coherence decreased with more preprocessing because stopwords, punctuations and symbols co-occurring frequently were removed, while exclusivity increased. Among the preprocessing steps, the removal of stopwords had the most significant impact on these metrics. One way to achieve a model with higher coherence values might involve removing only #Olympics2024 from the tweets and leave the remaining hashtags in the data set. Since hashtags often also represent thematic contextualizations, this might enhance topic interpretability as well.

At the same time, it is noticeable that small deviations can occur when topic models are recalculated. This can be attributed to two primary factors:

1.  Translation Variance: Deviations can arise from the inherent variability and nuances of automatic translation, which may slightly alter the content during preprocessing.

2.  Data Set Characteristics: The small size of the data set - comprising a limited number of documents and a relatively low word count per document—amplifies the impact of minor changes. Even small variations in the text can have a noticeable effect on the resulting topic models.

These factors underscore the importance of considering data set size and preprocessing consistency when analyzing text data, especially in studies using topic modeling techniques. Of course, the results from this small-scale, artificial data set, are not generalizable, as they heavily depend on the relationship between respective data and concrete preprocessing steps employed. @churchill2021 found coherence scores across LDA models to be relatively similar in their comparison, with differences arising more clearly between model types rather than preprocessing steps (without including manual evaluation). Ultimately, it is crucial to evaluate preprocessing steps in a context-sensitive manner, considering their influence on results and their suitability for the data and methods used.

# 5. Literature
