Text classification is the process of assigning tags or categories to text according to its content. It's one of the fundamental tasks in Natural Language Processing (NLP), with a broad range of applications including sentiment analysis, topic labeling, spam detection, and intent detection.
Unstructured data in the form of text is everywhere: web pages, social media, emails, chats, survey responses, support tickets, and more. Text is an extremely rich source of information, but extracting insights from it can be hard and time-consuming due to its unstructured nature. Businesses today are turning to text classification to structure text in a fast, cost-efficient way, enhancing decision-making and automating processes.
In this post, we'll answer some basic questions: What is text classification? How does text classification work? What algorithms are used to classify text? What are the most common business applications?
What Is Text Classification?
Text classification, also known as text categorization or text tagging, is the task of assigning a set of predefined categories to unstructured text. We use text classifiers to organize, structure, and categorize almost everything. News articles can be organized by topic, support tickets by urgency, chat conversations by language, brand mentions by sentiment, and so on.
As an example, take a look at the text below:
"The user interface is quite straightforward and easy to use."
A classifier can take this text as input, analyze its content, and automatically assign relevant tags — such as UI and Easy-to-use — that represent it.
How Does Text Classification Work?
Text classification can be done in two different ways: manual and automatic. In manual classification, a human annotator interprets the content of the text and categorizes it accordingly. This method can provide quality results, but it's time-consuming and expensive. Automatic classification applies machine learning, natural language processing, and other techniques to classify text — a faster, more cost-effective method.
Approaches to automatic text classification fall into three types of systems:
- Rule-based systems
- Machine-learning-based systems
- Hybrid systems
Rule-Based Systems
Rule-based approaches use handcrafted linguistic rules to classify text. These rules help the system use relevant elements of a text to identify categories based on content. Each rule consists of a pattern and a predicted group.
Rule-based systems are human-comprehensible and can be improved over time, but they have disadvantages. They require deep domain knowledge, and generating rules for a complex system can be quite challenging, usually requiring a lot of analysis and testing. Rule-based systems are also difficult to maintain and don't scale well, since adding new rules can affect the results of pre-existing ones.
Machine-Learning-Based Systems
Text classification with machine learning learns to make classifications based on past observations. By using pre-labeled examples as training data, a machine learning algorithm can learn the associations between pieces of text and the output (tags) expected for a given input (text).
In machine-learning-based systems, we start with feature extraction — the process of transforming text into a numerical representation, in the form of a vector. One popular approach is bag-of-words, where a vector represents the frequency of a word in a predefined dictionary of words.
For example, if our dictionary is the word set Computer, science, basketball, marvel, cool, is, not, and, and we want to vectorize the text "Computer science is cool," we'd get the vector representation (1, 1, 0, 0, 1, 1, 0, 0).
The machine learning algorithm is then fed training data — pairs of feature sets (vectors for each text sample) and tags (classification groups) — to produce a classification model.
After training, the model is ready to make predictions. The same feature extractor transforms unseen text into feature sets, which are fed into the classification model to get predictions on tags.
Text classification with machine learning is usually far more accurate than human-crafted rule systems, especially on complex classification tasks. Classifiers built with machine learning are also easier to maintain — you can always tag new examples to teach the model new tasks.
Hybrid Systems
As the name suggests, a hybrid system combines both rule-based and machine-learning-based approaches to further improve results.
In this tutorial, we'll build a text classifier with TF-Hub.
TensorFlow Hub
TensorFlow Hub is a library for the publication, discovery, and consumption of reusable parts of machine learning models. A module is a self-contained piece of a TensorFlow graph, along with its weights and assets, that can be reused across different tasks in a process known as transfer learning. Transfer learning can:
- Train a model with a smaller dataset,
- Improve generalization, and
- Speed up training.
Setup
More detailed information about installing TensorFlow can be found at tensorflow.org/install.
# Install TF-Hub.
!pip install seaborn
from absl import logging
import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import re
import seaborn as sns
Introduction to TensorFlow Datasets and Estimators
Datasets and Estimators are two key TensorFlow features you should use. Combined, they offer an easy way to create TensorFlow models and feed data to them.
We'll try to solve the Large Movie Review Dataset v1.0 task (Mass et al., 2011). The dataset consists of IMDB movie reviews labeled by positivity from 1 to 10. The task is to label the reviews as negative or positive.
# Load all files from a directory in a DataFrame.
def load_directory_data(directory):
data = {}
data["sentence"] = []
data["sentiment"] = []
for file_path in os.listdir(directory):
with tf.io.gfile.GFile(os.path.join(directory, file_path), "r") as f:
data["sentence"].append(f.read())
data["sentiment"].append(re.match("\d+_(\d+)\.txt", file_path).group(1))
return pd.DataFrame.from_dict(data)
# Merge positive and negative examples, add a polarity column and shuffle.
def load_dataset(directory):
pos_df = load_directory_data(os.path.join(directory, "pos"))
neg_df = load_directory_data(os.path.join(directory, "neg"))
pos_df["polarity"] = 1
neg_df["polarity"] = 0
return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)
# Download and process the dataset files.
def download_and_load_datasets(force_download=False):
dataset = tf.keras.utils.get_file(
fname="aclImdb.tar.gz",
origin="http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz",
extract=True)
train_df = load_dataset(os.path.join(os.path.dirname(dataset), "aclImdb", "train"))
test_df = load_dataset(os.path.join(os.path.dirname(dataset), "aclImdb", "test"))
return train_df, test_df
# Reduce logging output.
logging.set_verbosity(logging.ERROR)
train_df, test_df = download_and_load_datasets()
train_df.head()
This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. It provides 25,000 highly polar movie reviews for training and 25,000 for testing, plus additional unlabeled data. Raw text and already-processed bag-of-words formats are provided.
Model
Input Functions and the Estimator Framework
Estimators are a high-level way to create TensorFlow models, with pre-made models for common machine learning tasks — though you can also use them to build your own custom models.
Similar to a tf.keras.Model, an Estimator is a model-level abstraction. The tf.estimator API provides some capabilities currently still under development for tf.keras, including parameter-server-based training and full TFX integration.
You can run Estimator-based models on a local host or a distributed multi-server environment without changing your model, and run them on CPUs, GPUs, or TPUs without recoding.
Estimators provide a safe distributed training loop that controls how and when to load data, handle exceptions, create checkpoint files and recover from failures, and save summaries for TensorBoard.
When writing an application with Estimators, you must separate the data input pipeline from the model. This separation simplifies experimenting with different datasets.
# Training input on the whole training set with no limit on training epochs.
train_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
train_df, train_df["polarity"], num_epochs=None, shuffle=True)
# Prediction on the whole training set.
predict_train_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
train_df, train_df["polarity"], shuffle=False)
# Prediction on the test set.
predict_test_input_fn = tf.compat.v1.estimator.inputs.pandas_input_fn(
test_df, test_df["polarity"], shuffle=False)
Feature Columns
TF-Hub provides a feature column that applies a module to the given text feature and passes along the module's outputs. In this tutorial we'll use the nnlm-en-dim128 module. For our purposes, the important facts are:
- The module takes a batch of sentences in a 1-D tensor of strings as input.
- The module handles preprocessing of sentences (e.g. removal of punctuation and splitting on spaces).
- The module works with any input (e.g.
nnlm-en-dim128hashes words not present in its vocabulary into ~20,000 buckets).
Each tf.feature_column identifies a feature name, its type, and any input pre-processing. hub.text_embedding_column uses TF-Hub to construct a dense representation from a text feature, and can be used on an input feature whose values are strings of arbitrary size. The result has dtype float32 and shape [batch_size, num_features] with a known value of num_features.
embedded_text_feature_column = hub.text_embedding_column(
key="sentence",
module_spec="https://tfhub.dev/google/nnlm-en-dim128/1")
Classifier
For classification, we can use a DNN Classifier. A DNNClassifier, sometimes referred to as a "deep neural network," builds a feedforward multilayer neural network trained on a set of labeled data in order to classify similar, unlabeled data.
estimator = tf.estimator.DNNClassifier(
hidden_units=[500, 100],
feature_columns=[embedded_text_feature_column],
n_classes=2,
optimizer=tf.keras.optimizers.Adagrad(lr=0.003))
This defines two hidden dense layers with 500 and 100 units respectively, plus an input layer (sized by the feature_columns parameter) and the head (binary or multiclass, depending on n_classes).
Training
Now we train the estimator for a reasonable number of steps.
# Training for 5,000 steps means 640,000 training examples with the default
# batch size — roughly equivalent to 25 epochs, since the training dataset
# contains 25,000 examples.
estimator.train(input_fn=train_input_fn, steps=5000)
Prediction
We run predictions for both the training and test sets.
train_eval_result = estimator.evaluate(input_fn=predict_train_input_fn)
test_eval_result = estimator.evaluate(input_fn=predict_test_input_fn)
print("Training set accuracy: {accuracy}".format(**train_eval_result))
print("Test set accuracy: {accuracy}".format(**test_eval_result))
Confusion Matrix
A confusion matrix is a table often used to describe the performance of a classifier on a set of test data for which the true values are known. It allows us to visualize the performance of an algorithm.
def get_predictions(estimator, input_fn):
return [x["class_ids"][0] for x in estimator.predict(input_fn=input_fn)]
LABELS = ["negative", "positive"]
# Create a confusion matrix on training data.
cm = tf.math.confusion_matrix(train_df["polarity"], get_predictions(estimator, predict_train_input_fn))
# Normalize the confusion matrix so that each row sums to 1.
cm = tf.cast(cm, dtype=tf.float32)
cm = cm / tf.math.reduce_sum(cm, axis=1)[:, np.newaxis]
sns.heatmap(cm, annot=True, xticklabels=LABELS, yticklabels=LABELS)
plt.xlabel("Predicted")
plt.ylabel("True")
Scope for Further Improvement
- Regression on sentiment — We used a classifier to assign each example to a polarity class, but we actually have another categorical feature at our disposal: sentiment. Here, classes represent a scale, and the underlying value (positive/negative) could be mapped into a continuous range. We could compute a regression (DNN Regressor) instead of a classification (DNN Classifier).
- Larger module — For this tutorial, we used a small module to restrict memory use. Modules with larger vocabularies and larger embedding spaces could give additional accuracy.
- Parameter tuning — We can improve accuracy by tuning meta-parameters like the learning rate or number of steps, especially with a different module. A validation set is important if we want reasonable results, since it's easy to set up a model that learns to predict the training data without generalizing well to the test set.
- More complex model — We used a module that computes a sentence embedding by embedding each word and combining them via average. A sequential module (e.g. Universal Sentence Encoder) could better capture the nature of sentences, or you could use an ensemble of two or more TF-Hub modules.
- Regularization — To prevent overfitting, try an optimizer that does some sort of regularization, like Proximal Adagrad.
Advanced: Comparing Modules
Transfer learning makes it possible to save training resources and achieve good model generalization even on a small dataset. We'll demonstrate this by training with two different TF-Hub modules:
nnlm-en-dim128— a pretrained text embedding modulerandom-nnlm-en-dim128— a text embedding module with the same vocabulary and network asnnlm-en-dim128, but with weights randomly initialized and never trained on real data
And by training in two modes: training only the classifier (freezing the module), and training the classifier together with the module.
Let's run a couple of trainings and evaluations to see how using various modules affects accuracy.
def train_and_evaluate_with_module(hub_module, train_module=False):
embedded_text_feature_column = hub.text_embedding_column(
key="sentence", module_spec=hub_module, trainable=train_module)
estimator = tf.estimator.DNNClassifier(
hidden_units=[500, 100],
feature_columns=[embedded_text_feature_column],
n_classes=2,
optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.003))
estimator.train(input_fn=train_input_fn, steps=1000)
train_eval_result = estimator.evaluate(input_fn=predict_train_input_fn)
test_eval_result = estimator.evaluate(input_fn=predict_test_input_fn)
training_set_accuracy = train_eval_result["accuracy"]
test_set_accuracy = test_eval_result["accuracy"]
return {
"Training accuracy": training_set_accuracy,
"Test accuracy": test_set_accuracy
}
results = {}
results["nnlm-en-dim128"] = train_and_evaluate_with_module(
"https://tfhub.dev/google/nnlm-en-dim128/1")
results["nnlm-en-dim128-with-module-training"] = train_and_evaluate_with_module(
"https://tfhub.dev/google/nnlm-en-dim128/1", True)
results["random-nnlm-en-dim128"] = train_and_evaluate_with_module(
"https://tfhub.dev/google/random-nnlm-en-dim128/1")
results["random-nnlm-en-dim128-with-module-training"] = train_and_evaluate_with_module(
"https://tfhub.dev/google/random-nnlm-en-dim128/1", True)
pd.DataFrame.from_dict(results, orient="index")
We can already see some patterns, but first let's establish the baseline accuracy of the test set — the lower bound achievable by always outputting the label of the most represented class:
estimator.evaluate(input_fn=predict_test_input_fn)["accuracy_baseline"]
Assigning the most represented class gives us an accuracy of 50%. A couple of things to notice:
- Maybe surprisingly, a model can still learn on top of fixed, random embeddings. Even if every word in the dictionary maps to a random vector, the estimator can separate the space purely using its fully connected layers.
- Allowing the module to train with random embeddings increases both training and test accuracy compared to training just the classifier.
- Training the module with pre-trained embeddings also increases both accuracies — though note the overfitting on the training set. Training a pre-trained module can be dangerous, even with regularization, since the embedding weights no longer represent the language model trained on diverse data; instead, they converge to the ideal representation of the new dataset.
References
- Reference colab notebook
- monkeylearn.com/text-classification
- digitalvidya.com/blog/text-classification
- tensorflow.org/hub
- ai.stanford.edu — Learning Word Vectors for Sentiment Analysis (PDF)
- tensorflow.org/guide/estimator
- stats.stackexchange.com — Explanation of DNNClassifier in TensorFlow
- tensorflow.org/hub/api_docs — text_embedding_column