Submission through Canvas: upload a zip file with the name HW3-yourNETID-code.zip before the
deadline (above) Penalty for wrong formatting & file naming: 5% grade deduction
LSTM & TRANSFORMERS FOR TEXT MINING, TOPIC MINING
[+ LEARNING TO RANK]
(100 points)
For this homework we will continue working with the News dataset from previous homework.
https://github.com/mhjabreel/CharCnn Keras/blob/master/data/ag news csv/test.csv
1 Word Prediction with Word2Vec and LSTM (50 points)
For this question, you will need the test.csv file of the News dataset from both previous homework.
You will also need to install both TensorFlow (pip install tensorflow==2.0.0-alpha0) and
Keras (pip install keras) – alternatively you can use Keras from TensorFlow. After installation
of the libraries create a Jupyter notebook that does the following: 1) Download the news data file
above (test.csv) 2) Load it with Pandas and run the usual data preprocessing we have done so far
(transform to lowercase, remove numbers, punctuation, NAs, etc. stemming is optional) and select
the 3rd column as the document content 3) Modify the attached LSTM code to use a 100-dimensial
time series (instead of 3-dimensional one) and predict the next two words as follows. Create a
100-dimensional time-series, for which you will use a word2vec encoding and create vector words
of dimension 100. Predict the next two word vectors. For the final prediction, you have to indicate
the actual words (in English) that were predicted. Thus, find the words that are closer in meaning
to the 2 word-vectors you predicted. Specifically: 3.1) For word2vec feel free to use a pre-trained
word2vec (training is only needed for LSTM) and any Python library but I suggest you use gensim.
Once you encoded the collection using word2vec, you will have to modify the code to use these 100-
dimensional time series which come from the word vectors transitions from one word to the next
in the sentences of the news collection. Notice that the example code stacks the data horizontally.
This is because each time series is built separately one series at a time. However, since our “time”
series come from vector sequences, and thus we are building each element of the 100 sequences
at once, you will have to stack the data vertically (vstack) and iteratively one-word vector at a
time. 3.2) Modify the code to predict the next value of the skipgram word-vector sequence. To do
that you will need to modify the code to use the word-vectors from the dataset. Then, you will
have to partition the dataset with the entries corresponding to the known words (e.g. first n − 2
words of a sentence) for training and the unknown words (e.g. last 2 words) for prediction. 3.3)
Plot a histogram of the root mean squared error (RMSE) values of the predictions for both the
tanh activation function and the ReLU activation function – you can plot them side by side using
different colors for each histogram. This will provide an estimate of performance.
1
Up to this point, you are experimenting with the various documents in the collection of documents,
points 3.4 and 3.5 need to be done for a selected document as indicated next: 3.4) Select the
activation function (tanh or ReLU) with better distribution of RMSE and predict the last 2 word-
vectors of one document. The doc you will choose for this is the one with most words in the
collection. 3.5) Find the words that are closer in meaning to the word-vectors you predicted. If
you are using gensim, you can use the function model.wv.similar by vector to obtain the words
that are closer in meaning to a given word vector(s).
4) Add a boolean variable to alternatively include the stop words in the code. 5) Run the code
(Word2Vec encoding and LSTM prediction) but include the stop words when the flag of the previous
step is set to True. How do the RMSE of the predicted words compare with and without stop words
(you can alternatively plot the histogram of F-scores of the matched closest words)? How does the
quality of the prediction compare? Write your answers in a README file. 6) Set the flag to the
option (without or with stop words) that performed better for the code you submit.
Note. To simplify this problem, you can select only one sentence from each document instead of
the full document. That is, you can parse the sentences (before removing punctuation) and work
only with the sentence with the most words in each document.
# multivariate lstm code based on Brownlee (2020)
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense
from keras.layers import RepeatVector
from keras.layers import TimeDistributed
# split a multivariate sequence into samples
def split_sequences(sequences , n_steps_in , n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x , seq_y = sequences[i:end_ix , :], sequences[end_ix:out_end_ix , :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)
# define input sequence
in_seq1 = array([10 , 20 , 30 , 40 , 50 , 60 , 70, 80, 90])
in_seq2 = array([15 , 25 , 35 , 45 , 55 , 65 , 75, 85, 95])
in_seq3 = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
# convert to [rows , columns] structure
in_seq1 = in_seq1.reshape ((len(in_seq1), 1))
in_seq2 = in_seq2.reshape ((len(in_seq2), 1))
in_seq3 = in_seq3.reshape ((len(in_seq3), 1))
# horizontally stack columns
dataset = hstack ((in_seq1 , in_seq2 , in_seq3))
2
# choose a number of time steps
n_steps_in , n_steps_out = 3, 2
# covert into input/output
X, y = split_sequences(dataset , n_steps_in , n_steps_out)
# the dataset knows the number of features , e.g. 2
n_features = X.shape[2]
# define model
model = Sequential ()
model.add(LSTM(200 , activation=’relu’, input_shape=(n_steps_in , n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(200 , activation=’relu’, return_sequences=True))
model.add(TimeDistributed(Dense(n_features)))
model.compile(optimizer=’adam’, loss=’mse’)
# fit model
model.fit(X, y, epochs=300 , verbose=0)
# demonstrate prediction
x_input = array([[60 , 65 , 125], [70 , 75 , 145], [80, 85, 165]])
x_input = x_input.reshape ((1, n_steps_in , n_features))
yhat = model.predict(x_input , verbose=0)
print(yhat)
2 Word Prediction with Transformers - (30 points)
For this problem, you will use the transformers library (!pip install transformers) from Hug-
ging Face. You will also use Torch and Numpy. Similarly to Problem 1, you will use the test.csv
file of the News dataset from both previous homework. Unlike LSTM you will use pre-trained
models, BERT and GPT2, and thus, no training is needed for this problem (no partitioning of
the data in training and test sets, there will be only test sets). 1) Download the news data file
above 2) Load it with Pandas and run the usual data preprocessing we have done so far (transform
to lowercase, remove numbers, punctuation, NAs, etc. stemming is optional) and select the 3rd
column as document content 3) You will use BERT and GPT2 to predict the next (one) word as
follows. 3.1) Use the sample code below to create 2 functions (1 for BERT and 1 for GPT3) that use
the pre-trained models to forecast the next word given a prefix sequence. 3.2) Create a function
or process that iterates over the documents in the collection and use the first n − 1 words of a
sentence as a “prompt” or test sequence and the unknown word (e.g. the last word) for prediction.
3.3) Compare the predicted word with the actual word in the sentence to compute 2 performance
estimates: 1 mandatory, the accuracy, and one optional, the cosine-similarity distributions with
pseudo-feedback (extra credit, detailed below). Compute the accuracy for both models (number of
correctly predicted words/total predictions) - this number should be low for the default BERT and
a little higher for GTP2. Now add back the points (the punctuation) at the end of the sentences
and run your predictions again - the performance now should be a little higher for BERT.
Note. To simplify this problem, you can select only one sentence from each document instead of
the full document. That is, you can parse the sentences (before removing punctuation) and work
only with the sentence with the most words in each document.
EXTRA CREDIT (5 points): Run 1 additional evaluation metric, namely, a pseudo-feedback cosine-
similarity of a reference embedding. The process is as follows: use a pre-trained word2vec model
to encode both the real word and the predicted word (of both BERT and GPT2) and compute the
cosine similarity of their word-vectors. Plot the distribution/histogram of cosine similarities for
both BERT and GPT2; plot the histograms in the same plot using different colors for each model.
Which one is better? Write your answer in the README file.
3
Sample code to predict next word with pre-trained BERT
from transformers import pipeline
model = pipeline(’fill -mask’, model=’bert -base -uncased ’)
pred = model("Forecasts of Presidential [MASK]")
print("Predicted next word: ")
pred[0][’token_str ’]
Sample code to predict next word with pre-trained GPT2
import torch
from transformers import AutoModelForCausalLM , AutoTokenizer
import numpy as np
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model2 = AutoModelForCausalLM.from_pretrained("gpt2")
seq = "Forecasts of Presidential"
inputs = tokenizer(seq , return_tensors="pt")
input_ids = inputs["input_ids"]
for id in input_ids[0]:
word = tokenizer.decode(id)
with torch.no_grad ():
logits = model2( ** inputs).logits[:, -1, :]
pred_id = torch.argmax(logits).item()
pred_word = tokenizer.decode(pred_id)
print("Predicted next word: ")
print(pred_word)
3 Topic Modeling - Multinomial PMMModel and LDA(20 points)
For this problem, you can use the test.csv file from the News dataset. Each row corresponds to a
document and the third column is the content of the documents. For this problem, you will use the
gensim library to identify topics in the collection. The following code shows how to use LdaModel:
# Create the object for LDA model
lda1 = gensim.models.ldamodel.LdaModel
# Train the LDA model using the document term matrix.
ldamodel = lda1(matrix_of_doc_term , num_topics=10, id2word = D1 , passes=100)
matrix of doc term can be computed with the doc2bow function by providing a document from
the corpus to the function. The matrix is built by computing doc2bow for every document in the
corpus. The steps are:
1. create the dictionary D1 with gensim.corpora.Dictionary (use the collection)
2. compute the matrix_of_doc_term by computing D1.doc2bow for every document
in the collection
4
Finally, use ldamodel to plot:
a. The 5 most relevant terms for one of the topics
b. Plot a matrix comparing the distance/difference of the 10 topics. Use the Kullback-Leibler diver-
gence and 50 words. e.g. ldamodel.diff(ldamodel, distance=’kullback leibler’, num words=50).
You can use any library for the plots but Matplotlib or ggplot are recommended.
4 EXTRA CREDIT - Logistic Regression for Learning to Rank
with PageRank, HITS, and TF-IDF (25 points)
Create a function that trains a learning-to-rank algorithm for classifying web-pages as relevant
or not, using logistic regression, as follows. Use the set of pages you downloaded with the web
crawler you build in HW2 to build a graph of URL links. Out of the ten pages, select a query
or topic for which approximately only half of the pages will be relevant and half irrelevant. If
the pages you downloaded for HW2 are not easy to label this way, choose and download a dif-
ferent set. You will use this manual identification of relevant and non-relevant pages as your
ground truth of labels as indicated below. Use the content of the HTML page as content of
the document itself, index the pages as you wish (e.g. titles, file prefixes, numbers from 1 to
# of pages, etc.) but number them (for the matrix representations of PageRank and HITS) and
use the numbers as the indices of the nodes, and use the hyperlinks as edges in the graph. Specifi-
cally, use the page id (e.g. a number or similar) as the node and the hyperlink ()
as an edge between two pages. For instance if the page www.nytimes.com links to your last
page, say https://www.nytimes.com/2023/11/10/technology/personalized-ai-agents.html
you can index the first as node 1 and the second as node 10 and so on, then, you could assign a
link between nodes 1 and 10 (alternatively, 0 and 9). You will use the labeling of relevant pages
you created manually (indicated above) and build a dataset as follows:
Relevant? PageRankScore HITSScore TF IDF
y(1) x
(1)
1 x
(1)
2 x
(1)
3
y(2) x
(2)
1 x
(2)
2 x
(2)
3
. . .
y(10) x
(10)
1 x
(10)
2 x
(10)
3
Table 1: Dataset
Compute both HITS and PageRank scores (you don’t have to implement either PageRank nor HITS
and you can use any library of your choice), in addition to the TF-IDF score of the pages. Arrange
the three scores for each page and for the cases neded consider a set of words of your interest as
shown in Table 1. For instance, you can use: [reuters, stocks, friday, investment, market, prices] as
words that are relevant for the topic financial markets.
Then, use Scikit-learn (link) to apply a logistic regression classifier (no coding from scratch is
expected for logistic regression) to classify documents that are relevant from those that are irrelevant
using the dataset you just created. Thus, you are building a learning-to-rank program that will
consider not only the content of the pages but the hierarchy of importance of the documents to
rank them. Partition your data into training and test sets by using 60% pages as training and
40% as testing. Run the prediction of relevance for your pages and report the precision and recall
5
curve and F1-scores (for both training and test sets ). Make sure that the partitions have an equal
representation of each of the two types of pages in it (relevant and not relevant).
WHAT TO TURN IN
Mandatory (Graded) elements/files within the zip file:
1) The Jupyter notebook with the solutions to the problems (please, annotate & comment your
code) and
2) a README file with the answers to the questions in the problems
3) (Extra Credit) For the Learning-To-Rank question: In addition to the notebook (can be a
separate notebook but it is better to use the same), submit a PDF with a plot of the results and a
one-paragraph description of what you found most interesting.
Optional (no points will be awarded for this!):
You can also include in the README file a URL link to your Colab file where you can show what
you did in case that helps check what you did. However, this is just to verify outputs, and the
code in your Collab repositoty will not be considered for grading. Only the submitted code will be
graded.
REFERENCES
Brownlee, J. How to Develop LSTM Models for Time Series Forecasting. (2020).