Introduction to Text Mining

How has the way we analyze text changed over time?

We look at visitor reviews of the Colosseum in Rome to compare two approaches.

Era Approach Concept
Classic TF-IDF Words as weighted counts
Modern Embeddings Text as a point in meaning-space

Find the jupyter notebook to download here: Link

Download Colosseum reviews

# Download text dataset and install requirements
!curl -L -o ./rome-colosseum-visitor-reviews.zip https://www.kaggle.com/api/v1/datasets/download/uradkr/rome-colosseum-visitor-reviews
!unzip rome-colosseum-visitor-reviews.zip
!rm rome-colosseum-visitor-reviews.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1237k  100 1237k    0     0  1260k      0 --:--:-- --:--:-- --:--:-- 3239k
Archive:  rome-colosseum-visitor-reviews.zip
  inflating: rome_colosseum_visitor_reviews_final.csv  

Install requirements

# !pip install sentence-transformers umap-learn plotly pandas

Import libraries

from sentence_transformers import SentenceTransformer
import plotly.express as px
import umap
import pandas as pd
import re
/home/jelicic/.local/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
/home/jelicic/.local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (1.26.20) or chardet (7.4.3)/charset_normalizer (3.4.7) doesn't match a supported version!
  warnings.warn(

Helper functions

def clean_text(text):
    regexp = r"[^\w\s]"
    text = re.sub(regexp,'',text).strip().lower()
    return text

def sort_dict_by_values(d):
    return {k: v for k, v in sorted(d.items(), key=lambda item: item[1],reverse=True)}
    
df = pd.read_csv('rome_colosseum_visitor_reviews_final.csv')

print(df.shape[0])


df
8285
published_date travel_month published_platform title text rating tripType helpful_votes word_count review_length_tier sentiment_label published_year published_month travel_season is_verified_travel
0 2019-05-11 2019-05 OTHER Colosseum A must see for any visitor to Rome, but go to ... 5 COUPLES 2 90 Long (75-200w) Positive 2019 5 Spring True
1 2019-05-11 2019-04 OTHER Amazing Structure and Architecture from The Past It is amazing structure built in Roman empire,... 5 COUPLES 2 507 Very Long (200w+) Positive 2019 5 Spring True
2 2019-05-11 2019-05 MOBILE C Wonderful experience. We were blessed with a p... 5 COUPLES 2 20 Short (<25w) Positive 2019 5 Spring True
3 2019-05-11 2019-04 MOBILE SPECTACULAR Ok the Colosseum is an amazing place, iconic a... 5 FAMILY 2 149 Long (75-200w) Positive 2019 5 Spring True
4 2019-05-11 2018-06 OTHER More Amazing in Real Life! The first sight of the Colosseum is overwhelmi... 5 FAMILY 0 59 Medium (25-75w) Positive 2019 5 Summer True
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
8280 2026-06-17 2026-05 OTHER Book Direct for the Arena Ticket We visited the Colosseum using the Full Experi... 5 COUPLES 0 194 Long (75-200w) Positive 2026 6 Spring True
8281 2026-06-19 2026-06 MOBILE Family Tour The site is a must-do; even if the crowds are ... 5 FAMILY 0 59 Medium (25-75w) Positive 2026 6 Summer True
8282 2026-06-19 2026-06 OTHER Step back in time Amazing to experience this walk round would pr... 5 COUPLES 0 17 Short (<25w) Positive 2026 6 Summer True
8283 2026-06-20 2026-06 OTHER A must visit Fantastic experience full of history with ever... 5 COUPLES 0 36 Medium (25-75w) Positive 2026 6 Summer True
8284 2026-06-22 2026-06 MOBILE Excellent Brilliant experience booked through Get Your G... 5 COUPLES 0 39 Medium (25-75w) Positive 2026 6 Summer True

8285 rows × 15 columns

texts = df['text'].apply(clean_text)

print(texts.values[-1])
brilliant experience booked through get your guide which was very easy and our guide aphrodite was an excellent guide that pointed out all the main feat of the areas we visited would use get your guide again without doubt

Finding Important Words

If a word appears often in a document, it is probably important to that document. But words like the, and, is appear in every document and carry no signal.

TF-IDF penalizes words that are common across all documents:

\[\text{TF-IDF}(word, doc) = \underbrace{\text{count in doc}}_{\text{Term Frequency}} \times \underbrace{\frac{1}{\text{count across all docs}}}_{\text{Inverse Document Frequency}}\]

A word scores high only when it is frequent in this document and rare everywhere else.

Step 1: Document Frequency

How many reviews contain each word? Words that appear in many reviews carry less signal.



doc_freq = dict()

for text in texts:
    for word in text.split(' '):
        if doc_freq.get(word):
            doc_freq[word]+=1 
        else:
            doc_freq[word] = 1
        
doc_freq

#facn sorting
sort_dict_by_values(doc_freq)
{'the': 34340,
 'to': 17429,
 'and': 17256,
 'a': 13804,
 'of': 10915,
 'in': 8809,
 'it': 8426,
 'you': 7999,
 'is': 7730,
 'was': 7632,
 'we': 7587,
 '': 7507,
 'tour': 5650,
 'for': 5123,
 'i': 5007,
 'colosseum': 4957,
 'this': 3850,
 'with': 3825,
 'but': 3647,
 'that': 3632,
 'as': 3246,
 'so': 3110,
 'on': 3103,
 'there': 2941,
 'rome': 2899,
 'not': 2868,
 'are': 2867,
 'guide': 2825,
 'at': 2749,
 'time': 2619,
 'be': 2612,
 'visit': 2604,
 'see': 2549,
 'get': 2539,
 'very': 2531,
 'were': 2512,
 'have': 2454,
 'tickets': 2430,
 'our': 2387,
 'history': 2334,
 'its': 2291,
 'an': 2279,
 'if': 2046,
 'from': 1957,
 'place': 1928,
 'had': 1926,
 'all': 1878,
 'go': 1852,
 'can': 1846,
 'your': 1791,
 'ticket': 1759,
 'they': 1689,
 'line': 1636,
 'amazing': 1604,
 'one': 1560,
 'which': 1539,
 'would': 1533,
 'must': 1473,
 'just': 1465,
 'worth': 1464,
 'when': 1434,
 'people': 1414,
 'forum': 1386,
 'my': 1379,
 'about': 1368,
 'inside': 1361,
 'experience': 1357,
 'around': 1341,
 'great': 1335,
 'do': 1320,
 'roman': 1275,
 'by': 1274,
 'day': 1258,
 'us': 1248,
 'or': 1231,
 'up': 1207,
 'much': 1206,
 'recommend': 1192,
 'more': 1137,
 'what': 1108,
 'well': 1093,
 'no': 1088,
 'guided': 1084,
 'will': 1073,
 'also': 1060,
 'did': 1054,
 'only': 1041,
 'long': 1040,
 'really': 1017,
 'take': 984,
 'skip': 984,
 'out': 903,
 'outside': 867,
 'through': 862,
 'before': 860,
 'went': 856,
 'good': 848,
 'even': 838,
 'many': 837,
 'hill': 806,
 'like': 806,
 'queue': 805,
 'book': 794,
 'buy': 779,
 'dont': 767,
 'palatine': 749,
 'some': 748,
 'site': 748,
 'arena': 742,
 'booked': 737,
 'then': 731,
 'could': 702,
 'lot': 693,
 'where': 688,
 'who': 688,
 'been': 683,
 'walk': 676,
 'still': 675,
 'hours': 673,
 'how': 665,
 'took': 663,
 'didnt': 658,
 'got': 657,
 'than': 646,
 'too': 641,
 'visiting': 626,
 'most': 620,
 'beautiful': 616,
 'definitely': 612,
 'first': 610,
 'advance': 605,
 'ancient': 604,
 'tours': 602,
 'online': 600,
 'because': 599,
 'going': 595,
 'here': 595,
 'underground': 594,
 'building': 591,
 'into': 584,
 'interesting': 581,
 'visited': 574,
 'water': 571,
 'entry': 568,
 'make': 567,
 'back': 565,
 'way': 561,
 'best': 560,
 'sure': 557,
 'walking': 551,
 'has': 542,
 'without': 534,
 'early': 533,
 'floor': 532,
 'structure': 531,
 'other': 531,
 'me': 525,
 'group': 523,
 'she': 521,
 'entrance': 516,
 'years': 514,
 'need': 513,
 'queues': 510,
 'crowds': 506,
 'lines': 506,
 'wait': 504,
 'hot': 502,
 'after': 501,
 'them': 498,
 'once': 490,
 'access': 485,
 'over': 482,
 'highly': 475,
 'audio': 470,
 'hour': 469,
 'information': 466,
 'better': 459,
 'area': 457,
 'made': 453,
 'impressive': 451,
 'trip': 449,
 'night': 438,
 'free': 435,
 '2': 435,
 'he': 435,
 'lots': 434,
 'want': 433,
 'coliseum': 432,
 'full': 429,
 'busy': 426,
 'any': 420,
 'minutes': 399,
 'being': 396,
 'crowded': 393,
 'their': 393,
 'world': 392,
 'able': 392,
 'gladiators': 391,
 'part': 390,
 'think': 386,
 'during': 380,
 'money': 378,
 'such': 377,
 'everything': 372,
 'incredible': 372,
 'times': 370,
 'avoid': 367,
 'seeing': 366,
 'nice': 363,
 'bit': 356,
 'huge': 353,
 'while': 346,
 'city': 345,
 'little': 343,
 'however': 343,
 'own': 341,
 'knowledgeable': 335,
 'again': 333,
 'every': 333,
 'down': 330,
 'view': 329,
 '3': 329,
 'say': 327,
 'know': 323,
 'bought': 323,
 'find': 321,
 'monument': 321,
 'historical': 321,
 'feel': 319,
 'fantastic': 318,
 'enter': 318,
 'off': 316,
 'absolutely': 313,
 'those': 313,
 'enjoyed': 313,
 'used': 311,
 'pay': 307,
 'guides': 306,
 'architecture': 305,
 'these': 304,
 'waiting': 304,
 'security': 301,
 'trying': 300,
 'should': 300,
 'cant': 300,
 'easy': 298,
 'right': 297,
 'come': 295,
 'look': 294,
 'getting': 293,
 'staff': 291,
 'though': 289,
 'two': 287,
 'few': 286,
 'official': 286,
 'quite': 285,
 'wonderful': 284,
 'places': 284,
 'excellent': 284,
 'pictures': 283,
 'things': 282,
 'enough': 281,
 'morning': 281,
 'different': 281,
 'away': 280,
 'booking': 279,
 'same': 279,
 'small': 277,
 'always': 276,
 'thing': 274,
 'her': 274,
 'built': 272,
 'told': 272,
 'tourists': 272,
 'days': 272,
 'ago': 271,
 'included': 269,
 'truly': 269,
 'life': 266,
 'enjoy': 266,
 'level': 265,
 'tourist': 261,
 'walked': 261,
 'extra': 260,
 'seen': 260,
 'imagine': 258,
 'found': 258,
 'photos': 258,
 'old': 257,
 'sell': 256,
 'try': 252,
 'both': 252,
 'use': 252,
 'itself': 252,
 'pass': 250,
 'attraction': 250,
 'another': 249,
 'something': 247,
 'areas': 247,
 'each': 246,
 'learn': 246,
 'big': 245,
 'sites': 244,
 'informative': 244,
 'said': 244,
 'youre': 242,
 'website': 241,
 'next': 240,
 'italy': 238,
 'whole': 236,
 'loved': 235,
 'heat': 234,
 'magnificent': 234,
 'taking': 233,
 'give': 233,
 'never': 233,
 'price': 232,
 'top': 232,
 'views': 230,
 'understand': 229,
 'paid': 229,
 'having': 228,
 'person': 227,
 'miss': 222,
 'id': 218,
 'everyone': 215,
 'gladiator': 214,
 'gave': 214,
 'least': 213,
 'point': 213,
 'construction': 211,
 'wasnt': 211,
 'standing': 211,
 'saw': 209,
 'am': 209,
 'roma': 207,
 'bring': 207,
 'may': 206,
 'almost': 205,
 'ruins': 204,
 'sun': 204,
 'available': 202,
 'spend': 201,
 'awesome': 200,
 'yourself': 199,
 'looking': 198,
 'wanted': 198,
 'euros': 198,
 'due': 197,
 'spent': 197,
 'second': 196,
 'left': 196,
 'actually': 194,
 'prepared': 194,
 'especially': 194,
 'last': 193,
 'plenty': 193,
 'levels': 192,
 'arrived': 192,
 'size': 192,
 'less': 192,
 'although': 190,
 'past': 189,
 'felt': 189,
 'now': 189,
 'work': 188,
 'fascinating': 188,
 'summer': 187,
 'family': 187,
 'since': 185,
 '1': 183,
 'his': 183,
 'animals': 183,
 'spectacular': 182,
 'allowed': 182,
 'doing': 182,
 'english': 181,
 'wonder': 179,
 'ahead': 179,
 'iconic': 178,
 '5': 177,
 'help': 175,
 'breathtaking': 175,
 'glad': 175,
 'romans': 175,
 'piece': 174,
 'recommended': 174,
 'front': 173,
 'let': 173,
 'private': 173,
 'under': 173,
 'read': 171,
 'hard': 170,
 'purchase': 169,
 'groups': 169,
 'ever': 168,
 'metro': 168,
 'done': 168,
 'course': 168,
 'office': 167,
 'couldnt': 166,
 'colloseum': 166,
 '10': 166,
 'anyone': 165,
 'fast': 165,
 'historic': 165,
 '2000': 165,
 'theres': 163,
 'late': 162,
 'visitors': 161,
 'company': 161,
 'museum': 159,
 'extremely': 159,
 'probably': 159,
 'love': 159,
 'otherwise': 158,
 'nothing': 158,
 'already': 158,
 'stand': 157,
 '30': 156,
 'etc': 156,
 'stairs': 156,
 'steps': 155,
 'stunning': 155,
 'street': 155,
 'thought': 154,
 'sight': 153,
 'im': 153,
 'massive': 153,
 'half': 152,
 'slot': 152,
 'wonders': 152,
 'pretty': 151,
 'show': 151,
 'number': 150,
 '20': 150,
 'far': 149,
 'real': 149,
 'large': 149,
 '15': 148,
 'check': 147,
 'attractions': 147,
 'purchased': 146,
 'possible': 146,
 'until': 145,
 'super': 144,
 'came': 144,
 'appreciate': 144,
 'missed': 144,
 'cannot': 144,
 'shoes': 143,
 'close': 143,
 'stories': 143,
 'keep': 143,
 'knowledge': 142,
 'despite': 142,
 'bad': 141,
 'photo': 140,
 'three': 140,
 'cost': 138,
 'via': 138,
 'upper': 138,
 'suggest': 138,
 'empire': 137,
 'new': 137,
 'learned': 136,
 'simply': 135,
 'afternoon': 135,
 'stop': 135,
 'along': 135,
 'start': 134,
 'main': 133,
 '12': 133,
 'leave': 132,
 'explore': 131,
 '\nthe': 131,
 'list': 131,
 'round': 130,
 'shade': 130,
 'awe': 129,
 'year': 129,
 'gets': 129,
 'behind': 128,
 'friendly': 128,
 'disappointed': 128,
 'mustsee': 128,
 'helpful': 127,
 'buying': 127,
 'several': 127,
 'colosseo': 127,
 'cool': 126,
 'expected': 125,
 'looked': 125,
 'quickly': 124,
 'given': 124,
 'kids': 124,
 'amphitheater': 123,
 'gives': 123,
 'kept': 123,
 'ground': 123,
 'within': 123,
 'wear': 121,
 'weather': 121,
 'takes': 121,
 'expensive': 121,
 'tell': 121,
 'scale': 121,
 'later': 119,
 'anything': 119,
 'makes': 119,
 'high': 119,
 'ive': 119,
 'expect': 118,
 'vatican': 118,
 'difficult': 118,
 'does': 118,
 'amount': 118,
 'children': 118,
 'fun': 118,
 'grandeur': 118,
 'yes': 117,
 '4': 117,
 'end': 117,
 'put': 116,
 'mind': 116,
 'selling': 116,
 'italian': 116,
 'today': 116,
 'bottle': 116,
 'rather': 115,
 'maybe': 115,
 'stay': 115,
 'believe': 114,
 'pre': 114,
 'parts': 114,
 'including': 114,
 'waited': 114,
 'ask': 113,
 'perfect': 113,
 'thats': 112,
 'overall': 112,
 'sold': 112,
 'youll': 112,
 'unfortunately': 112,
 'aware': 111,
 'crowd': 110,
 'public': 110,
 'evening': 110,
 'hear': 110,
 'facts': 110,
 'decided': 110,
 'questions': 109,
 'else': 109,
 'straight': 108,
 'near': 108,
 'why': 108,
 'details': 108,
 'euro': 107,
 'wouldnt': 107,
 'season': 107,
 '\n\nthe': 107,
 'save': 106,
 'special': 106,
 'needed': 106,
 'queuing': 105,
 'couple': 105,
 'easily': 105,
 'watch': 104,
 'wow': 104,
 'arrive': 104,
 'option': 103,
 'lucky': 103,
 'sense': 103,
 'asked': 102,
 'side': 102,
 'meeting': 102,
 'provided': 102,
 'open': 102,
 'everywhere': 101,
 'short': 101,
 'happy': 101,
 'entire': 101,
 'includes': 101,
 'pace': 100,
 'station': 100,
 'limited': 100,
 'plan': 100,
 'might': 99,
 'comfortable': 99,
 'viator': 99,
 'wont': 99,
 'brilliant': 99,
 'either': 98,
 'toilets': 98,
 'marvel': 98,
 'reservation': 98,
 'longer': 98,
 'chance': 97,
 'touts': 97,
 'started': 97,
 'thousands': 97,
 'kind': 97,
 'feeling': 96,
 'famous': 96,
 'month': 96,
 'paying': 96,
 'culture': 96,
 'brought': 95,
 '\nwe': 95,
 'happened': 95,
 'knew': 94,
 'important': 94,
 'story': 94,
 'engineering': 94,
 'minute': 94,
 'bottles': 94,
 'atmosphere': 94,
 'allow': 94,
 'unless': 93,
 'explained': 93,
 'lovely': 93,
 'travel': 92,
 '100': 92,
 'idea': 91,
 'sights': 91,
 'service': 91,
 'totally': 91,
 'complete': 91,
 'isnt': 91,
 'bus': 91,
 'beauty': 91,
 'thank': 90,
 'wish': 90,
 'looks': 90,
 'please': 89,
 'admission': 89,
 'nearby': 89,
 'per': 89,
 'plus': 89,
 'name': 89,
 'remember': 89,
 'surrounding': 88,
 'gate': 88,
 'signs': 88,
 'sheer': 88,
 'using': 88,
 'fact': 88,
 'months': 87,
 'steep': 87,
 'app': 87,
 'value': 87,
 'unique': 86,
 'clear': 86,
 'highlight': 86,
 'phone': 86,
 'hand': 86,
 'closed': 86,
 'tried': 85,
 'someone': 85,
 'directly': 85,
 'ourselves': 85,
 'doesnt': 85,
 'sellers': 85,
 'architectural': 85,
 'reason': 85,
 'between': 85,
 'across': 85,
 'landmark': 84,
 'venue': 84,
 'lift': 84,
 'making': 84,
 'modern': 84,
 'info': 84,
 'managed': 84,
 'instead': 84,
 'elevator': 83,
 'amphitheatre': 83,
 'mins': 83,
 '\n\nwe': 83,
 '18': 83,
 'organized': 83,
 'rude': 83,
 'picture': 82,
 'quick': 82,
 'whilst': 82,
 'self': 82,
 'tip': 82,
 'local': 81,
 'pick': 81,
 'throughout': 81,
 'preserved': 81,
 'inspiring': 80,
 'others': 80,
 'taken': 80,
 'timed': 80,
 'various': 80,
 'battles': 79,
 '7': 79,
 'forget': 79,
 'skiptheline': 79,
 'entering': 79,
 '45': 79,
 'advise': 79,
 'him': 79,
 'explain': 79,
 'spot': 79,
 'husband': 78,
 'wrong': 78,
 'careful': 78,
 'incredibly': 78,
 'fountains': 78,
 'true': 77,
 'finally': 77,
 'organised': 77,
 'week': 77,
 'shop': 77,
 'absolute': 76,
 'lower': 75,
 'called': 75,
 'system': 75,
 'offer': 75,
 'together': 75,
 'eyes': 75,
 'games': 75,
 'unforgettable': 75,
 'original': 74,
 'beware': 74,
 'rich': 74,
 'obviously': 74,
 'provide': 74,
 'moment': 74,
 'hotel': 74,
 'above': 74,
 'alone': 74,
 'prior': 74,
 'sunday': 74,
 'seemed': 73,
 'move': 73,
 'palantine': 73,
 'enjoyable': 73,
 'join': 73,
 'opportunity': 73,
 'entered': 73,
 'chose': 73,
 'onto': 73,
 'u': 72,
 'prebooked': 72,
 'advice': 72,
 'needs': 72,
 'space': 72,
 'restoration': 72,
 'coming': 72,
 'seems': 71,
 'impressed': 71,
 'werent': 71,
 'build': 71,
 'july': 71,
 'packed': 70,
 'events': 70,
 'points': 70,
 'soon': 70,
 'worked': 70,
 'imposing': 70,
 'lit': 70,
 'spectators': 70,
 'interior': 70,
 'helped': 70,
 'problem': 69,
 'showed': 69,
 'set': 69,
 'symbol': 69,
 'below': 69,
 'include': 69,
 'order': 69,
 'head': 69,
 'certainly': 69,
 'turned': 69,
 'stood': 69,
 'majestic': 69,
 'centuries': 69,
 'ad': 69,
 'overwhelming': 68,
 'track': 68,
 'saying': 68,
 'rest': 68,
 'man': 68,
 'lifetime': 68,
 'general': 68,
 'lost': 68,
 'stone': 68,
 'walls': 68,
 'stuff': 67,
 'unbelievable': 67,
 'ended': 67,
 'job': 67,
 'return': 67,
 'allows': 67,
 'remains': 67,
 'bucket': 67,
 'fully': 67,
 'headout': 67,
 'refund': 66,
 'rick': 66,
 'exploring': 66,
 'interested': 66,
 'climb': 66,
 'fabulous': 66,
 'guy': 66,
 'heard': 66,
 'hat': 66,
 'completely': 66,
 'amazed': 65,
 'floors': 65,
 'que': 65,
 '6': 65,
 'words': 65,
 'fill': 64,
 'vendors': 64,
 '\n\ni': 64,
 'disappointing': 64,
 'earlier': 64,
 'october': 64,
 'actual': 63,
 'live': 63,
 'wife': 63,
 'fine': 63,
 'video': 63,
 'charge': 62,
 'review': 62,
 'popular': 62,
 'known': 62,
 'thanks': 62,
 'stands': 62,
 'yet': 62,
 'prebook': 62,
 'exhibits': 62,
 'steves': 61,
 'exit': 61,
 'scaffolding': 61,
 'covered': 61,
 'turn': 61,
 '\ni': 61,
 'emperor': 61,
 'beat': 60,
 'rain': 60,
 'gift': 60,
 '24': 60,
 'skipped': 60,
 'monuments': 60,
 'buildings': 60,
 'waste': 60,
 'gone': 60,
 'romes': 60,
 'fountain': 60,
 'ok': 59,
 'imagination': 59,
 'annoying': 59,
 'exactly': 59,
 'offering': 59,
 'twice': 59,
 'breath': 59,
 'basic': 59,
 'beforehand': 58,
 'middle': 58,
 'knowledgable': 58,
 'works': 58,
 'cheaper': 58,
 'run': 58,
 'ones': 58,
 'added': 58,
 'clearly': 58,
 'card': 58,
 'goes': 58,
 'august': 58,
 'location': 57,
 'personal': 57,
 'email': 57,
 'largest': 57,
 'young': 57,
 'toured': 57,
 'follow': 57,
 'detail': 57,
 'stadium': 57,
 'century': 57,
 'located': 57,
 'choose': 57,
 'third': 57,
 'spots': 56,
 'fought': 56,
 'grand': 56,
 'shame': 56,
 '25': 56,
 'opinion': 56,
 'men': 56,
 'sit': 56,
 '35': 56,
 'crazy': 55,
 'hundreds': 55,
 'impossible': 55,
 'process': 55,
 'accessible': 55,
 'combined': 55,
 'ruined': 55,
 'seem': 55,
 'shows': 55,
 'regular': 55,
 'perspective': 55,
 'case': 55,
 'note': 54,
 'add': 54,
 '40': 54,
 'bother': 54,
 'passed': 54,
 'myself': 54,
 'son': 54,
 'romanum': 54,
 'options': 54,
 'food': 54,
 'moved': 54,
 'pm': 54,
 'scam': 53,
 'confusing': 53,
 'opposite': 53,
 'literally': 53,
 'prices': 53,
 'explanation': 53,
 'met': 52,
 'normal': 52,
 'stayed': 52,
 'walks': 52,
 'remarkable': 52,
 'emperors': 52,
 'june': 52,
 'held': 52,
 'meant': 52,
 'direct': 52,
 'offered': 52,
 'disabled': 52,
 'stones': 52,
 '16': 52,
 'displays': 51,
 'entertainment': 51,
 'scammers': 51,
 'moving': 51,
 'exterior': 51,
 'home': 51,
 'hop': 51,
 '50': 51,
 'alive': 51,
 'art': 51,
 'magical': 51,
 'answer': 51,
 'interest': 50,
 '8': 50,
 'hills': 50,
 'insight': 50,
 'daughter': 50,
 'cold': 50,
 'necessary': 50,
 'giving': 50,
 'road': 50,
 'liked': 50,
 'issues': 50,
 'route': 50,
 '80': 50,
 'quiet': 50,
 'guys': 49,
 'weeks': 49,
 'fee': 49,
 'describe': 49,
 'total': 49,
 'thinking': 49,
 'movies': 49,
 'poor': 49,
 'care': 49,
 'often': 49,
 'nearly': 49,
 'spoke': 49,
 'youve': 49,
 'surprised': 49,
 'seating': 49,
 'power': 49,
 'major': 49,
 'speak': 48,
 'europe': 48,
 'age': 48,
 'jump': 48,
 'learning': 48,
 'anyway': 48,
 'gladiatorial': 48,
 'viewing': 48,
 'ages': 48,
 'valid': 48,
 'tiqets': 48,
 'funny': 48,
 'visitor': 47,
 'booth': 47,
 'explanations': 47,
 'mobility': 47,
 'palentine': 47,
 ...}

Step 2: Term Frequency

For each review, count how often each word appears.

term_freq = []

for doc in texts:
    tokenized = dict()
    for word in doc.split(' '):  # fixed: was 'text' (outer loop variable leak)
        if tokenized.get(word):
            tokenized[word] += 1
        else:
            tokenized[word] = 1
    term_freq.append(tokenized)

sort_dict_by_values(term_freq[-1])
{'guide': 4,
 'get': 2,
 'your': 2,
 'was': 2,
 'the': 2,
 'brilliant': 1,
 'experience': 1,
 'booked': 1,
 'through': 1,
 'which': 1,
 'very': 1,
 'easy': 1,
 'and': 1,
 'our': 1,
 'aphrodite': 1,
 'an': 1,
 'excellent': 1,
 'that': 1,
 'pointed': 1,
 'out': 1,
 'all': 1,
 'main': 1,
 'feat': 1,
 'of': 1,
 'areas': 1,
 'we': 1,
 'visited': 1,
 'would': 1,
 'use': 1,
 'again': 1,
 'without': 1,
 'doubt': 1}

Step 3: TF-IDF Score

Multiply term frequency by inverse document frequency. Words that appear in many reviews score low.

tf_idf = []

for tf in term_freq:
    doc_tf_idf = dict()
    for word in tf:
        doc_tf_idf[word] =  round(tf[word] * (1/doc_freq[word]),4)
    
    tf_idf.append(doc_tf_idf)

sort_dict_by_values(tf_idf[-1])
{'aphrodite': 0.3333,
 'feat': 0.0526,
 'pointed': 0.0357,
 'doubt': 0.0294,
 'brilliant': 0.0101,
 'main': 0.0075,
 'areas': 0.004,
 'use': 0.004,
 'excellent': 0.0035,
 'easy': 0.0034,
 'again': 0.003,
 'without': 0.0019,
 'visited': 0.0017,
 'booked': 0.0014,
 'guide': 0.0014,
 'through': 0.0012,
 'your': 0.0011,
 'out': 0.0011,
 'get': 0.0008,
 'experience': 0.0007,
 'would': 0.0007,
 'which': 0.0006,
 'all': 0.0005,
 'very': 0.0004,
 'our': 0.0004,
 'an': 0.0004,
 'was': 0.0003,
 'that': 0.0003,
 'and': 0.0001,
 'the': 0.0001,
 'of': 0.0001,
 'we': 0.0001}

What TF-IDF tells us

aphrodite scores 0.33 because it appears multiple times in this review but almost nowhere else across 8,000+ reviews. It is the word that sets this document apart.

the, and, of score near 0 because they appear in every review.

Limitation: TF-IDF treats text as a bag of words. It does not know that “amazing” and “incredible” are synonyms, or that “not worth it” is negative.

Words that Define Positive and Negative Reviews

TF-IDF scores tell us which words matter to a single document. If we add those scores across a group of reviews, we get the words most associated with that group.

ratings = df['rating'].values
pos_idx = [i for i, r in enumerate(ratings) if r >= 4]
neg_idx = [i for i, r in enumerate(ratings) if r <= 2]

def top_group_words(indices, tf_idf_list, min_doc_freq=5):
    agg = {}
    for i in indices:
        for word, score in tf_idf_list[i].items():
            if word and doc_freq.get(word, 0) >= min_doc_freq:
                agg[word] = agg.get(word, 0) + score
    return sort_dict_by_values(agg)

pos_top = top_group_words(pos_idx, tf_idf)
neg_top = top_group_words(neg_idx, tf_idf)

num_display = 10

print(f"Positive reviews ({len(pos_idx)}, rated 4-5 stars)")
for word, score in list(pos_top.items())[:num_display]:
    print(f"\t{word:20s} {score:.4f}")

print(f"\nNegative reviews ({len(neg_idx)}, rated 1-2 stars)")
for word, score in list(neg_top.items())[:num_display]:
    print(f"\t{word:20s} {score:.4f}")
Positive reviews (7472, rated 4-5 stars)
    masterpiece          1.0013
    titus                1.0011
    disappoint           1.0011
    tips                 1.0010
    monumental           1.0010
    constructed          1.0010
    spectators           1.0010
    powerful             1.0009
    society              1.0009
    pleased              1.0009

Negative reviews (360, rated 1-2 stars)
    245                  1.0000
    unacceptable         0.8888
    corridor             0.8571
    girlfriend           0.8000
    racist               0.8000
    parties              0.8000
    nonexistent          0.8000
    message              0.7500
    cancel               0.7145
    upset                0.7145

Modern Approach: Text Embeddings

TF-IDF treats text as a bag of words. It does not know that “amazing” and “incredible” mean the same thing, or that “not bad” is positive.

Language models turn each piece of text into a vector: a point in high-dimensional space where similar meanings end up close together.

We use all-MiniLM-L6-v2 to embed the reviews, then reduce to 2D to see the structure.

model = SentenceTransformer('all-MiniLM-L6-v2')
/home/jelicic/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:187: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 12040). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:119.)

  return torch._C._cuda_getDeviceCount() > 0

Loading weights: 100%|██████████| 103/103 [00:00<00:00, 13930.97it/s]

BertModel LOAD REPORT from: sentence-transformers/all-MiniLM-L6-v2

Key                     | Status     |  | 

------------------------+------------+--+-

embeddings.position_ids | UNEXPECTED |  | 



Notes:

- UNEXPECTED:   can be ignored when loading from different task/architecture; not ok if you expect identical arch.
# Each review -> a 384-dimensional vector 
texts = df['text'].fillna('').tolist()
embeddings = model.encode(texts, show_progress_bar=True, batch_size=64)
Batches: 100%|██████████| 130/130 [00:48<00:00,  2.70it/s]
# Reduce 384 dimensions -> 2 dimensions with UMAP
reducer = umap.UMAP(n_components=2, random_state=42, n_neighbors=10)
embeddings_2d = reducer.fit_transform(embeddings)

print(f"Reduced: {embeddings.shape} -> {embeddings_2d.shape}")
/home/jelicic/.local/lib/python3.10/site-packages/umap/umap_.py:1943: UserWarning:

n_jobs value -1 overridden to 1 by setting random_state. Use no seed for parallelism.
Reduced: (8285, 384) -> (8285, 2)
import textwrap

def wrap_text(text, width=55):
    return '<br>'.join(textwrap.wrap(str(text), width=width))

plot_df = pd.DataFrame({
    'x': embeddings_2d[:, 0],
    'y': embeddings_2d[:, 1],
    'rating': df['rating'].values,
    'review': df['text'].fillna('').str[:400].apply(wrap_text).values,
})

fig = px.scatter(
    plot_df,
    x='x', y='y',
    color='rating',
    color_continuous_scale='RdYlGn',
    range_color=[1, 5],
    custom_data=['rating', 'review'],
    title='Colosseum Reviews',
    opacity=0.7,
)

fig.update_traces(
    marker=dict(size=6),
    hovertemplate='<b>%{customdata[0]} star(s)</b><br><br>%{customdata[1]}<extra></extra>',
)

fig.update_layout(
    xaxis=dict(showticklabels=False, title=''),
    yaxis=dict(showticklabels=False, title=''),
    margin=dict(l=40, r=160, t=50, b=40),
    coloraxis_colorbar=dict(
        tickvals=[1, 2, 3, 4, 5],
        ticktext=['1 star', '2 stars', '3 stars', '4 stars', '5 stars'],
        title='Rating',
    ),
)

fig.show()
Unable to display output for mime type(s): application/vnd.plotly.v1+json
Back to top