Movies assigned from the quiz:
- Pariyerum Perumal
- Jaws: The Revenge
- Aruvi
- One Flew Over the Cuckoo's Nest
- Anniyan
Data Collection
In [4]:
import csv
from bs4 import BeautifulSoup
import requests
import pandas as pd
import time
time.sleep(2)
import warnings
warnings.filterwarnings('ignore')
In [5]:
urls = []
url1 = 'https://www.imdb.com/title/tt8176054/reviews/?ref_=tt_ql_2'
url2 = 'https://www.imdb.com/title/tt0093300/reviews/?ref_=tt_ql_2'
url3 = 'https://www.imdb.com/title/tt5867800/reviews/?ref_=tt_ql_2'
url4 = 'https://www.imdb.com/title/tt0073486/reviews/?ref_=tt_ql_2'
url5 = 'https://www.imdb.com/title/tt0376127/reviews/?ref_=tt_ql_2'
In [6]:
urls.append(url1)
urls.append(url2)
urls.append(url3)
urls.append(url4)
urls.append(url5)
In [7]:
content = []
for url in urls:
page = requests.get(url, timeout=2.50)
page_content = page.content
soup = BeautifulSoup(page_content, 'html.parser')
content.append(soup.find_all('div', class_= 'review-container'))
print(content)
In [9]:
movie = pd.DataFrame(columns=['Review','Rating'])
In [11]:
review = []
rating = []
count = 0
for cc in content:
for c in cc:
count+= 1
print('\nMovie review ', count)
#Get review.
str = c.find_all('a', attrs={'class':'title'})
rReview =''
for s in str:
#print('Review is: ',s.get_text())
rReview = s.get_text()
#Get rating.
ratings = c.find_all('span', attrs={'class':''})
rVal = []
for r in ratings:
str1 = r.get_text().strip()
rVal.append(str1)
val = rVal[0]
if(len(val) > 2):
continue
else:
review.append(rReview)
rating.append(val)
print('Review: ', rReview)
print('Rating: ',val)
movie['Review'] = review
movie['Rating'] = rating
Movie review 1 Review: Speaks humanity Rating: 10 Movie review 2 Review: The dark underbelly- Casteism Rating: 10 Movie review 3 Review: Deep intense movie, brilliantly narrated script. Rating: 10 Movie review 4 Review: Another intense film after merku thodarchi malai. Rating: 9 Movie review 5 Review: Caste and religion are against humanity Rating: 8 Movie review 6 Review: Excellent Rating: 10 Movie review 7 Review: Best movie of the year/decade Rating: 10 Movie review 8 Review: A riveting piece of cinema that enthralls you throughout Rating: 9 Movie review 9 Review: Must whatch Rating: 10 Movie review 10 Review: Deep and touching Tamil Movie Rating: 9 Movie review 11 Review: Debut at the best!!! Rating: 10 Movie review 12 Review: such a masterpiece Rating: 7 Movie review 13 Review: Over hyped Rating: 5 Movie review 14 Review: I think this is as good as a film can get to real life and make an intense impact. Rating: 10 Movie review 15 Review: Watch in theatre for motivating good films Rating: 9 Movie review 16 Review: Best film in Indian Cinema Rating: 10 Movie review 17 Review: Pariyerum Perumal Rating: 10 Movie review 18 Review: Once in a blue moon kind of movie Rating: 10 Movie review 19 Review: Realistic Portrayal of Dark side of India. Rating: 9 Movie review 20 Review: Excellent anti-caste message through Pariyerum Perumal Rating: 9 Movie review 21 Review: A hard hitting, soulful movie based upon an undying love story. Rating: 10 Movie review 22 Review: Movie and story Excellent, Political commentary 👎👎👎👎 Rating: 6 Movie review 23 Review: Find someone who praises you in the same way Indians praise their movies Rating: 1 Movie review 24 Review: An eye opener for this generation Rating: 10 Movie review 25 Review: Perumal father scandalous scene Rating: 10 Movie review 26 Review: Apparently sharks do take things personally. Rating: 2 Movie review 27 Review: Dumb Dumb... Dumb-dumb-dumb-dumb! Rating: 2 Movie review 28 Review: "I've always wanted to make love to an angry welder." Rating: 2 Movie review 29 Movie review 30 Review: The worst Rating: 1 Movie review 31 Review: So bad, it's incredible... Rating: 1 Movie review 32 Review: This time it's personal! Rating: 1 Movie review 33 Review: An embarrassing film. Rating: 1 Movie review 34 Review: Arguably the worst film ever made Rating: 1 Movie review 35 Review: Utter claptrap... and that's being generous Rating: 1 Movie review 36 Review: Michael Caine's worst hour. Rating: 1 Movie review 37 Review: The roadhouse of Jaws movies 10 stars. Tropical fun Rating: 10 Movie review 38 Review: I love this movie and do not care Rating: 6 Movie review 39 Review: Pant-wettingly hilarious Rating: 1 Movie review 40 Movie review 41 Review: Jaw dropping Rating: 3 Movie review 42 Review: Forgive me lord for I have have sinned, I watched Jaws: The Revenge Rating: 1 Movie review 43 Review: A perfect example of so bad it's good entertainment Rating: 5 Movie review 44 Review: So there is a category beneath 'so bad it's good'... Rating: 1 Movie review 45 Review: Jaws 4, People 0 Rating: 1 Movie review 46 Review: The shark goes roar Rating: 2 Movie review 47 Review: Universal cash in on a mechanical fish that was already dead Rating: 3 Movie review 48 Review: Underrated. Second best in the series. *** out of 5 Rating: 10 Movie review 49 Review: Not Half as Bad as Most People Make it Out to Be Rating: 6 Movie review 50 Review: Could have been so much better Rating: 5 Movie review 51 Review: Good emotional drama spoilt by its reviews Rating: 7 Movie review 52 Review: If it doesn't arouse, is it art? ARUVI does Rating: 7 Movie review 53 Review: powerful stream that bashes up left and right Rating: 9 Movie review 54 Review: A lifetime Experience Rating: 10 Movie review 55 Review: Rolling, SIR!! Rating: 10 Movie review 56 Review: Life isn't black and white Rating: 10 Movie review 57 Review: Certain pain cannot be explained by just words but "Aruvi" does. Rating: 10 Movie review 58 Review: Best Tamil film of 2017 Rating: 10 Movie review 59 Review: My Review Rating: 10 Movie review 60 Review: Middle part is amazing; but overall, contrived and sentimental Rating: 6 Movie review 61 Review: Aruvi is sensational and emotional classic on sensitive topic. One of the most mind-shattering build up for any woman centric film recently. Rating: 8 Movie review 62 Review: half commercial half documentary Rating: 6 Movie review 63 Review: Misleading Rating Rating: 2 Movie review 64 Review: Masterpiece Rating: 10 Movie review 65 Review: Best fil with all Kinds of Senses Rating: 10 Movie review 66 Review: Best Social political film Rating: 9 Movie review 67 Review: An eclectic experience ! A MUST WATCH !! Rating: 9 Movie review 68 Review: Aruvi- Simple yet Powerful. Rating: 9 Movie review 69 Review: Waterfall of human mixed-emotions Rating: 10 Movie review 70 Review: Highly Overrated for an average plot Rating: 6 Movie review 71 Review: Emotional Experience Rating: 10 Movie review 72 Review: . Rating: 7 Movie review 73 Review: Good attempt at total social commentary, but ultimately flawed.. Rating: 5 Movie review 74 Review: I laughed, I cried Rating: 10 Movie review 75 Movie review 76 Review: Both uplifting and disheartening, sometimes both at once Rating: 10 Movie review 77 Review: The spirit of freedom vs. the spirit of legal-ism Rating: 9 Movie review 78 Review: Touching and moving, a great cinematic experience Rating: 10 Movie review 79 Review: "What an excellent movie" is all that went through my mind after seeing this masterpiece Rating: 10 Movie review 80 Review: A masterpiece Rating: 10 Movie review 81 Review: A great order vs. chaos tale that everyone can relate to Rating: 10 Movie review 82 Review: Poetic - Powerful - Simple: The Greatness of Cuckoo's Nest. Rating: 10 Movie review 83 Review: Jack Nicholson at his finest Rating: 9 Movie review 84 Movie review 85 Review: One of the greatest of all time. Rating: 10 Movie review 86 Review: Best film of its era Rating: 10 Movie review 87 Review: It's not that easy Rating: 8 Movie review 88 Review: Underwhelming... Rating: 6 Movie review 89 Review: "You're not a goddamn loony now!" Rating: 5 Movie review 90 Review: A perfect mixture of entertainment and drama. Rating: 9 Movie review 91 Review: Best movie ever Rating: 10 Movie review 92 Review: You have to be crazy to hate this movie! This movie is a masterpiece Rating: 9 Movie review 93 Review: One of the greatest movies and life lessons of all time Rating: 10 Movie review 94 Review: Jack Nicholson's best yet! Rating: 10 Movie review 95 Review: One Flew Over One of the Best Movies Ever Made!!! (And that person was me) Rating: 10 Movie review 96 Review: Extraordinary Rating: 10 Movie review 97 Movie review 98 Review: Way over-rated Rating: 6 Movie review 99 Review: the single-most shamelessy over rated movie of the 70s Rating: 3 Movie review 100 Review: The Ultimate Backfire Rating: 10 Movie review 101 Review: complete surprise Rating: 8 Movie review 102 Review: Aparichit(Anniyan)- the perfect entertainer! Rating: 9 Movie review 103 Review: sarva dharmath parithyajyam........... Rating: 8 Movie review 104 Review: The movie that will move you in every scene Rating: 10 Movie review 105 Review: Anniyan is what every movie wants to be: An entertainer Rating: 10 Movie review 106 Review: A-m-a-z-i-n-g-! Rating: 10 Movie review 107 Review: definitely a MUST-see!!!!! Rating: 10 Movie review 108 Review: Extraordinary viewing experience Rating: 10 Movie review 109 Review: Must watch film Rating: 10 Movie review 110 Review: What do you get when you mix a bit of Indian and Mudhalvan? Rating: 7 Movie review 111 Review: Shankar trend setter Rating: 9 Movie review 112 Review: Could have been way better Rating: 7 Movie review 113 Movie review 114 Review: Shankar, where art thou flame? Rating: 1 Movie review 115 Review: Total cinema.perfect.speechless.must watch.!!! Rating: 10 Movie review 116 Review: Mind-blowingggggggg👍 Rating: 10 Movie review 117 Review: Shankar's yet another chiller Rating: 9 Movie review 118 Review: Best movie Rating: 10 Movie review 119 Review: Anniyan (2005) Rating: 10 Movie review 120 Movie review 121 Review: Superb Thriller Rating: 9 Movie review 122 Review: The 27 crore question? Rating: 7 Movie review 123 Review: preview of anniyan movie Rating: 10 Movie review 124 Review: Don't expect anything outstanding and enjoy this Indian clone Rating: 7 Movie review 125 Review: Shankar's worst movie Rating: 5
In [12]:
movie.head()
Out[12]:
Review | Rating | |
---|---|---|
0 | Speaks humanity\n | 10 |
1 | The dark underbelly- Casteism\n | 10 |
2 | Deep intense movie, brilliantly narrated scri... | 10 |
3 | Another intense film after merku thodarchi ma... | 9 |
4 | Caste and religion are against humanity\n | 8 |
In [13]:
movie.shape
Out[13]:
(118, 2)
In [15]:
movie.to_csv('RianYan-2424072.csv', index=False)
Text Processing and Analysis
In [17]:
import string
import re
#import nltk
#nltk.download()
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.model_selection import train_test_split
In [18]:
textFeatures = movie['Review'].copy()
textFeatures.shape
Out[18]:
(118,)
In [19]:
#Preparing text for Wordcloud
text = []
for t in textFeatures:
text.append(t)
all_text = ', '.join(t for t in text)
#print(all_text)
print(len(all_text))
4638
In [22]:
from os import path
from PIL import Image
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
In [24]:
#Create stopword list
stopwords = set(STOPWORDS)
stopwords.update(["br", "im", "thats"]) #"im","lol","Xa","film"])
# Generate a word cloud image
wordcloud = WordCloud(stopwords=stopwords, background_color="white").generate(all_text)
# Display the image
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()
#save the generated image to a file
#wordcloud.to_file("wordcloud_cb_all.png")
Sentiment Identification using VADER
In [37]:
import nltk
from nltk.sentiment.vader import SentimentIntensityAnalyzer
downloaded vader lexicon after installing certificates and running nltk.download('vader_lexicon') on the shell as suggested on https://stackoverflow.com/a/42890688
In [38]:
sid = SentimentIntensityAnalyzer()
c=0
for t in text:
c+=1
print(c, t)
ss = sid.polarity_scores(t)
print(ss)
if(ss['compound'] >= 0.05):
print('positive')
elif(ss['compound'] <= -0.05):
print('negative')
else:
print('neutral')
print('\n')
1 Speaks humanity {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 2 The dark underbelly- Casteism {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 3 Deep intense movie, brilliantly narrated script. {'neg': 0.0, 'neu': 0.43, 'pos': 0.57, 'compound': 0.6486} 4 Another intense film after merku thodarchi malai. {'neg': 0.0, 'neu': 0.822, 'pos': 0.178, 'compound': 0.0772} 5 Caste and religion are against humanity {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 6 Excellent {'neg': 0.0, 'neu': 0.0, 'pos': 1.0, 'compound': 0.5719} 7 Best movie of the year/decade {'neg': 0.0, 'neu': 0.488, 'pos': 0.512, 'compound': 0.6369} 8 A riveting piece of cinema that enthralls you throughout {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 9 Must whatch {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 10 Deep and touching Tamil Movie {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 11 Debut at the best!!! {'neg': 0.0, 'neu': 0.371, 'pos': 0.629, 'compound': 0.7249} 12 such a masterpiece {'neg': 0.0, 'neu': 0.196, 'pos': 0.804, 'compound': 0.6249} 13 Over hyped {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 14 I think this is as good as a film can get to real life and make an intense impact. {'neg': 0.0, 'neu': 0.781, 'pos': 0.219, 'compound': 0.4939} 15 Watch in theatre for motivating good films {'neg': 0.0, 'neu': 0.45, 'pos': 0.55, 'compound': 0.7269} 16 Best film in Indian Cinema {'neg': 0.0, 'neu': 0.488, 'pos': 0.512, 'compound': 0.6369} 17 Pariyerum Perumal {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 18 Once in a blue moon kind of movie {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 19 Realistic Portrayal of Dark side of India. {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 20 Excellent anti-caste message through Pariyerum Perumal {'neg': 0.0, 'neu': 0.575, 'pos': 0.425, 'compound': 0.5719} 21 A hard hitting, soulful movie based upon an undying love story. {'neg': 0.103, 'neu': 0.588, 'pos': 0.309, 'compound': 0.5859} 22 Movie and story Excellent, Political commentary 👎👎👎👎 {'neg': 0.0, 'neu': 0.619, 'pos': 0.381, 'compound': 0.5719} 23 Find someone who praises you in the same way Indians praise their movies {'neg': 0.0, 'neu': 0.611, 'pos': 0.389, 'compound': 0.7906} 24 An eye opener for this generation {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 25 Perumal father scandalous scene {'neg': 0.531, 'neu': 0.469, 'pos': 0.0, 'compound': -0.5267} 26 Apparently sharks do take things personally. {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 27 Dumb Dumb... Dumb-dumb-dumb-dumb! {'neg': 0.642, 'neu': 0.358, 'pos': 0.0, 'compound': -0.5562} 28 "I've always wanted to make love to an angry welder." {'neg': 0.213, 'neu': 0.516, 'pos': 0.271, 'compound': 0.2263} 29 The worst {'neg': 0.804, 'neu': 0.196, 'pos': 0.0, 'compound': -0.6249} 30 So bad, it's incredible... {'neg': 0.558, 'neu': 0.442, 'pos': 0.0, 'compound': -0.5849} 31 This time it's personal! {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 32 An embarrassing film. {'neg': 0.565, 'neu': 0.435, 'pos': 0.0, 'compound': -0.3818} 33 Arguably the worst film ever made {'neg': 0.604, 'neu': 0.396, 'pos': 0.0, 'compound': -0.7269} 34 Utter claptrap... and that's being generous {'neg': 0.0, 'neu': 0.602, 'pos': 0.398, 'compound': 0.5106} 35 Michael Caine's worst hour. {'neg': 0.577, 'neu': 0.423, 'pos': 0.0, 'compound': -0.6249} 36 The roadhouse of Jaws movies 10 stars. Tropical fun {'neg': 0.0, 'neu': 0.708, 'pos': 0.292, 'compound': 0.5106} 37 I love this movie and do not care {'neg': 0.222, 'neu': 0.423, 'pos': 0.355, 'compound': 0.3761} 38 Pant-wettingly hilarious {'neg': 0.0, 'neu': 0.27, 'pos': 0.73, 'compound': 0.4019} 39 Jaw dropping {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 40 Forgive me lord for I have have sinned, I watched Jaws: The Revenge {'neg': 0.234, 'neu': 0.621, 'pos': 0.145, 'compound': -0.3182} 41 A perfect example of so bad it's good entertainment {'neg': 0.247, 'neu': 0.22, 'pos': 0.532, 'compound': 0.6337} 42 So there is a category beneath 'so bad it's good'... {'neg': 0.36, 'neu': 0.64, 'pos': 0.0, 'compound': -0.6696} 43 Jaws 4, People 0 {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 44 The shark goes roar {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 45 Universal cash in on a mechanical fish that was already dead {'neg': 0.323, 'neu': 0.677, 'pos': 0.0, 'compound': -0.6486} 46 Underrated. Second best in the series. *** out of 5 {'neg': 0.0, 'neu': 0.656, 'pos': 0.344, 'compound': 0.6369} 47 Not Half as Bad as Most People Make it Out to Be {'neg': 0.0, 'neu': 0.794, 'pos': 0.206, 'compound': 0.431} 48 Could have been so much better {'neg': 0.0, 'neu': 0.611, 'pos': 0.389, 'compound': 0.4902} 49 Good emotional drama spoilt by its reviews {'neg': 0.0, 'neu': 0.526, 'pos': 0.474, 'compound': 0.5423} 50 If it doesn't arouse, is it art? ARUVI does {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 51 powerful stream that bashes up left and right {'neg': 0.0, 'neu': 0.714, 'pos': 0.286, 'compound': 0.4215} 52 A lifetime Experience {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 53 Rolling, SIR!! {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 54 Life isn't black and white {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 55 Certain pain cannot be explained by just words but "Aruvi" does. {'neg': 0.169, 'neu': 0.709, 'pos': 0.122, 'compound': -0.1531} 56 Best Tamil film of 2017 {'neg': 0.0, 'neu': 0.488, 'pos': 0.512, 'compound': 0.6369} 57 My Review {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 58 Middle part is amazing; but overall, contrived and sentimental {'neg': 0.0, 'neu': 0.567, 'pos': 0.433, 'compound': 0.6542} 59 Aruvi is sensational and emotional classic on sensitive topic. One of the most mind-shattering build up for any woman centric film recently. {'neg': 0.0, 'neu': 0.929, 'pos': 0.071, 'compound': 0.1531} 60 half commercial half documentary {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 61 Misleading Rating {'neg': 0.73, 'neu': 0.27, 'pos': 0.0, 'compound': -0.4019} 62 Masterpiece {'neg': 0.0, 'neu': 0.0, 'pos': 1.0, 'compound': 0.6249} 63 Best fil with all Kinds of Senses {'neg': 0.0, 'neu': 0.588, 'pos': 0.412, 'compound': 0.6369} 64 Best Social political film {'neg': 0.0, 'neu': 0.417, 'pos': 0.583, 'compound': 0.6369} 65 An eclectic experience ! A MUST WATCH !! {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 66 Aruvi- Simple yet Powerful. {'neg': 0.0, 'neu': 0.517, 'pos': 0.483, 'compound': 0.4215} 67 Waterfall of human mixed-emotions {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 68 Highly Overrated for an average plot {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 69 Emotional Experience {'neg': 0.0, 'neu': 0.385, 'pos': 0.615, 'compound': 0.1531} 70 . {'neg': 0.0, 'neu': 0.0, 'pos': 0.0, 'compound': 0.0} 71 Good attempt at total social commentary, but ultimately flawed.. {'neg': 0.0, 'neu': 0.804, 'pos': 0.196, 'compound': 0.2382} 72 I laughed, I cried {'neg': 0.464, 'neu': 0.0, 'pos': 0.536, 'compound': 0.1027} 73 Both uplifting and disheartening, sometimes both at once {'neg': 0.286, 'neu': 0.714, 'pos': 0.0, 'compound': -0.4215} 74 The spirit of freedom vs. the spirit of legal-ism {'neg': 0.0, 'neu': 0.441, 'pos': 0.559, 'compound': 0.765} 75 Touching and moving, a great cinematic experience {'neg': 0.0, 'neu': 0.549, 'pos': 0.451, 'compound': 0.6249} 76 "What an excellent movie" is all that went through my mind after seeing this masterpiece {'neg': 0.0, 'neu': 0.603, 'pos': 0.397, 'compound': 0.8616} 77 A masterpiece {'neg': 0.0, 'neu': 0.0, 'pos': 1.0, 'compound': 0.6249} 78 A great order vs. chaos tale that everyone can relate to {'neg': 0.234, 'neu': 0.506, 'pos': 0.259, 'compound': 0.1027} 79 Poetic - Powerful - Simple: The Greatness of Cuckoo's Nest. {'neg': 0.0, 'neu': 0.714, 'pos': 0.286, 'compound': 0.4215} 80 Jack Nicholson at his finest {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 81 One of the greatest of all time. {'neg': 0.0, 'neu': 0.588, 'pos': 0.412, 'compound': 0.6369} 82 Best film of its era {'neg': 0.0, 'neu': 0.488, 'pos': 0.512, 'compound': 0.6369} 83 It's not that easy {'neg': 0.445, 'neu': 0.555, 'pos': 0.0, 'compound': -0.3412} 84 Underwhelming... {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 85 "You're not a goddamn loony now!" {'neg': 0.0, 'neu': 0.584, 'pos': 0.416, 'compound': 0.4303} 86 A perfect mixture of entertainment and drama. {'neg': 0.0, 'neu': 0.381, 'pos': 0.619, 'compound': 0.7579} 87 Best movie ever {'neg': 0.0, 'neu': 0.323, 'pos': 0.677, 'compound': 0.6369} 88 You have to be crazy to hate this movie! This movie is a masterpiece {'neg': 0.312, 'neu': 0.488, 'pos': 0.2, 'compound': -0.3164} 89 One of the greatest movies and life lessons of all time {'neg': 0.0, 'neu': 0.704, 'pos': 0.296, 'compound': 0.6369} 90 Jack Nicholson's best yet! {'neg': 0.0, 'neu': 0.4, 'pos': 0.6, 'compound': 0.6696} 91 One Flew Over One of the Best Movies Ever Made!!! (And that person was me) {'neg': 0.0, 'neu': 0.734, 'pos': 0.266, 'compound': 0.7249} 92 Extraordinary {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 93 Way over-rated {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 94 the single-most shamelessy over rated movie of the 70s {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 95 The Ultimate Backfire {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 96 complete surprise {'neg': 0.0, 'neu': 0.323, 'pos': 0.677, 'compound': 0.2732} 97 Aparichit(Anniyan)- the perfect entertainer! {'neg': 0.0, 'neu': 0.233, 'pos': 0.767, 'compound': 0.7644} 98 sarva dharmath parithyajyam........... {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 99 The movie that will move you in every scene {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 100 Anniyan is what every movie wants to be: An entertainer {'neg': 0.0, 'neu': 0.776, 'pos': 0.224, 'compound': 0.3818} 101 A-m-a-z-i-n-g-! {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 102 definitely a MUST-see!!!!! {'neg': 0.0, 'neu': 0.205, 'pos': 0.795, 'compound': 0.5951} 103 Extraordinary viewing experience {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 104 Must watch film {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 105 What do you get when you mix a bit of Indian and Mudhalvan? {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 106 Shankar trend setter {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 107 Could have been way better {'neg': 0.0, 'neu': 0.58, 'pos': 0.42, 'compound': 0.4404} 108 Shankar, where art thou flame? {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 109 Total cinema.perfect.speechless.must watch.!!! {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 110 Mind-blowingggggggg👍 {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 111 Shankar's yet another chiller {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 112 Best movie {'neg': 0.0, 'neu': 0.192, 'pos': 0.808, 'compound': 0.6369} 113 Anniyan (2005) {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 114 Superb Thriller {'neg': 0.0, 'neu': 0.0, 'pos': 1.0, 'compound': 0.6705} 115 The 27 crore question? {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 116 preview of anniyan movie {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0} 117 Don't expect anything outstanding and enjoy this Indian clone {'neg': 0.24, 'neu': 0.522, 'pos': 0.238, 'compound': -0.0052} 118 Shankar's worst movie {'neg': 0.672, 'neu': 0.328, 'pos': 0.0, 'compound': -0.6249} negative
Sentiment Classification using Machine Learning
Prepare 'Truth Set'
In [40]:
label = []
for r in movie['Rating']:
r = int(r)
if (r>5):
label.append('1') #Positive
elif(r<5):
label.append('-1') #Negative
elif(r==5):
label.append('0') #Netural
movie['class-label'] = label
In [41]:
movie['class-label'].value_counts()
Out[41]:
class-label 1 91 -1 21 0 6 Name: count, dtype: int64
In [42]:
movie = movie[movie['class-label']!='0']
In [43]:
movie['class-label'].value_counts()
Out[43]:
class-label 1 91 -1 21 Name: count, dtype: int64
In [44]:
textFeatures = movie['Review'].copy()
textFeatures.shape
Out[44]:
(112,)
In [53]:
import nltk
#nltk.download('punkt')
# Stemming using TextBlob library for stemming
from textblob import TextBlob
In [56]:
def textblob_tokenizer(input_str):
blob = TextBlob(input_str.lower())
tokens = blob.words
words = [token.stem() for token in tokens]
return words
In [58]:
#Toy example:
print(textblob_tokenizer('Q: studed studing!!! I miss uuuu! It's'))
['q', 'stude', 'stude', 'i', 'miss', 'uuuu', 'it', '039', 's']
In [59]:
#countvectorizer convers each review into a vector based on the word count.
countvectorizer = CountVectorizer(analyzer= 'word', stop_words= 'english',
tokenizer=textblob_tokenizer)
#convers text into a vector based on tf-idf weighting scheme.
tfidfvectorizer = TfidfVectorizer(analyzer= 'word', stop_words= 'english',
tokenizer=textblob_tokenizer)
In [60]:
#textFeatures
In [61]:
count_matrix = countvectorizer.fit_transform(textFeatures)
tfidf_matrix = tfidfvectorizer.fit_transform(textFeatures)
In [62]:
#print(tfidf_matrix) #print elements of the matrix.
In [63]:
print(tfidf_matrix.shape)
print(count_matrix.shape)
(112, 264) (112, 264)
Build ML models
In [65]:
features_train, features_test, labels_train, labels_test = train_test_split(tfidf_matrix, movie['class-label'], test_size=0.3,random_state=8)
print(features_train.shape, features_test.shape, labels_train.shape, labels_test.shape)
(78, 264) (34, 264) (78,) (34,)
In [66]:
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.metrics import accuracy_score
In [68]:
#SVM classifier
from sklearn.svm import SVC
print("\nEvaluation for SVM \n")
svc = SVC(kernel='sigmoid', gamma=1.0)
svc.fit(features_train, labels_train)
prediction = svc.predict(features_test)
acc = accuracy_score(labels_test,prediction)
print('Accuracy:',acc)
Evaluation for SVM Accuracy: 0.8529411764705882
In [69]:
from sklearn.metrics import precision_score
prec = precision_score(labels_test,prediction, average='weighted')
print('Precision:',prec)
Precision: 0.7275086505190311
In [71]:
from sklearn.metrics import recall_score
recall = recall_score(labels_test,prediction, average='weighted')
print('Recall:',recall)
Recall: 0.8529411764705882
In [72]:
from sklearn.metrics import f1_score
f1 = f1_score(labels_test,prediction, average='weighted')
print('F-1 measure: ', f1)
print('\nConfusion Matrix:\n')
print(confusion_matrix(labels_test, prediction))
print(classification_report(labels_test, prediction))
print(prediction)
F-1 measure: 0.7852474323062557 Confusion Matrix: [[ 0 5] [ 0 29]] precision recall f1-score support -1 0.00 0.00 0.00 5 1 0.85 1.00 0.92 29 accuracy 0.85 34 macro avg 0.43 0.50 0.46 34 weighted avg 0.73 0.85 0.79 34 ['1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1' '1']
In [73]:
#Decision Tree
print("\nEvaluation for Decision Tree \n")
from sklearn.tree import DecisionTreeClassifier
dtree = DecisionTreeClassifier()
dtree.fit(features_train, labels_train)
prediction = dtree.predict(features_test)
acc = accuracy_score(labels_test,prediction)
print('Accuracy: ',acc)
prec = precision_score(labels_test,prediction, average='weighted')
print('Precision: ',prec)
recall = recall_score(labels_test,prediction, average='weighted')
print('Recall: ',recall)
f1 = f1_score(labels_test,prediction, average='weighted')
print('F-1 measure: ',f1)
print('\nConfusion Matrix:\n')
print(confusion_matrix(labels_test, prediction))
print(classification_report(labels_test, prediction))
Evaluation for Decision Tree Accuracy: 0.8529411764705882 Precision: 0.8411764705882353 Recall: 0.8529411764705882 F-1 measure: 0.8460175030464162 Confusion Matrix: [[ 2 3] [ 2 27]] precision recall f1-score support -1 0.50 0.40 0.44 5 1 0.90 0.93 0.92 29 accuracy 0.85 34 macro avg 0.70 0.67 0.68 34 weighted avg 0.84 0.85 0.85 34
In [ ]: