Data Parallel in Spark
Contents
Data Parallel in Spark#
Download Movie Dataset
Unzip the movie data file.
import os
import sys
spark_path = os.environ['SPARK_HOME']
sys.path.append(spark_path + "/bin")
sys.path.append(spark_path + "/python")
sys.path.append(spark_path + "/python/pyspark/")
sys.path.append(spark_path + "/python/lib")
sys.path.append(spark_path + "/python/lib/pyspark.zip")
sys.path.append(spark_path + "/python/lib/py4j-0.10.9-src.zip")
import findspark
findspark.init()
import pyspark
number_cores = 8
memory_gb = 12
conf = (pyspark.SparkConf().
setMaster('local[{}]'.format(number_cores)).
set('spark.driver.memory', '{}g'.format(memory_gb)))
sc = pyspark.SparkContext(conf=conf)
# You need to provide either an absolute or relative path to your data location here.
# This notation is for Windows DOS commands
!dir e:\data\ml-latest\
Volume in drive E is WAREHOUSE
Volume Serial Number is 6077-E9E2
Directory of e:\data\ml-latest
09/11/2022 10:25 PM <DIR> .
09/11/2022 10:24 PM <DIR> ..
09/26/2018 05:44 PM 414,851,573 genome-scores.csv
09/26/2018 05:44 PM 18,103 genome-tags.csv
09/26/2018 05:37 PM 1,267,039 links.csv
09/26/2018 05:35 PM 2,858,223 movies.csv
09/26/2018 05:31 PM 759,200,511 ratings.csv
09/26/2018 05:46 PM 9,784 README.txt
09/26/2018 05:11 PM 39,744,990 tags.csv
7 File(s) 1,217,950,223 bytes
2 Dir(s) 1,106,383,896,576 bytes free
ratings = sc.textFile("e:/data/ml-latest/ratings.csv")
ratings.take(5)
['userId,movieId,rating,timestamp',
'1,307,3.5,1256677221',
'1,481,3.5,1256677456',
'1,1091,1.5,1256677471',
'1,1257,4.5,1256677460']
The cache function tells Spark to keep the data in memory (after an action has been invoked)
ratings.cache()
e:/data/ml-latest/ratings.csv MapPartitionsRDD[1] at textFile at NativeMethodAccessorImpl.java:0
%%time
ratings.count()
Wall time: 20.4 s
27753445
%%time
ratings.count()
Wall time: 18.2 s
27753445
%%time
ratings.count()
Wall time: 17.6 s
27753445
Average ratings#
Finding the average ratings over the years of each individual movies
Procedural:
For each movie
add together all rating values,
count all rating entries
divide sum of rating values by count of rating entries
Data flow:
Backtracking from the result:
Individual average ratings of all movies
Average rating of one movie
Collections of all ratings for that movie
Sum
Count
List of all ratings
ratingHeader = ratings.first()
print(ratingHeader)
userId,movieId,rating,timestamp
ratingsOnly = ratings.filter(lambda x: x != ratingHeader)
ratingsOnly.take(5)
['1,307,3.5,1256677221',
'1,481,3.5,1256677456',
'1,1091,1.5,1256677471',
'1,1257,4.5,1256677460',
'1,1449,4.5,1256677264']
s = '1,307,3.5,1256677221'
s.split(",")[2]
'3.5'
movieRatings = ratingsOnly.map(lambda line: (line.split(",")[1], float(line.split(",")[2])))
movieRatings.take(5)
[('307', 3.5), ('481', 3.5), ('1091', 1.5), ('1257', 4.5), ('1449', 4.5)]
Possible approaches in aggregating data#
groupByKeyandmapValuesreduceByKeyandcountByKey
groupByKey and mapValues
groupByKeyRatings = movieRatings.groupByKey()
groupByKeyRatings.take(5)
[('3826', <pyspark.resultiterable.ResultIterable at 0x24ed82cdef0>),
('104', <pyspark.resultiterable.ResultIterable at 0x24ed82cd438>),
('153', <pyspark.resultiterable.ResultIterable at 0x24ed82cdf98>),
('165', <pyspark.resultiterable.ResultIterable at 0x24ed82cd208>),
('181', <pyspark.resultiterable.ResultIterable at 0x24ed82ed7f0>)]
mapValuesToListRatings = groupByKeyRatings.mapValues(list)
mapValuesToListRatings.take(5)
[('3826',
[2.0,
3.0,
3.0,
3.0,
4.0,
3.0,
1.0,
3.0,
3.0,
1.0,
2.0,
2.0,
3.0,
2.0,
3.0,
3.0,
2.0,
2.0,
2.0,
0.5,
3.0,
3.0,
1.0,
1.5,
2.0,
4.0,
2.0,
3.0,
1.0,
4.0,
3.5,
2.0,
2.5,
2.0,
3.5,
2.0,
2.5,
2.0,
3.5,
3.0,
0.5,
1.5,
1.5,
1.0,
1.0,
2.5,
1.5,
1.0,
4.0,
0.5,
2.0,
2.5,
3.5,
2.0,
0.5,
1.0,
2.0,
2.0,
2.0,
2.5,
2.0,
2.0,
3.0,
3.0,
3.5,
3.5,
2.0,
3.0,
0.5,
2.0,
1.0,
2.0,
2.5,
2.5,
1.0,
1.5,
3.0,
3.0,
3.0,
2.0,
2.5,
3.0,
3.5,
2.0,
4.0,
0.5,
4.0,
2.5,
3.0,
4.5,
4.0,
2.5,
2.0,
3.0,
2.5,
3.0,
2.5,
2.0,
2.5,
2.0,
2.0,
2.0,
5.0,
4.0,
3.0,
2.0,
1.0,
3.0,
2.5,
1.0,
1.5,
2.5,
1.5,
1.0,
3.0,
3.0,
4.0,
3.0,
3.0,
3.0,
2.5,
4.5,
2.5,
3.0,
2.0,
0.5,
4.5,
2.0,
3.0,
3.5,
3.0,
2.0,
2.0,
1.0,
3.5,
2.5,
3.5,
4.5,
2.0,
3.0,
2.5,
1.5,
1.0,
2.5,
2.5,
3.0,
2.0,
2.5,
2.0,
4.0,
2.0,
3.0,
2.0,
3.5,
1.5,
0.5,
3.0,
1.5,
3.0,
4.0,
1.5,
4.0,
3.0,
1.0,
1.0,
0.5,
2.0,
4.0,
3.5,
2.5,
2.0,
3.0,
3.5,
4.0,
1.0,
3.0,
3.0,
1.0,
1.0,
3.0,
2.5,
2.5,
2.5,
1.0,
4.0,
3.5,
3.5,
3.0,
3.0,
2.0,
3.0,
1.5,
4.0,
3.0,
3.5,
3.0,
2.0,
3.0,
3.0,
3.0,
3.0,
2.5,
2.5,
2.0,
1.0,
0.5,
3.0,
3.5,
3.0,
4.0,
2.0,
3.0,
2.0,
3.5,
1.5,
3.0,
1.5,
1.0,
3.5,
2.0,
2.5,
2.0,
5.0,
4.5,
3.0,
2.5,
3.5,
1.5,
1.5,
2.0,
2.5,
3.0,
2.0,
1.0,
2.0,
1.0,
2.0,
1.0,
2.5,
3.0,
2.0,
3.0,
2.5,
1.0,
2.5,
1.5,
3.5,
3.5,
2.5,
2.0,
1.0,
3.5,
2.0,
1.0,
5.0,
2.5,
2.0,
1.5,
2.0,
1.0,
3.5,
4.0,
0.5,
2.0,
3.0,
1.0,
3.5,
2.5,
4.0,
5.0,
3.5,
1.5,
1.0,
2.0,
2.0,
2.0,
3.5,
2.5,
3.0,
3.5,
5.0,
3.0,
2.5,
4.0,
2.0,
0.5,
4.0,
1.5,
3.0,
4.0,
3.0,
3.5,
2.0,
3.5,
3.5,
2.0,
3.0,
3.0,
1.0,
5.0,
3.0,
2.0,
2.5,
3.0,
2.5,
4.0,
2.0,
3.0,
3.5,
3.0,
4.5,
3.0,
1.0,
4.0,
3.0,
2.0,
4.0,
3.5,
1.0,
2.5,
3.5,
2.0,
2.0,
1.0,
2.5,
2.0,
1.0,
2.5,
3.5,
3.0,
3.0,
1.0,
2.5,
4.0,
2.5,
2.0,
3.5,
3.0,
4.0,
4.0,
4.0,
1.0,
4.5,
4.0,
1.5,
3.0,
2.5,
3.0,
2.0,
2.0,
1.5,
3.0,
3.0,
3.0,
2.0,
3.0,
0.5,
3.5,
4.0,
1.5,
2.5,
2.5,
3.5,
1.0,
2.5,
3.5,
1.0,
0.5,
4.0,
2.5,
2.5,
1.5,
2.0,
1.5,
3.0,
3.0,
3.0,
1.5,
3.0,
3.0,
0.5,
2.5,
3.0,
4.5,
2.0,
3.0,
2.5,
4.0,
2.5,
4.0,
3.0,
2.0,
2.0,
2.0,
3.5,
4.0,
3.0,
3.0,
3.5,
2.0,
3.5,
3.5,
2.0,
2.5,
3.0,
3.0,
3.0,
1.0,
4.0,
3.0,
1.0,
5.0,
3.0,
3.5,
2.5,
2.0,
4.0,
4.0,
2.0,
4.0,
2.5,
2.0,
2.0,
2.0,
2.5,
1.5,
4.0,
1.5,
2.0,
2.0,
5.0,
4.0,
1.0,
1.0,
3.0,
4.0,
2.0,
3.0,
2.5,
3.0,
3.5,
4.0,
4.0,
3.5,
2.0,
4.0,
4.0,
2.5,
2.5,
3.0,
2.5,
2.0,
1.0,
3.5,
0.5,
2.0,
4.0,
3.5,
2.5,
1.5,
2.0,
3.5,
2.0,
3.0,
3.0,
3.0,
3.5,
1.5,
2.0,
2.0,
2.5,
1.0,
3.0,
4.0,
1.5,
3.0,
2.0,
2.5,
3.0,
2.5,
2.0,
2.0,
2.5,
4.0,
2.5,
2.5,
2.5,
2.0,
3.0,
3.0,
3.0,
3.5,
1.5,
2.0,
2.0,
3.0,
4.0,
3.5,
3.0,
3.0,
2.0,
3.0,
2.0,
3.0,
3.5,
3.0,
3.0,
2.0,
3.0,
2.0,
0.5,
2.0,
2.5,
2.0,
2.5,
2.0,
0.5,
2.0,
4.0,
1.5,
3.0,
4.0,
2.0,
3.0,
3.5,
2.5,
2.0,
2.5,
0.5,
3.0,
2.5,
3.0,
3.0,
2.5,
3.0,
3.0,
2.5,
3.0,
2.0,
2.0,
2.0,
2.0,
2.0,
2.5,
2.5,
2.5,
3.0,
2.0,
3.0,
1.0,
2.0,
3.5,
2.5,
2.5,
3.0,
3.5,
1.5,
1.5,
3.0,
1.0,
3.0,
1.0,
0.5,
2.0,
0.5,
3.5,
4.0,
2.0,
0.5,
2.0,
2.0,
1.5,
2.0,
4.0,
2.5,
4.5,
1.5,
2.0,
4.0,
2.0,
2.0,
3.0,
3.5,
1.0,
1.0,
3.5,
1.0,
3.0,
3.0,
1.0,
1.5,
2.5,
1.0,
2.5,
4.0,
1.5,
3.5,
0.5,
2.0,
2.5,
2.0,
2.5,
3.0,
2.5,
2.0,
0.5,
3.0,
1.0,
3.0,
2.5,
3.0,
2.0,
4.0,
4.0,
3.0,
1.0,
4.0,
3.0,
4.5,
3.5,
2.0,
2.5,
3.0,
3.0,
1.5,
4.0,
0.5,
3.0,
3.0,
2.5,
3.0,
3.0,
3.0,
2.5,
3.0,
5.0,
2.0,
2.0,
1.0,
2.0,
1.0,
2.0,
1.0,
2.0,
2.0,
2.0,
2.5,
3.0,
3.0,
0.5,
3.0,
2.0,
2.0,
0.5,
3.0,
4.0,
1.5,
4.0,
2.0,
2.5,
1.0,
1.0,
2.0,
1.0,
3.5,
3.5,
3.0,
3.0,
1.0,
3.0,
3.0,
2.0,
3.0,
4.0,
3.0,
3.0,
3.0,
2.0,
1.0,
4.0,
2.5,
2.0,
4.0,
3.0,
3.5,
2.5,
4.0,
2.5,
1.5,
2.0,
2.0,
3.5,
1.0,
4.0,
4.0,
3.0,
0.5,
5.0,
2.0,
4.5,
2.5,
3.0,
2.0,
3.0,
4.0,
3.5,
1.5,
1.5,
1.0,
2.0,
2.5,
3.5,
3.0,
4.0,
2.5,
3.0,
1.5,
1.0,
3.5,
2.0,
3.0,
3.0,
3.0,
2.0,
1.0,
1.5,
2.0,
3.5,
2.0,
0.5,
3.0,
2.5,
2.5,
2.0,
2.5,
2.5,
1.5,
2.0,
0.5,
2.0,
1.0,
3.5,
3.0,
2.5,
4.0,
2.5,
2.5,
3.5,
1.5,
4.0,
1.5,
1.0,
3.5,
3.0,
2.5,
2.5,
2.0,
5.0,
4.0,
1.5,
4.5,
2.5,
1.0,
1.5,
3.0,
3.0,
3.0,
3.0,
2.0,
2.0,
2.0,
3.0,
3.0,
3.0,
2.5,
4.0,
1.5,
1.5,
2.0,
2.0,
3.0,
0.5,
2.0,
1.5,
2.0,
4.0,
2.0,
3.5,
3.5,
3.5,
2.5,
0.5,
5.0,
1.0,
3.0,
2.5,
2.0,
1.5,
1.0,
4.0,
1.5,
5.0,
2.0,
3.5,
1.0,
2.5,
3.0,
2.5,
3.0,
2.0,
3.0,
2.0,
4.0,
3.0,
3.5,
4.0,
1.0,
3.5,
3.0,
3.5,
4.0,
3.0,
3.5,
3.0,
3.0,
2.5,
3.0,
2.5,
1.5,
1.0,
1.5,
2.0,
2.5,
4.0,
3.5,
2.0,
4.0,
3.0,
3.0,
1.5,
1.5,
4.0,
2.5,
2.0,
2.0,
1.0,
2.0,
1.5,
2.0,
3.0,
2.5,
4.0,
1.5,
3.5,
4.5,
2.0,
4.0,
3.0,
2.0,
2.0,
3.0,
1.5,
2.0,
4.0,
3.5,
4.0,
3.0,
1.0,
0.5,
5.0,
3.0,
2.0,
3.5,
0.5,
3.5,
4.0,
1.0,
2.0,
3.0,
1.5,
2.0,
3.0,
0.5,
2.5,
2.0,
0.5,
3.5,
3.0,
2.0,
1.0,
0.5,
2.0,
2.5,
2.5,
2.5,
3.0,
2.0,
4.0,
3.0,
1.0,
3.0,
2.0,
4.0,
3.0,
1.5,
3.5,
1.0,
4.0,
3.5,
1.0,
3.0,
2.0,
4.0,
3.5,
2.0,
3.0,
3.0,
2.0,
3.0,
2.0,
2.0,
2.5,
2.0,
3.5,
3.0,
5.0,
1.0,
3.0,
5.0,
1.0,
2.0,
1.5,
3.0,
1.5,
1.0,
2.5,
3.5,
3.0,
1.5,
4.0,
1.0,
3.0,
3.0,
1.0,
2.0,
2.5,
3.0,
2.0,
3.0,
1.0,
2.0,
3.0,
4.0,
2.0,
4.5,
4.0,
3.5,
3.0,
1.5,
1.5,
4.5,
4.0,
3.0,
3.0,
2.0,
3.0,
1.5,
4.0,
2.5,
4.0,
1.0,
0.5,
3.0,
3.5,
3.0,
3.0,
3.0,
3.0,
3.0,
1.0,
3.0,
2.0,
3.5,
1.0,
3.0,
2.0,
2.5,
1.5,
3.0,
2.0,
1.5,
4.0,
3.0,
4.0,
1.0,
2.0,
3.5,
3.0,
3.0,
1.0,
...]),
('104',
[3.0,
4.0,
2.0,
2.5,
3.5,
4.0,
2.0,
3.5,
3.0,
4.0,
5.0,
4.0,
4.0,
5.0,
3.0,
3.5,
4.0,
3.0,
3.0,
4.0,
4.0,
1.0,
3.5,
3.0,
3.5,
1.0,
5.0,
4.0,
4.0,
4.5,
3.0,
4.0,
1.0,
3.0,
2.0,
4.5,
3.0,
3.0,
4.0,
1.0,
3.0,
2.5,
2.5,
5.0,
4.0,
5.0,
1.5,
4.0,
3.0,
2.0,
3.0,
4.0,
5.0,
3.0,
2.5,
4.0,
5.0,
2.0,
4.0,
4.0,
4.0,
4.5,
3.5,
3.5,
2.5,
3.0,
4.0,
4.0,
5.0,
4.5,
0.5,
3.5,
5.0,
3.0,
4.0,
5.0,
1.0,
4.0,
5.0,
3.0,
3.0,
3.0,
4.0,
3.5,
3.5,
1.5,
2.0,
3.5,
4.0,
5.0,
4.0,
3.5,
2.0,
4.0,
3.0,
3.0,
2.0,
4.0,
4.0,
3.5,
2.0,
4.5,
4.0,
2.0,
2.0,
2.0,
4.0,
1.0,
2.5,
5.0,
3.5,
3.0,
3.5,
3.0,
3.0,
3.0,
3.0,
4.0,
3.0,
5.0,
5.0,
2.0,
3.5,
4.0,
2.5,
3.5,
2.5,
3.5,
4.0,
2.0,
3.5,
5.0,
4.0,
3.5,
4.5,
4.0,
4.0,
3.0,
3.0,
5.0,
2.0,
4.0,
3.5,
2.5,
5.0,
3.5,
3.0,
4.5,
4.0,
3.5,
3.0,
4.0,
3.0,
5.0,
3.0,
2.0,
4.0,
3.0,
4.0,
3.0,
3.0,
5.0,
1.5,
3.0,
4.0,
1.0,
1.5,
3.0,
3.0,
2.5,
5.0,
4.0,
4.5,
3.5,
4.5,
4.0,
4.0,
3.0,
3.0,
2.0,
4.0,
2.0,
1.0,
3.0,
5.0,
4.0,
3.5,
2.5,
4.0,
2.5,
4.0,
3.0,
3.0,
4.0,
4.5,
3.5,
3.0,
3.0,
3.0,
4.0,
2.0,
4.0,
5.0,
0.5,
1.0,
0.5,
3.0,
3.5,
3.0,
4.0,
3.5,
5.0,
4.0,
4.0,
3.0,
4.0,
5.0,
0.5,
3.5,
5.0,
3.0,
4.0,
4.0,
4.0,
4.0,
2.5,
3.0,
3.0,
3.0,
4.5,
1.0,
5.0,
3.0,
5.0,
3.5,
3.0,
5.0,
3.0,
4.5,
4.5,
3.0,
3.0,
3.0,
4.5,
4.0,
3.0,
2.0,
4.0,
4.0,
4.0,
3.0,
5.0,
4.5,
1.0,
3.0,
3.5,
4.0,
4.5,
3.0,
4.0,
3.5,
0.5,
3.0,
4.0,
4.0,
4.0,
4.5,
1.0,
4.0,
3.5,
3.0,
3.0,
1.5,
5.0,
5.0,
4.5,
5.0,
5.0,
2.5,
5.0,
4.5,
4.0,
2.5,
4.0,
4.0,
3.0,
4.0,
4.0,
3.0,
2.0,
4.5,
4.0,
4.0,
4.0,
3.5,
4.0,
1.0,
3.0,
0.5,
4.5,
4.0,
3.0,
5.0,
3.0,
5.0,
1.0,
5.0,
3.0,
4.0,
3.0,
3.5,
3.5,
4.0,
3.5,
5.0,
5.0,
3.5,
2.5,
3.0,
2.0,
4.0,
1.0,
3.0,
4.0,
3.5,
5.0,
4.0,
2.0,
4.0,
3.0,
5.0,
2.0,
2.5,
4.0,
3.5,
3.0,
2.0,
3.0,
5.0,
2.0,
3.0,
5.0,
5.0,
3.5,
4.0,
4.0,
5.0,
4.5,
3.0,
4.5,
1.0,
3.0,
5.0,
5.0,
3.0,
5.0,
3.0,
4.0,
4.0,
5.0,
4.0,
4.0,
2.5,
5.0,
4.0,
3.5,
3.0,
3.5,
3.0,
1.5,
5.0,
3.0,
3.0,
4.0,
3.0,
3.5,
3.0,
5.0,
3.0,
5.0,
3.0,
3.0,
3.5,
4.0,
2.0,
4.0,
5.0,
4.5,
3.0,
3.0,
2.0,
3.5,
0.5,
4.0,
3.5,
4.0,
3.5,
1.0,
4.0,
3.5,
5.0,
4.0,
5.0,
2.0,
2.5,
3.5,
2.5,
3.0,
3.0,
5.0,
0.5,
3.5,
3.5,
3.0,
3.5,
4.0,
2.0,
1.0,
2.5,
3.0,
3.0,
4.5,
3.0,
3.0,
5.0,
1.0,
2.5,
4.5,
5.0,
1.5,
3.0,
3.0,
4.0,
3.5,
4.0,
3.0,
5.0,
5.0,
4.0,
3.0,
2.0,
5.0,
5.0,
3.0,
3.5,
3.0,
3.5,
3.0,
3.0,
2.5,
3.5,
3.0,
3.5,
4.5,
2.0,
4.0,
4.0,
3.0,
3.0,
4.0,
4.5,
3.0,
3.0,
4.0,
5.0,
3.5,
4.0,
3.0,
1.0,
2.5,
4.0,
5.0,
3.0,
4.0,
3.0,
4.0,
5.0,
4.0,
4.0,
0.5,
3.5,
2.0,
2.0,
4.0,
3.0,
4.0,
3.0,
3.5,
3.0,
4.5,
5.0,
1.5,
4.0,
2.5,
4.0,
5.0,
4.0,
4.0,
2.0,
3.0,
4.0,
0.5,
3.0,
4.0,
4.0,
3.0,
3.0,
3.0,
3.5,
3.5,
2.5,
5.0,
4.0,
1.0,
1.0,
2.5,
4.5,
4.0,
3.0,
4.0,
2.0,
4.0,
4.0,
3.0,
3.0,
4.0,
4.0,
0.5,
3.5,
4.0,
2.0,
3.0,
3.0,
3.0,
2.0,
4.0,
4.5,
4.0,
3.0,
2.0,
3.5,
0.5,
5.0,
3.0,
5.0,
3.0,
2.0,
2.0,
4.5,
2.5,
1.0,
3.0,
4.0,
1.0,
4.0,
4.0,
4.0,
4.0,
3.0,
4.0,
1.0,
4.0,
3.5,
3.5,
3.0,
5.0,
4.0,
3.5,
3.0,
1.0,
2.0,
3.5,
4.0,
5.0,
3.5,
5.0,
4.0,
1.0,
3.0,
2.0,
4.5,
3.5,
3.0,
3.5,
2.0,
3.0,
3.5,
2.0,
4.0,
5.0,
4.0,
3.0,
5.0,
3.0,
4.0,
5.0,
4.0,
3.0,
5.0,
3.0,
5.0,
5.0,
3.0,
3.0,
4.0,
2.0,
5.0,
3.0,
1.0,
3.0,
3.0,
3.5,
4.0,
4.0,
5.0,
4.0,
2.0,
5.0,
3.5,
5.0,
2.5,
4.5,
0.5,
4.0,
4.0,
1.5,
5.0,
3.5,
4.0,
3.0,
3.0,
5.0,
2.5,
2.5,
2.0,
1.0,
3.5,
4.5,
5.0,
2.5,
2.5,
4.0,
4.0,
3.0,
3.0,
4.0,
1.5,
3.0,
3.0,
5.0,
4.5,
3.0,
5.0,
3.0,
1.0,
4.0,
3.0,
3.0,
4.5,
4.0,
3.0,
5.0,
4.0,
4.0,
4.0,
1.0,
1.0,
3.5,
2.0,
2.0,
1.0,
2.0,
3.5,
4.5,
5.0,
4.0,
3.0,
1.0,
5.0,
5.0,
4.0,
4.0,
4.0,
2.0,
1.5,
3.0,
4.0,
2.0,
4.0,
4.0,
4.0,
4.5,
4.0,
3.5,
3.5,
4.5,
3.5,
4.0,
3.0,
4.0,
2.0,
5.0,
4.0,
4.0,
4.5,
3.0,
4.5,
5.0,
3.0,
0.5,
4.0,
1.0,
5.0,
4.0,
2.5,
3.0,
3.0,
3.0,
4.5,
3.0,
1.0,
5.0,
3.0,
4.5,
1.0,
4.0,
3.0,
3.5,
4.5,
4.0,
3.0,
3.0,
5.0,
4.0,
3.0,
3.0,
4.0,
4.5,
4.0,
4.0,
4.5,
2.0,
3.0,
4.0,
3.5,
3.0,
0.5,
4.5,
3.5,
3.0,
1.0,
5.0,
2.0,
3.5,
3.5,
5.0,
2.5,
3.0,
3.0,
2.0,
3.0,
4.0,
4.5,
3.0,
2.5,
3.0,
1.0,
3.0,
2.5,
4.0,
2.0,
3.0,
2.0,
2.0,
3.0,
0.5,
4.0,
3.0,
3.0,
3.5,
4.0,
4.0,
5.0,
3.5,
2.5,
3.0,
3.0,
3.5,
4.0,
3.0,
4.0,
3.0,
4.0,
3.0,
4.5,
4.0,
5.0,
3.0,
3.5,
1.5,
4.0,
3.5,
3.0,
4.0,
2.0,
3.0,
5.0,
2.0,
3.0,
4.0,
4.0,
4.0,
3.0,
1.0,
3.0,
3.5,
4.0,
3.0,
4.0,
3.0,
3.0,
3.0,
4.5,
3.0,
4.0,
4.5,
3.5,
4.0,
4.0,
3.0,
5.0,
3.5,
4.0,
3.0,
1.0,
3.0,
3.5,
3.0,
2.5,
4.0,
2.5,
4.0,
5.0,
5.0,
4.0,
4.0,
4.0,
5.0,
3.5,
4.0,
3.5,
4.0,
5.0,
3.0,
1.5,
4.0,
4.0,
3.0,
5.0,
3.0,
3.0,
5.0,
4.0,
5.0,
3.0,
1.0,
5.0,
3.0,
2.5,
2.0,
2.5,
4.0,
4.5,
1.5,
2.0,
4.5,
4.0,
3.0,
4.0,
3.0,
4.0,
2.0,
2.0,
1.5,
3.5,
5.0,
3.5,
3.5,
4.0,
4.0,
2.0,
2.0,
1.0,
4.0,
3.5,
2.5,
2.0,
4.5,
3.5,
3.0,
3.0,
3.0,
4.0,
4.0,
4.0,
4.0,
4.0,
5.0,
4.0,
5.0,
3.0,
2.0,
5.0,
2.5,
5.0,
3.0,
4.0,
3.0,
2.0,
4.5,
3.5,
3.0,
4.0,
3.0,
4.0,
3.0,
3.0,
4.0,
2.5,
2.0,
3.0,
4.0,
2.0,
4.5,
2.0,
4.0,
3.0,
3.0,
2.0,
1.5,
4.0,
3.0,
4.0,
3.0,
3.0,
5.0,
5.0,
3.0,
4.5,
4.0,
2.0,
4.0,
5.0,
5.0,
3.0,
1.0,
3.0,
2.0,
4.0,
3.0,
3.5,
3.0,
2.5,
2.5,
4.0,
3.0,
3.0,
3.0,
4.0,
4.0,
4.0,
3.5,
3.0,
3.0,
1.0,
4.0,
3.0,
4.0,
3.5,
5.0,
3.0,
5.0,
4.0,
5.0,
3.5,
3.5,
2.5,
3.0,
2.5,
4.0,
1.5,
2.5,
4.0,
1.0,
1.5,
1.5,
2.5,
5.0,
3.0,
2.5,
...]),
('153',
[2.0,
4.0,
4.0,
4.0,
4.0,
3.0,
4.0,
3.5,
2.5,
3.0,
4.0,
2.0,
3.0,
3.0,
4.0,
3.0,
1.0,
3.5,
3.0,
4.0,
3.0,
3.0,
2.0,
3.0,
4.0,
3.0,
3.0,
1.0,
2.0,
3.0,
1.0,
2.0,
1.5,
3.0,
3.0,
3.0,
4.0,
2.0,
2.0,
3.0,
2.5,
3.5,
5.0,
2.0,
5.0,
1.0,
5.0,
2.0,
1.0,
3.5,
2.0,
3.0,
3.0,
2.0,
4.0,
1.0,
3.0,
3.0,
4.5,
3.0,
2.5,
1.0,
3.0,
3.0,
1.0,
2.0,
3.0,
3.0,
3.0,
3.0,
3.0,
4.0,
3.0,
4.0,
2.0,
2.5,
4.0,
3.0,
3.5,
3.0,
3.0,
5.0,
4.0,
2.0,
3.5,
4.0,
3.0,
2.0,
2.0,
2.0,
3.0,
1.0,
3.0,
2.0,
3.0,
3.0,
4.0,
3.0,
2.0,
3.0,
4.0,
3.0,
3.0,
1.5,
3.0,
4.0,
5.0,
3.0,
3.0,
3.0,
3.0,
0.5,
2.0,
3.0,
3.0,
5.0,
3.0,
3.0,
2.0,
3.0,
2.5,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
2.5,
3.0,
4.0,
3.0,
2.5,
3.0,
2.0,
3.5,
3.0,
1.5,
4.0,
3.0,
3.0,
4.0,
2.5,
2.5,
2.0,
3.0,
2.0,
1.0,
3.0,
4.0,
3.0,
2.5,
2.5,
3.0,
3.0,
3.0,
2.0,
3.0,
4.0,
3.0,
3.0,
3.0,
3.5,
5.0,
3.0,
2.5,
2.0,
4.0,
3.0,
2.0,
2.0,
2.0,
3.0,
3.0,
1.0,
3.5,
3.5,
1.0,
3.0,
3.0,
3.0,
4.0,
4.0,
4.5,
1.0,
3.0,
3.0,
4.0,
3.0,
3.0,
1.0,
2.0,
5.0,
3.0,
3.0,
4.0,
3.0,
2.0,
3.0,
3.0,
2.5,
3.0,
3.0,
0.5,
3.5,
4.0,
1.5,
1.0,
3.0,
4.0,
3.5,
3.0,
3.0,
3.5,
1.5,
2.0,
2.5,
3.0,
3.0,
3.0,
2.0,
4.0,
3.5,
3.0,
3.0,
3.0,
3.0,
3.0,
5.0,
2.5,
4.0,
3.0,
3.5,
1.0,
3.0,
3.0,
4.0,
3.0,
1.0,
3.0,
3.0,
2.0,
2.0,
2.0,
3.5,
3.0,
3.0,
4.0,
3.5,
3.0,
4.0,
4.0,
2.5,
2.0,
3.0,
2.0,
3.0,
3.0,
3.0,
3.0,
1.5,
1.0,
4.0,
2.0,
3.0,
3.0,
3.0,
2.5,
5.0,
3.0,
3.0,
3.0,
2.0,
2.0,
2.5,
4.0,
4.0,
4.0,
3.0,
2.0,
5.0,
3.0,
3.0,
4.0,
3.0,
3.0,
1.0,
1.0,
3.0,
1.0,
1.5,
3.0,
3.5,
3.0,
3.0,
1.0,
4.0,
3.0,
3.0,
4.0,
4.0,
3.0,
3.5,
3.5,
3.5,
4.0,
5.0,
3.0,
4.0,
3.0,
2.0,
5.0,
0.5,
2.0,
3.0,
2.5,
3.0,
3.0,
5.0,
4.0,
2.5,
5.0,
3.0,
2.0,
1.0,
2.5,
3.0,
2.0,
3.0,
3.0,
2.0,
3.0,
3.0,
3.0,
4.0,
3.0,
4.0,
3.0,
2.0,
2.0,
5.0,
3.0,
3.0,
3.0,
3.0,
1.0,
0.5,
4.0,
4.0,
3.0,
3.0,
5.0,
3.5,
3.0,
2.0,
4.0,
3.0,
4.0,
2.0,
1.0,
3.0,
3.0,
4.0,
3.0,
3.0,
3.0,
5.0,
3.0,
3.0,
3.0,
3.0,
3.0,
4.0,
3.0,
3.0,
3.5,
3.0,
2.0,
3.0,
4.0,
5.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
2.5,
5.0,
3.0,
3.5,
3.0,
3.0,
2.0,
3.5,
3.0,
3.5,
3.0,
3.0,
3.0,
3.0,
4.0,
4.0,
1.0,
1.0,
3.0,
5.0,
4.0,
3.0,
2.0,
1.0,
4.0,
4.0,
5.0,
2.0,
3.0,
3.0,
2.0,
2.0,
2.5,
2.0,
1.5,
2.0,
1.0,
3.0,
3.0,
2.0,
4.0,
3.0,
4.0,
2.0,
5.0,
4.0,
2.5,
3.5,
2.0,
3.0,
3.0,
3.0,
3.0,
4.0,
2.0,
3.5,
4.0,
1.0,
1.5,
4.0,
3.0,
3.0,
3.0,
2.5,
1.5,
5.0,
4.0,
1.0,
5.0,
2.0,
2.0,
3.0,
2.0,
3.0,
3.0,
3.0,
3.0,
4.0,
2.0,
3.0,
3.5,
4.0,
3.5,
4.0,
2.0,
1.0,
4.0,
3.0,
3.0,
3.0,
3.0,
3.0,
1.0,
3.5,
3.0,
2.0,
2.0,
4.0,
3.0,
5.0,
3.0,
2.0,
3.0,
3.0,
3.0,
3.0,
2.5,
2.0,
1.0,
2.0,
2.0,
3.0,
3.0,
3.5,
2.0,
3.0,
3.0,
3.0,
2.0,
3.0,
2.0,
2.0,
4.0,
3.0,
3.0,
2.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.5,
3.0,
3.0,
3.0,
5.0,
3.0,
2.0,
3.0,
5.0,
2.0,
3.0,
3.0,
3.0,
3.5,
3.0,
2.0,
3.0,
2.0,
5.0,
3.0,
3.0,
3.0,
0.5,
3.0,
2.5,
2.0,
3.0,
3.0,
3.0,
4.0,
1.0,
3.5,
2.0,
3.0,
1.5,
1.0,
3.0,
2.5,
3.0,
1.0,
3.0,
3.0,
3.0,
0.5,
4.5,
3.0,
3.0,
3.0,
3.0,
2.0,
4.0,
2.5,
3.0,
3.0,
1.0,
2.5,
4.0,
3.0,
5.0,
4.0,
1.0,
5.0,
1.0,
2.0,
3.0,
3.0,
3.5,
2.5,
2.0,
4.0,
2.5,
4.0,
3.0,
3.0,
3.5,
2.0,
3.0,
4.0,
2.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
2.5,
3.0,
4.0,
2.5,
2.0,
2.5,
3.0,
1.5,
1.0,
3.0,
1.0,
3.0,
3.0,
2.0,
2.0,
4.0,
3.0,
3.0,
3.0,
3.5,
5.0,
2.0,
4.0,
2.0,
3.0,
4.0,
3.0,
3.0,
3.0,
4.0,
3.0,
4.0,
4.5,
5.0,
4.0,
5.0,
3.0,
2.0,
4.0,
1.0,
2.0,
1.5,
5.0,
4.0,
3.0,
4.0,
3.0,
4.0,
2.5,
4.0,
3.0,
3.0,
3.0,
3.0,
3.5,
4.0,
2.0,
3.0,
3.0,
3.0,
3.0,
3.0,
4.0,
3.0,
2.5,
3.0,
2.0,
3.0,
4.0,
3.0,
3.0,
3.0,
4.0,
3.0,
1.0,
3.0,
3.5,
4.0,
1.0,
3.0,
3.0,
5.0,
3.0,
2.5,
3.0,
3.0,
1.0,
3.0,
3.0,
3.0,
3.0,
5.0,
3.0,
4.0,
2.0,
5.0,
1.5,
3.0,
3.0,
4.0,
3.0,
3.0,
2.0,
3.0,
3.0,
0.5,
1.0,
3.0,
3.0,
3.0,
3.0,
3.5,
3.0,
3.0,
3.0,
2.0,
2.0,
3.0,
2.0,
3.5,
3.0,
1.5,
3.0,
0.5,
1.0,
3.0,
5.0,
4.0,
3.0,
4.0,
4.5,
2.0,
4.0,
4.0,
3.0,
4.0,
3.0,
4.5,
3.0,
3.0,
3.0,
5.0,
3.0,
3.0,
2.5,
3.0,
3.0,
3.0,
2.0,
4.0,
2.5,
1.5,
3.0,
2.5,
3.0,
3.0,
2.5,
2.0,
4.0,
3.0,
3.0,
3.0,
3.5,
2.5,
5.0,
2.5,
3.0,
2.0,
3.0,
3.0,
3.0,
4.0,
3.0,
4.0,
3.0,
3.0,
3.0,
3.0,
2.0,
4.0,
0.5,
3.0,
3.0,
4.0,
3.0,
3.0,
3.0,
2.0,
3.0,
3.0,
4.0,
0.5,
4.0,
3.0,
4.0,
2.0,
2.0,
2.5,
3.0,
2.0,
3.0,
4.0,
2.0,
4.0,
3.0,
1.0,
4.0,
3.0,
4.0,
3.0,
3.0,
2.0,
3.0,
3.0,
1.0,
3.0,
3.0,
1.0,
3.0,
4.0,
3.0,
3.0,
4.0,
2.0,
5.0,
2.0,
4.0,
3.5,
1.5,
3.0,
2.0,
2.0,
3.0,
5.0,
3.0,
2.5,
1.5,
2.0,
2.0,
1.5,
3.0,
2.0,
2.5,
3.0,
5.0,
2.0,
3.0,
2.5,
4.0,
3.0,
4.5,
2.5,
2.5,
1.0,
1.5,
3.0,
4.0,
3.0,
2.0,
4.0,
1.0,
3.0,
3.0,
3.0,
4.0,
4.0,
3.0,
3.0,
4.0,
3.0,
4.0,
3.0,
3.5,
3.0,
0.5,
3.0,
3.5,
3.0,
3.0,
3.0,
4.0,
2.0,
2.0,
3.0,
2.0,
4.0,
3.0,
3.0,
2.0,
1.0,
4.0,
3.0,
4.0,
4.0,
4.0,
3.0,
2.0,
3.0,
2.0,
3.0,
1.0,
2.0,
3.0,
1.0,
3.0,
1.0,
4.0,
3.0,
4.0,
3.0,
2.0,
3.0,
3.0,
3.0,
3.0,
3.5,
3.5,
3.0,
1.0,
2.0,
3.0,
4.0,
3.0,
4.0,
4.0,
3.0,
4.0,
3.0,
2.0,
4.0,
5.0,
3.5,
2.0,
4.0,
4.0,
3.0,
3.0,
3.0,
4.0,
1.0,
1.5,
3.0,
3.0,
3.0,
4.0,
2.0,
3.0,
0.5,
1.0,
2.5,
1.5,
3.0,
3.0,
2.5,
3.0,
3.0,
3.0,
4.0,
3.0,
4.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
4.5,
3.0,
2.0,
3.0,
2.0,
0.5,
5.0,
2.0,
2.0,
3.0,
1.0,
2.0,
3.5,
3.0,
5.0,
2.0,
3.0,
1.0,
3.0,
2.5,
3.5,
1.0,
3.0,
2.5,
3.0,
1.0,
2.5,
1.0,
...]),
('165',
[4.5,
4.0,
3.0,
4.0,
4.0,
5.0,
4.5,
3.0,
4.0,
3.0,
3.5,
3.5,
2.0,
3.5,
4.0,
4.0,
4.0,
2.0,
4.0,
1.0,
3.0,
3.0,
4.0,
3.0,
3.0,
5.0,
3.0,
3.0,
4.0,
4.0,
3.0,
1.0,
2.0,
3.0,
2.0,
3.0,
3.0,
4.0,
1.0,
4.0,
4.0,
3.0,
2.5,
1.0,
3.5,
2.5,
3.0,
3.0,
2.5,
3.5,
3.0,
3.0,
3.0,
1.0,
5.0,
4.0,
3.0,
3.0,
2.0,
3.0,
4.0,
3.0,
4.0,
4.0,
4.0,
3.5,
1.0,
3.0,
5.0,
4.0,
4.0,
4.0,
3.0,
1.0,
4.0,
3.0,
4.0,
4.0,
3.0,
3.5,
5.0,
3.0,
4.0,
3.0,
3.0,
4.0,
4.0,
4.0,
4.0,
4.0,
4.0,
4.0,
2.0,
4.0,
3.0,
4.0,
4.0,
2.0,
3.0,
3.5,
4.0,
3.0,
4.0,
4.0,
4.0,
3.0,
3.0,
4.0,
4.0,
5.0,
3.0,
4.0,
3.0,
4.0,
4.5,
4.0,
4.0,
3.0,
4.0,
3.5,
5.0,
5.0,
5.0,
5.0,
5.0,
3.0,
3.5,
2.0,
4.0,
2.0,
5.0,
2.0,
3.0,
4.0,
3.0,
4.5,
4.0,
4.0,
2.0,
4.0,
5.0,
4.0,
2.0,
3.0,
3.0,
3.0,
2.0,
2.0,
3.0,
3.0,
4.0,
3.5,
4.0,
4.0,
4.0,
4.0,
2.0,
1.0,
4.0,
3.5,
3.5,
3.0,
4.0,
4.0,
3.0,
3.0,
5.0,
4.0,
4.0,
4.0,
5.0,
4.0,
4.0,
2.0,
3.0,
4.0,
3.0,
4.0,
5.0,
2.5,
5.0,
3.5,
5.0,
3.0,
3.0,
5.0,
3.0,
4.0,
4.0,
1.0,
3.0,
2.5,
5.0,
3.0,
1.0,
5.0,
3.0,
4.0,
2.0,
3.0,
3.5,
2.5,
4.5,
3.0,
4.0,
4.0,
4.0,
4.0,
3.0,
2.5,
3.5,
3.0,
4.0,
3.0,
5.0,
3.0,
2.5,
5.0,
3.0,
4.0,
4.0,
4.0,
3.0,
3.5,
4.0,
3.0,
4.0,
2.5,
4.0,
4.0,
3.0,
2.5,
4.5,
5.0,
3.5,
5.0,
3.5,
1.5,
4.0,
5.0,
3.5,
4.0,
4.0,
4.0,
3.0,
4.5,
3.0,
4.0,
5.0,
3.0,
3.0,
3.5,
3.0,
3.0,
5.0,
5.0,
2.0,
4.0,
3.0,
3.0,
4.0,
3.0,
4.0,
3.0,
3.0,
3.0,
4.0,
3.0,
3.0,
4.0,
2.0,
3.5,
5.0,
4.0,
3.5,
4.5,
4.0,
3.0,
4.0,
2.0,
4.0,
4.0,
3.0,
2.0,
5.0,
3.5,
4.0,
4.5,
2.5,
3.0,
4.0,
1.0,
3.0,
3.0,
5.0,
5.0,
3.0,
4.0,
4.0,
4.0,
5.0,
3.5,
3.0,
3.5,
3.0,
4.0,
4.0,
3.0,
3.0,
3.0,
5.0,
3.0,
4.0,
4.0,
5.0,
4.0,
4.0,
4.0,
2.5,
3.5,
5.0,
4.0,
5.0,
4.0,
4.5,
3.0,
3.0,
3.0,
4.5,
4.5,
5.0,
3.0,
4.0,
4.0,
4.0,
3.0,
4.0,
3.0,
4.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
5.0,
3.0,
4.0,
4.5,
3.0,
4.0,
5.0,
2.0,
3.0,
5.0,
3.0,
3.0,
4.0,
4.0,
2.0,
3.0,
3.0,
4.0,
5.0,
4.0,
3.0,
4.0,
3.0,
4.0,
4.0,
3.5,
1.0,
2.5,
3.0,
3.0,
5.0,
4.0,
4.0,
1.0,
4.0,
3.0,
4.0,
3.0,
4.5,
3.0,
4.0,
4.0,
5.0,
1.0,
4.0,
3.5,
3.0,
3.0,
4.5,
3.0,
3.0,
5.0,
4.0,
3.5,
3.0,
3.0,
3.5,
3.5,
4.0,
3.0,
3.5,
4.0,
3.0,
3.5,
2.0,
4.0,
3.0,
3.0,
4.0,
4.0,
3.0,
4.0,
3.5,
3.0,
2.5,
3.0,
3.0,
5.0,
3.5,
4.0,
3.0,
1.5,
4.0,
3.0,
3.0,
3.0,
3.5,
4.0,
4.0,
2.0,
2.5,
3.0,
4.0,
3.5,
4.0,
4.0,
4.0,
2.0,
4.0,
2.0,
3.5,
4.0,
4.0,
5.0,
3.0,
4.0,
4.0,
3.0,
4.0,
4.0,
4.0,
3.0,
2.0,
3.5,
4.0,
3.5,
3.0,
4.0,
2.0,
1.0,
4.0,
3.0,
3.0,
3.0,
4.5,
4.0,
3.0,
4.0,
4.5,
4.0,
3.0,
4.0,
4.0,
4.0,
4.0,
3.0,
4.0,
5.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.0,
4.0,
3.0,
1.0,
3.5,
3.0,
4.0,
1.0,
4.0,
3.0,
2.0,
3.0,
4.0,
3.0,
4.0,
4.0,
3.0,
3.0,
4.0,
4.0,
3.5,
3.0,
3.5,
4.0,
5.0,
4.0,
3.0,
3.0,
4.0,
3.0,
2.0,
3.0,
3.0,
5.0,
4.0,
4.0,
3.0,
3.5,
1.0,
4.0,
4.0,
4.0,
4.0,
4.5,
4.0,
3.0,
5.0,
4.0,
3.0,
3.0,
3.0,
5.0,
3.0,
3.0,
3.5,
4.0,
4.0,
3.0,
4.0,
3.0,
3.0,
4.0,
3.5,
4.0,
5.0,
1.0,
5.0,
2.0,
2.0,
5.0,
4.0,
4.0,
3.0,
3.0,
3.0,
2.0,
4.0,
3.0,
2.0,
4.0,
2.0,
3.0,
5.0,
3.0,
3.0,
2.0,
4.0,
5.0,
3.0,
4.0,
3.5,
3.0,
3.0,
4.5,
4.0,
3.0,
3.0,
4.0,
3.0,
3.0,
4.0,
3.5,
4.0,
1.5,
3.0,
1.0,
2.0,
2.0,
4.0,
3.0,
4.0,
3.0,
4.0,
4.0,
3.0,
4.0,
4.0,
5.0,
5.0,
4.0,
2.5,
3.5,
4.0,
4.0,
5.0,
3.0,
2.0,
4.0,
2.5,
4.0,
4.5,
4.0,
5.0,
2.0,
3.0,
3.0,
4.0,
3.0,
4.0,
4.0,
3.5,
3.5,
5.0,
4.0,
5.0,
4.0,
4.0,
2.5,
4.0,
3.0,
3.0,
3.5,
4.0,
4.0,
2.0,
4.0,
4.0,
3.5,
3.0,
3.0,
1.0,
5.0,
3.5,
4.0,
2.0,
4.0,
1.5,
3.0,
3.0,
4.0,
4.0,
4.0,
3.0,
3.0,
4.0,
4.5,
3.0,
3.0,
3.5,
4.0,
2.0,
3.0,
5.0,
3.0,
5.0,
2.5,
4.0,
4.0,
3.5,
3.0,
2.5,
3.0,
4.0,
3.0,
3.0,
3.0,
4.5,
3.0,
2.0,
3.0,
4.0,
4.0,
4.0,
4.0,
3.0,
4.0,
5.0,
5.0,
4.0,
3.0,
3.0,
5.0,
4.0,
3.0,
5.0,
3.5,
3.0,
3.0,
3.5,
2.5,
4.0,
4.5,
3.0,
1.0,
5.0,
3.5,
3.0,
3.5,
4.0,
4.0,
4.0,
4.5,
4.5,
4.0,
4.0,
3.0,
3.0,
3.0,
3.5,
3.0,
4.0,
5.0,
3.5,
1.5,
5.0,
4.0,
4.0,
4.0,
4.5,
3.5,
2.0,
3.0,
1.0,
3.0,
5.0,
3.0,
4.0,
5.0,
4.0,
5.0,
5.0,
3.0,
4.0,
4.0,
3.0,
5.0,
4.0,
3.0,
5.0,
2.5,
3.0,
4.0,
5.0,
3.0,
4.0,
3.0,
5.0,
4.0,
3.0,
5.0,
3.0,
4.0,
4.0,
3.0,
3.0,
3.0,
2.0,
3.0,
4.5,
4.0,
3.0,
3.0,
3.0,
5.0,
4.0,
3.0,
5.0,
4.5,
3.0,
3.0,
2.0,
2.0,
3.0,
3.0,
4.0,
4.0,
3.0,
3.0,
5.0,
0.5,
2.0,
3.0,
5.0,
4.5,
3.5,
4.0,
4.0,
4.0,
3.0,
5.0,
4.0,
2.0,
3.0,
2.5,
2.5,
3.0,
5.0,
4.0,
4.0,
5.0,
3.0,
2.0,
4.0,
3.5,
4.0,
4.0,
3.0,
4.5,
4.0,
4.0,
4.5,
4.0,
3.0,
3.0,
3.0,
5.0,
3.5,
2.0,
4.0,
3.0,
3.0,
3.0,
4.0,
3.5,
4.0,
3.0,
2.5,
4.0,
4.0,
5.0,
2.0,
3.0,
5.0,
5.0,
2.0,
4.0,
3.5,
3.0,
4.0,
3.5,
3.0,
3.0,
4.0,
4.0,
4.0,
3.0,
2.0,
3.0,
3.0,
4.0,
2.5,
4.0,
3.0,
5.0,
4.5,
4.0,
3.0,
3.5,
3.0,
4.5,
3.0,
4.0,
3.0,
3.0,
5.0,
4.0,
2.5,
4.0,
3.0,
3.0,
3.0,
4.0,
5.0,
3.0,
3.0,
3.0,
3.0,
3.0,
3.5,
4.0,
3.5,
4.0,
5.0,
5.0,
3.0,
3.0,
4.0,
3.0,
3.0,
3.0,
4.0,
3.0,
3.0,
3.0,
4.0,
4.0,
2.5,
3.0,
4.0,
3.0,
4.0,
3.5,
5.0,
2.5,
3.0,
3.0,
3.0,
3.0,
2.5,
4.0,
3.5,
4.0,
3.0,
3.5,
4.0,
3.0,
4.0,
4.0,
4.0,
5.0,
4.0,
3.0,
4.5,
3.0,
4.0,
2.5,
4.0,
3.0,
3.0,
4.0,
4.0,
3.0,
3.0,
3.0,
5.0,
3.0,
4.0,
3.0,
3.0,
4.0,
1.0,
5.0,
3.0,
2.0,
3.0,
5.0,
4.0,
4.0,
3.0,
3.0,
1.5,
3.0,
4.0,
3.0,
4.0,
3.5,
3.0,
5.0,
3.0,
4.0,
4.0,
3.0,
5.0,
4.0,
3.0,
5.0,
2.0,
4.0,
3.0,
3.0,
2.0,
3.0,
5.0,
5.0,
1.0,
2.0,
4.0,
1.0,
3.0,
3.5,
4.0,
4.0,
...]),
('181',
[0.5,
2.0,
5.0,
1.0,
0.5,
3.0,
0.5,
1.0,
1.0,
3.0,
2.0,
3.0,
1.0,
5.0,
3.0,
1.0,
2.0,
2.5,
3.0,
2.0,
1.0,
1.0,
0.5,
3.0,
3.0,
1.0,
3.0,
1.0,
3.0,
3.0,
1.0,
2.0,
3.0,
1.0,
1.0,
2.5,
0.5,
5.0,
2.5,
0.5,
1.0,
2.0,
4.0,
0.5,
2.0,
1.0,
4.0,
3.0,
4.5,
3.0,
2.0,
1.0,
3.5,
2.0,
2.0,
1.0,
1.0,
4.0,
2.0,
1.0,
1.0,
1.0,
4.0,
4.0,
3.5,
3.0,
1.0,
1.0,
2.5,
4.0,
2.5,
1.0,
1.0,
1.0,
1.0,
2.0,
2.0,
1.0,
2.0,
0.5,
3.0,
0.5,
3.0,
3.0,
3.5,
1.0,
4.0,
0.5,
1.0,
5.0,
2.0,
3.0,
3.0,
3.0,
3.0,
1.0,
1.5,
3.0,
1.0,
2.0,
1.0,
3.0,
3.0,
2.0,
0.5,
3.5,
3.0,
2.0,
4.0,
3.0,
3.0,
1.0,
1.0,
3.0,
1.0,
3.0,
1.0,
1.0,
0.5,
3.0,
2.0,
3.0,
1.0,
2.0,
2.0,
5.0,
1.0,
1.0,
1.0,
1.0,
1.5,
3.0,
3.0,
0.5,
2.0,
3.0,
3.0,
2.0,
2.0,
0.5,
0.5,
3.0,
1.0,
0.5,
4.0,
3.0,
2.0,
0.5,
1.0,
4.0,
4.0,
1.0,
1.0,
1.0,
2.5,
2.0,
1.0,
2.0,
1.0,
1.0,
2.0,
1.0,
1.0,
1.0,
1.0,
1.0,
2.5,
1.0,
3.0,
1.0,
3.0,
4.0,
0.5,
3.0,
2.0,
1.0,
2.5,
3.5,
3.0,
1.0,
2.0,
4.0,
1.0,
1.0,
2.0,
1.0,
4.5,
1.0,
0.5,
0.5,
1.0,
1.0,
2.0,
2.0,
3.0,
1.0,
4.0,
1.0,
1.0,
2.0,
1.0,
2.0,
3.0,
1.0,
1.0,
3.0,
1.5,
3.0,
1.0,
1.0,
3.0,
0.5,
2.0,
3.0,
0.5,
0.5,
1.0,
1.0,
0.5,
2.0,
3.0,
1.0,
1.0,
2.0,
1.0,
2.5,
2.0,
4.5,
4.0,
0.5,
1.0,
2.0,
3.0,
3.0,
1.0,
5.0,
3.5,
1.0,
2.0,
1.0,
1.0,
3.5,
2.5,
1.0,
1.0,
2.0,
1.0,
1.0,
1.0,
3.0,
2.0,
1.5,
1.0,
2.0,
2.5,
2.5,
2.0,
2.0,
0.5,
2.5,
1.0,
1.5,
1.0,
1.0,
0.5,
0.5,
2.0,
3.5,
1.0,
3.0,
1.0,
1.0,
2.0,
0.5,
2.0,
1.0,
3.0,
2.0,
2.0,
1.0,
3.0,
1.5,
2.0,
4.0,
2.0,
5.0,
1.0,
2.0,
3.0,
1.0,
2.0,
1.0,
1.0,
3.0,
2.0,
0.5,
1.0,
0.5,
0.5,
0.5,
3.5,
1.0,
3.0,
1.0,
1.0,
2.5,
5.0,
5.0,
2.0,
4.0,
0.5,
1.0,
2.0,
1.0,
1.0,
1.0,
1.0,
2.0,
1.0,
2.0,
3.0,
2.5,
3.0,
3.0,
4.0,
0.5,
1.0,
1.0,
1.0,
3.0,
1.0,
1.0,
0.5,
1.0,
2.5,
2.0,
3.0,
1.0,
1.0,
5.0,
1.0,
4.0,
4.0,
3.0,
2.0,
3.0,
3.0,
1.0,
3.0,
1.5,
1.5,
1.0,
1.5,
1.0,
2.0,
2.0,
3.0,
3.0,
1.0,
3.0,
3.0,
0.5,
1.0,
1.0,
1.0,
0.5,
2.0,
3.0,
2.0,
2.0,
1.0,
1.0,
2.0,
2.0,
3.0,
0.5,
3.5,
3.0,
2.5,
1.0,
3.0,
3.0,
2.0,
2.0,
1.0,
1.0,
3.0,
2.0,
3.0,
0.5,
1.0,
2.0,
3.0,
2.0,
3.0,
4.0,
5.0,
2.0,
3.0,
3.5,
2.0,
4.5,
0.5,
0.5,
2.0,
2.5,
1.0,
1.0,
1.0,
0.5,
3.0,
2.0,
3.0,
5.0,
5.0,
1.0,
3.0,
3.0,
3.0,
3.0,
5.0,
1.0,
1.0,
2.0,
5.0,
1.0,
0.5,
2.0,
1.0,
4.5,
2.0,
1.0,
2.0,
2.0,
2.0,
2.0,
0.5,
3.0,
0.5,
2.0,
1.0,
2.0,
1.0,
0.5,
1.0,
1.0,
3.0,
0.5,
4.0,
5.0,
2.0,
2.0,
2.0,
3.0,
1.5,
3.0,
0.5,
1.0,
0.5,
3.0,
0.5,
1.0,
1.0,
2.0,
2.0,
0.5,
0.5,
1.0,
3.0,
1.0,
3.0,
4.0,
2.5,
1.0,
0.5,
4.0,
1.0,
2.0,
3.0,
2.5,
4.0,
3.0,
4.0,
2.5,
2.0,
1.0,
2.0,
4.0,
3.0,
1.0,
1.5,
2.0,
1.0,
2.0,
1.0,
1.0,
2.0,
1.0,
1.0,
3.0,
3.0,
0.5,
2.0,
3.0,
2.0,
1.0,
1.0,
3.0,
1.0,
1.0,
2.0,
4.0,
1.0,
3.0,
3.0,
3.0,
3.0,
1.0,
0.5,
4.0,
1.0,
4.0,
2.0,
1.0,
3.0,
1.0,
1.0,
1.0,
3.0,
2.0,
1.5,
4.0,
3.0,
2.0,
1.0,
3.0,
1.5,
4.0,
1.0,
1.5,
0.5,
0.5,
3.5,
1.0,
3.0,
2.0,
1.0,
2.0,
2.0,
4.0,
1.5,
0.5,
1.0,
4.0,
1.0,
1.0,
2.0,
1.0,
1.0,
1.5,
2.0,
0.5,
0.5,
2.0,
2.0,
1.0,
3.0,
3.0,
0.5,
3.0,
4.0,
4.0,
3.0,
2.0,
3.0,
3.0,
1.5,
3.0,
0.5,
2.0,
0.5,
2.0,
2.5,
0.5,
0.5,
0.5,
0.5,
1.0,
1.0,
1.0,
3.0,
0.5,
2.0,
5.0,
2.0,
1.0,
0.5,
1.0,
3.5,
3.0,
1.0,
1.0,
2.0,
2.0,
4.0,
3.0,
2.0,
1.0,
1.0,
2.0,
1.0,
1.0,
3.0,
1.0,
1.0,
3.5,
1.0,
1.0,
3.0,
0.5,
2.5,
2.0,
1.0,
0.5,
1.0,
1.0,
3.5,
1.0,
4.0,
1.5,
1.0,
2.5,
5.0,
1.0,
3.0,
3.0,
2.5,
1.0,
5.0,
1.0,
5.0,
3.0,
5.0,
4.0,
2.0,
2.0,
2.0,
2.0,
1.0,
1.0,
0.5,
0.5,
3.0,
4.5,
1.0,
1.0,
2.5,
3.5,
3.0,
1.0,
2.5,
5.0,
1.0,
0.5,
1.0,
1.0,
2.0,
2.0,
0.5,
0.5,
1.5,
1.0,
3.5,
1.0,
1.0,
2.0,
3.0,
1.0,
1.0,
3.0,
2.0,
1.0,
1.0,
4.0,
1.0,
1.0,
0.5,
1.5,
2.5,
2.0,
5.0,
1.0,
0.5,
2.0,
0.5,
1.0,
1.0,
2.0,
3.0,
1.0,
3.0,
1.0,
3.0,
1.0,
4.0,
4.0,
2.0,
1.0,
1.5,
3.0,
1.0,
1.0,
2.0,
2.0,
3.0,
2.0,
2.0,
1.0,
2.0,
3.0,
1.0,
1.5,
5.0,
1.0,
5.0,
3.0,
5.0,
5.0,
3.0,
2.0,
2.0,
2.5,
3.5,
2.0,
1.0,
1.0,
4.0,
1.0,
1.0,
1.5,
1.0,
3.0,
3.0,
3.0,
3.5,
1.0,
1.5,
1.0,
3.0,
4.0,
2.0,
1.0,
2.0,
1.0,
0.5,
3.0,
2.0,
4.0,
1.5,
2.0,
3.0,
1.0,
3.0,
1.0,
3.0,
1.0,
2.0,
1.0,
3.0,
2.0,
1.0,
1.0,
1.0,
2.5,
3.0,
3.0,
4.0,
2.0,
3.0,
0.5,
5.0,
1.0,
4.0,
1.0,
4.5,
1.0,
1.0,
1.0,
3.0,
2.0,
3.0,
1.0,
1.0,
0.5,
3.0,
3.0,
1.0,
1.0,
0.5,
1.0,
0.5,
4.5,
3.0,
2.0,
1.0,
3.0,
1.0,
0.5,
3.0,
1.0,
1.0,
1.0,
3.0,
1.0,
2.0,
1.0,
1.0,
3.0,
1.0,
2.0,
2.0,
4.0,
3.0,
3.0,
1.0,
3.0,
3.0,
1.5,
1.5,
5.0,
1.0,
3.0,
0.5,
2.5,
2.0,
5.0,
0.5,
1.0,
3.5,
1.0,
1.5,
1.0,
1.0,
2.0,
3.0,
3.0,
3.0,
2.0,
1.5,
1.0,
1.0,
0.5,
0.5,
0.5,
1.0,
3.5,
5.0,
1.0,
2.0,
2.0,
5.0,
3.0,
1.0,
2.0,
1.5,
1.0,
0.5,
1.0,
2.0,
2.0,
1.0,
1.0,
2.0,
0.5,
2.5,
1.0,
0.5,
4.5,
3.0,
1.0,
0.5,
1.0,
0.5,
1.0,
2.0,
0.5,
2.5,
2.0,
2.5,
3.0,
2.0,
3.0,
1.0,
1.0,
4.0,
3.0,
3.0,
2.0,
1.0,
1.0,
0.5,
3.0,
0.5,
3.0,
3.0,
1.5,
0.5,
3.0,
1.0,
1.0,
3.0,
1.0,
1.0,
2.0,
2.0,
1.0,
3.0,
3.0,
5.0,
3.0,
1.0,
3.0,
0.5,
2.0,
2.0,
1.0,
1.0,
2.0,
3.0,
1.0,
3.0,
3.0,
1.0,
2.0,
2.0,
2.0,
1.5,
0.5,
1.0,
1.0,
4.0,
3.0,
1.0,
1.0,
1.0,
1.0,
1.5,
2.5,
2.5,
3.5,
1.0,
2.0,
1.0,
1.0,
2.0,
1.0,
2.0,
5.0,
1.0,
3.0,
1.0,
3.0,
1.0,
2.5,
1.0,
1.0,
4.5,
3.0,
1.0,
3.0,
3.0,
1.0,
0.5,
3.0,
2.0,
1.0,
1.0,
5.0,
3.0,
3.0,
3.0,
3.0,
1.0,
2.0,
2.0,
3.5,
3.0,
2.0,
1.5,
1.0,
3.0,
1.0,
4.0,
3.5,
2.5,
1.0,
2.0,
0.5,
1.0,
1.0,
...])]
avgRatings01 = mapValuesToListRatings.mapValues(lambda V: sum(V) / float(len(V)))
avgRatings01.take(10)
[('3826', 2.5461339626882444),
('104', 3.3942778801636178),
('153', 2.9020622558025204),
('165', 3.5077587042963825),
('181', 1.9860046651116294),
('253', 3.5052233084147977),
('423', 3.008343508343508),
('494', 3.3739313292884723),
('762', 2.414987080103359),
('1396', 3.6338256484149856)]
test = [2.0, 4.0, 3.0]
sum(test) / len(test)
3.0
reduceByKey and countByKey
countsByKey = movieRatings.countByKey()
countsByKey
defaultdict(int,
{'307': 7958,
'481': 6037,
'1091': 6138,
'1257': 5902,
'1449': 6867,
'1590': 8511,
'1591': 6508,
'2134': 7020,
'2478': 7797,
'2840': 6047,
'2986': 6060,
'3020': 7783,
'3424': 7265,
'3698': 8269,
'3826': 8898,
'3893': 5259,
'170': 9574,
'849': 9878,
'1186': 8643,
'1235': 9937,
'1244': 10249,
'1296': 7470,
'1663': 9581,
'1962': 10219,
'2108': 7729,
'2243': 6398,
'2352': 7517,
'2707': 8198,
'2746': 10142,
'2915': 9590,
'3363': 9001,
'640': 3393,
'828': 1736,
'960': 173,
'1221': 38875,
'1321': 9175,
'1645': 15215,
'1825': 230,
'1985': 1129,
'2024': 856,
'2028': 54027,
'3171': 192,
'1': 68469,
'2': 27143,
'5': 15474,
'6': 28683,
'10': 33086,
'11': 19669,
'16': 21165,
'19': 24913,
'20': 4658,
'23': 4871,
'25': 24841,
'31': 11802,
'32': 54846,
'34': 35903,
'36': 23955,
'39': 29422,
'41': 5082,
'44': 13098,
'45': 9719,
'47': 55240,
'50': 62180,
'61': 2465,
'62': 23359,
'65': 5837,
'66': 2586,
'70': 15821,
'76': 3726,
'85': 2834,
'95': 25847,
'100': 4451,
'104': 22247,
'110': 68803,
'141': 26041,
'145': 15321,
'150': 58665,
'151': 13600,
'153': 38647,
'158': 15876,
'160': 15995,
'161': 26316,
'163': 17167,
'165': 41244,
'169': 4062,
'172': 14458,
'173': 17131,
'175': 5783,
'180': 9727,
'181': 3001,
'185': 26803,
'186': 13063,
'193': 8937,
'198': 9919,
'204': 9299,
'216': 10663,
'223': 26772,
'231': 37823,
'253': 31302,
'255': 1127,
'256': 11730,
'260': 81815,
'267': 4614,
'288': 26413,
'292': 28986,
'293': 36504,
'296': 92406,
'315': 12506,
'316': 36996,
'317': 17944,
'318': 97999,
'327': 8466,
'329': 30053,
'332': 2280,
'333': 13032,
'338': 4925,
'339': 25959,
'344': 45608,
'349': 34547,
'350': 16960,
'353': 18091,
'356': 97040,
'361': 5565,
'364': 47305,
'367': 38699,
'368': 18954,
'370': 16947,
'377': 46475,
'379': 6547,
'380': 50864,
'382': 6049,
'405': 3880,
'410': 20625,
'420': 16381,
'423': 2457,
'432': 13948,
'434': 27627,
'435': 15481,
'440': 20214,
'442': 20711,
'454': 27660,
'455': 10071,
'457': 58031,
'466': 15391,
'471': 12308,
'474': 20306,
'480': 76451,
'485': 15516,
'494': 14504,
'500': 38484,
'514': 4897,
'517': 3711,
'520': 17142,
'524': 9271,
'527': 71516,
'539': 31775,
'541': 39441,
'546': 6443,
'552': 15092,
'586': 32738,
'587': 32210,
'588': 51827,
'589': 64258,
'590': 51877,
'592': 54448,
'593': 87899,
'595': 42400,
'597': 38504,
'608': 54500,
'610': 8702,
'647': 8716,
'648': 45064,
'653': 16831,
'724': 10633,
'733': 37936,
'736': 38662,
'741': 8985,
'762': 11610,
'778': 31545,
'780': 58949,
'784': 17388,
'785': 12589,
'786': 19078,
'788': 22507,
'798': 3418,
'805': 13931,
'818': 3156,
'832': 16707,
'836': 5613,
'852': 12021,
'858': 60904,
'875': 548,
'879': 2388,
'919': 27436,
'924': 31418,
'941': 1048,
'1015': 4366,
'1020': 8163,
'1036': 40284,
'1037': 7836,
'1047': 9873,
'1049': 5633,
'1059': 12541,
'1079': 22088,
'1080': 25221,
'1088': 14100,
'1089': 37331,
'1090': 18825,
'1092': 13203,
'1094': 13547,
'1100': 6350,
'1101': 23448,
'1126': 3456,
'1127': 19492,
'1136': 40866,
'1196': 65822,
'1198': 63505,
'1200': 34572,
'1203': 17931,
'1207': 17988,
'1208': 28986,
'1210': 66023,
'1214': 39282,
'1215': 14444,
'1220': 23154,
'1222': 27809,
'1240': 42254,
'1242': 13871,
'1259': 25115,
'1265': 40836,
'1270': 57492,
'1274': 10035,
'1275': 13201,
'1291': 42043,
'1320': 16293,
'1339': 11860,
'1356': 22492,
'1370': 18415,
'1372': 12577,
'1374': 18024,
'1375': 12663,
'1376': 15376,
'1377': 15120,
'1385': 7425,
'1391': 20078,
'1393': 25884,
'1396': 13880,
'1405': 10479,
'1407': 19844,
'1425': 2268,
'1445': 1422,
'1466': 12695,
'1476': 6560,
'1485': 22136,
'1500': 16248,
'1515': 4617,
'1517': 26784,
'1518': 3296,
'1527': 36269,
'1544': 17684,
'1552': 17080,
'1562': 11569,
'1580': 44287,
'1584': 24808,
'1586': 8491,
'1592': 1641,
'1597': 12683,
'1606': 930,
'1608': 17114,
'1610': 24936,
'1615': 2821,
'1616': 4531,
'1617': 32553,
'1625': 20986,
'1641': 17621,
'1642': 724,
'1647': 720,
'1653': 24537,
'1665': 4216,
'1673': 16149,
'1676': 19257,
'1682': 36016,
'1687': 5644,
'1690': 13736,
'1704': 42769,
'1721': 44787,
'1722': 14906,
'1732': 29805,
'1747': 12474,
'1748': 13705,
'1760': 3193,
'1769': 2933,
'1779': 7226,
'1784': 26316,
'1792': 5638,
'1805': 9155,
'1810': 4249,
'1831': 9112,
'1876': 14511,
'1882': 9913,
'1884': 11197,
'1892': 5221,
'1909': 14472,
'1917': 26818,
'1918': 11088,
'1921': 11000,
'1923': 28747,
'1961': 32726,
'1967': 10284,
'1968': 27894,
'2000': 20905,
'2001': 15956,
'2002': 11055,
'2003': 13710,
'2004': 5536,
'2006': 16837,
'2011': 25529,
'2012': 25117,
'2021': 10725,
'2036': 1236,
'2046': 6068,
'2053': 5354,
'2054': 19163,
'2058': 10474,
'2081': 17219,
'2082': 5490,
'2094': 8100,
'2100': 13776,
'2105': 12327,
'2115': 26764,
'2126': 5485,
'2133': 5172,
'2140': 9567,
'2150': 10242,
'2152': 870,
'2153': 3034,
'2161': 12426,
'2167': 14719,
'2174': 23829,
'2193': 10802,
'2194': 18165,
'2231': 7955,
'2268': 18541,
'2273': 13050,
'2278': 12317,
'2291': 26408,
'2294': 12974,
'2296': 2600,
'2302': 17015,
'2311': 4647,
'2334': 3430,
'2335': 7655,
'2353': 18459,
'2355': 25521,
'2378': 7100,
'2379': 4039,
'2381': 3161,
'2391': 7387,
'2393': 10066,
'2405': 7158,
'2406': 15539,
'2407': 12658,
'2424': 14625,
'2433': 4339,
'2450': 4102,
'2468': 3133,
'2470': 14185,
'2471': 6264,
'2490': 9885,
'2501': 9156,
'2502': 25628,
'2505': 4764,
'2539': 11112,
'2541': 10942,
'2542': 20843,
'2549': 1765,
'2571': 84545,
'2600': 6337,
'2605': 10326,
'2616': 7025,
'2617': 21696,
'2628': 37179,
'2640': 17550,
'2641': 10114,
'2672': 4799,
'2683': 28715,
'2688': 6870,
'2700': 20341,
'2701': 13580,
'2706': 28303,
'2710': 22775,
'2712': 18712,
'2716': 35343,
'2717': 10259,
'2718': 3499,
'2720': 4920,
'2722': 9269,
'2723': 9868,
'2762': 52270,
'2763': 14291,
'2764': 3177,
'2770': 11558,
'2791': 22746,
'2792': 5123,
'2794': 3704,
'2797': 23162,
'2803': 5933,
'2805': 5545,
'2808': 4986,
'2816': 1025,
'2858': 60820,
'2860': 2270,
'2881': 9427,
'2889': 2041,
'2890': 14396,
'2916': 24573,
'2918': 29300,
'2947': 13966,
'2948': 8742,
'2949': 8794,
'2953': 9050,
'2959': 65678,
'2985': 17133,
'2987': 24678,
'2989': 4779,
'2990': 4962,
'2991': 5853,
'2993': 5129,
'2997': 33390,
'3000': 13865,
'3033': 14135,
'3039': 11516,
'3052': 19617,
'3063': 1286,
'3082': 10764,
'3087': 6459,
'3107': 9313,
'3113': 6380,
'3114': 29820,
'3146': 4331,
'3156': 4997,
'3173': 4855,
'3190': 1137,
'3208': 2041,
'3247': 8901,
'3248': 3256,
'3249': 2923,
'3250': 3101,
'3252': 10279,
'3253': 15529,
'3254': 6113,
'3255': 12027,
'3256': 10884,
'3263': 6604,
'3264': 5580,
'3268': 2067,
'3275': 11214,
'3298': 5291,
'3300': 10085,
'3301': 8909,
'3316': 2686,
'3354': 7995,
'3388': 2221,
'3448': 13784,
'3452': 5169,
'3479': 4117,
'3481': 19130,
'3489': 10857,
'3513': 2847,
'3527': 18186,
'3552': 10624,
'3555': 8580,
'3578': 48666,
'3593': 4965,
'3608': 6786,
'3617': 9578,
'3623': 21073,
'3635': 5108,
'3638': 5545,
'3639': 4620,
'3697': 6654,
'3702': 11155,
'3703': 11194,
'3704': 6754,
'3717': 12888,
'3740': 6225,
'3745': 6628,
'3751': 20834,
'3753': 17008,
'3755': 13029,
'3763': 3516,
'3793': 36030,
'3809': 6721,
'3825': 5594,
'3827': 6413,
'3835': 975,
'3841': 1774,
'3868': 9443,
'3869': 5820,
'3879': 1635,
'3889': 1438,
'3916': 10366,
'3948': 21797,
'3952': 2756,
'3977': 20583,
'3980': 4499,
'3981': 4237,
'3984': 5518,
'3990': 865,
'3994': 17734,
'3996': 31224,
'3997': 2484,
'4005': 2389,
'4011': 26552,
'4015': 6684,
'4018': 12862,
'4019': 7274,
'4022': 27083,
'4025': 13682,
'4034': 19848,
'4052': 2255,
'4084': 5213,
'4085': 13132,
'4104': 1165,
'4121': 3385,
'4148': 9981,
'4167': 1865,
'4207': 1126,
'4214': 3891,
'4223': 9796,
'4226': 43739,
'4232': 5767,
'4238': 3668,
'4239': 9727,
'4246': 16667,
'4262': 15597,
'4270': 11758,
'4306': 46826,
'4308': 15516,
'4321': 8294,
'4343': 6230,
'4344': 7867,
'4351': 4758,
'4367': 10190,
'4396': 1808,
'4446': 7993,
'4448': 5076,
'4483': 800,
'4487': 2686,
'4544': 1793,
'4545': 4109,
'4571': 10197,
'4621': 4233,
'4638': 9838,
'4643': 11497,
'4678': 2060,
'4679': 3329,
'4701': 8309,
'4718': 11672,
'4719': 1522,
'4721': 480,
'4734': 8026,
'4776': 11651,
'4844': 2881,
'4874': 8868,
'4878': 28701,
'4886': 37112,
'4889': 2100,
'4890': 6717,
'4896': 27434,
'4901': 6886,
'4958': 3784,
'4963': 33429,
'4975': 11889,
'4979': 16906,
'4980': 4283,
'4992': 4178,
'4993': 61883,
'4995': 33936,
'5010': 14223,
'5049': 4632,
'5064': 6827,
'5069': 482,
'5107': 1314,
'5110': 4130,
'5152': 4321,
'5218': 19216,
'5219': 7423,
'5254': 7415,
'5283': 2698,
'5299': 13868,
'5349': 32068,
'5378': 22663,
'5388': 8276,
'5400': 4564,
'5418': 30822,
'5438': 846,
'5445': 33163,
'5459': 15145,
'5463': 3983,
'5464': 10524,
'5481': 11887,
'5502': 14572,
'5504': 1874,
'5507': 6432,
'5541': 3610,
'5563': 672,
'5665': 561,
'5679': 13526,
'5782': 4894,
'5816': 22971,
'5872': 7433,
'5876': 1921,
'5882': 2414,
'5903': 10970,
'5944': 4347,
'5945': 6660,
'5952': 56696,
'5954': 5805,
'5956': 10950,
'5959': 1878,
'5989': 28798,
'5999': 509,
'6003': 3547,
'6016': 21558,
'6059': 3789,
'6104': 2734,
'6188': 7588,
'6213': 2062,
'6264': 2432,
'6281': 7707,
'6283': 3601,
'6294': 2049,
'6322': 1185,
'6333': 21404,
'6365': 24505,
'6373': 15764,
'6377': 37000,
'6378': 14255,
'6383': 5553,
'6534': 7380,
'6535': 3097,
'6537': 12393,
'6539': 39674,
'6541': 6916,
'6548': 4686,
'6550': 3216,
'6564': 4733,
'6586': 3741,
'6595': 3564,
'6659': 3966,
'6679': 221,
'6708': 6222,
'6754': 7031,
'6764': 2145,
'6794': 1095,
'6807': 10557,
'6857': 2126,
'6874': 35659,
'6879': 3678,
'6934': 19081,
'6947': 8581,
'6966': 1542,
'6979': 5910,
'7004': 4042,
'7102': 843,
'7143': 16660,
'7147': 18598,
'7153': 57378,
'7346': 3597,
'7348': 1067,
'7361': 35064,
'7369': 1104,
'7373': 9694,
'7438': 29990,
'7444': 5605,
'7454': 6152,
'7458': 10967,
'7481': 1341,
'7569': 2838,
'7573': 1280,
'7649': 1142,
'7704': 274,
'7810': 606,
'7811': 467,
'7812': 490,
'7841': 967,
'7842': 1851,
'8132': 3853,
'8169': 2007,
'8360': 21573,
'8361': 11058,
'8371': 6032,
'8376': 10549,
'8464': 11625,
'8528': 8382,
'8622': 10409,
'8633': 1683,
'8636': 21989,
'8641': 9450,
'8644': 17848,
'8665': 21339,
'8781': 3373,
'8798': 11286,
'8807': 5535,
'8810': 5319,
'8861': 3932,
'8874': 19923,
'8912': 812,
'8914': 4451,
'8919': 236,
'8961': 31857,
'8972': 9283,
'8982': 248,
'8985': 3575,
'9005': 412,
'26819': 396,
'26870': 481,
'27773': 12152,
'27846': 1845,
'27905': 298,
'31184': 867,
'31696': 8479,
'31804': 1518,
'31878': 5319,
'32302': 305,
'32587': 23594,
'33004': 10000,
'33437': 1708,
'33493': 19565,
'33646': 2351,
'33679': 13324,
'33794': 32027,
'33834': 1579,
'34162': 11470,
'34319': 7646,
'36529': 9992,
'37380': 1861,
'37733': 5412,
'38061': 7577,
'39231': 1558,
'39435': 1930,
'40278': 4179,
'40614': 879,
'42721': 492,
'42738': 3927,
'43558': 649,
'43679': 1649,
'43708': 385,
'43936': 2464,
'44191': 27101,
'44193': 2012,
'44199': 8952,
'44399': 274,
'44759': 547,
'48774': 15428,
'51662': 19749,
'51935': 4781,
'52950': 724,
'53000': 6177,
'53972': 7073,
'53996': 10415,
'54286': 21677,
'55765': 7853,
'56367': 19131,
'728': 3751,
'1147': 4084,
'1193': 42181,
'1213': 35934,
'2060': 3062,
'2329': 34110,
'2485': 7285,
'3178': 5370,
'3949': 23400,
'4973': 37167,
'5137': 442,
'5528': 5871,
'5995': 18209,
'8784': 12323,
'8873': 5119,
'8949': 9592,
'8950': 9235,
'27831': 4122,
'30749': 12353,
'30820': 1100,
'31410': 6101,
'44195': 11106,
'44204': 749,
'46578': 20038,
'46976': 9631,
'48394': 20509,
'48516': 26162,
'48696': 1821,
'48780': 23132,
'49272': 21412,
'49530': 12907,
'50068': 3154,
'51255': 14379,
'52952': 1790,
'54503': 12999,
'54997': 7074,
'55118': 4903,
'55247': 10668,
'55290': 4590,
'55820': 18523,
'56782': 10062,
'21': 25699,
'111': 32885,
'208': 31310,
'224': 9139,
'227': 4851,
'261': 12058,
'273': 8384,
'337': 19633,
'515': 11598,
'555': 17382,
'666': 191,
'381': 5654,
'531': 9220,
'880': 6101,
'1041': 6054,
'1416': 6538,
'2160': 9931,
'3328': 5133,
'3744': 5079,
'3986': 6179,
'4033': 3852,
'4903': 2800,
'41997': 6475,
'3': 15585,
'7': 15301,
'12': 4524,
'14': 6838,
'52': 10398,
'135': 7737,
'140': 5649,
'605': 3887,
'637': 6954,
'743': 7573,
'765': 6007,
'869': 590,
'1057': 3745,
'1113': 1125,
'1317': 347,
'1363': 1843,
'1390': 1735,
'1409': 7159,
'1457': 3149,
'43': 2667,
'162': 6547,
'176': 2486,
'246': 11485,
'299': 2740,
'348': 8563,
'501': 2144,
'534': 5077,
'551': 24134,
'594': 20715,
'661': 12747,
'745': 13866,
'750': 29484,
'837': 6893,
'902': 12786,
'903': 17926,
'904': 22264,
'908': 19613,
'910': 13753,
'912': 31095,
'913': 14585,
'916': 7459,
'923': 21512,
'946': 1603,
'994': 5591,
'1029': 10926,
'1077': 7891,
'1093': 9238,
'1095': 6978,
'1131': 3556,
'1148': 17337,
'1188': 8678,
'1199': 17032,
'1202': 1867,
'1206': 32436,
'1219': 24234,
'1223': 9205,
'1225': 23768,
'1230': 19355,
'1234': 17906,
'1247': 20769,
'1249': 10447,
'1255': 2238,
'1258': 32129,
'1282': 14715,
'1284': 6470,
'1288': 18607,
'1307': 26020,
'1566': 7688,
'1635': 6280,
'1639': 16300,
'1688': 5216,
'1907': 12173,
'1952': 7448,
'1953': 10208,
'1957': 8065,
'2010': 6834,
'2078': 11538,
'2080': 12634,
'2087': 9082,
'2102': 1434,
'2116': 6164,
'2138': 3094,
'2186': 6137,
'2300': 7663,
'2324': 26995,
'2360': 2742,
'2394': 8314,
'2396': 27336,
'2687': 7475,
'2761': 11116,
'2973': 4479,
'3022': 2695,
'3067': 3126,
'3072': 9669,
'3077': 1242,
'3081': 16385,
'3083': 5165,
'3203': 3516,
'3246': 5330,
'3260': 3129,
'3265': 3099,
'48': 16358,
'1777': 16498,
'105': 11284,
'596': 15833,
'1380': 19077,
'2657': 15818,
'3897': 22796,
'362': 9088,
'720': 9674,
'838': 9226,
'2144': 10721,
'2724': 10551,
'2908': 9559,
'2968': 9769,
'3210': 9819,
'3863': 10172,
'1111': 1984,
'1281': 4881,
'3034': 6169,
'3147': 33140,
'5618': 23227,
'6863': 14719,
'7254': 16342,
'7669': 2919,
'8645': 2503,
'33166': 12260,
'35836': 15332,
'40339': 1945,
'40629': 6432,
'45517': 8391,
'47099': 9614,
'50872': 19803,
'51086': 2871,
'53125': 11762,
'55721': 1779,
'58559': 44741,
'59315': 26682,
'59784': 11540,
'60069': 28116,
'63082': 20006,
'63859': 3603,
'64969': 5780,
'68157': 23050,
'68954': 26143,
'71033': 2390,
'71899': 2701,
'72356': 1020,
'72641': 7504,
'73017': 14876,
'76093': 13908,
'78105': 3133,
'78499': 14841,
'79091': 9651,
'79132': 41475,
'80463': 14065,
'81591': 13770,
'81845': 14976,
'81847': 7697,
'83803': 607,
'84152': 9884,
'85342': 1673,
'85736': 788,
'86298': 2010,
'86504': 1800,
'86880': 5314,
'86882': 6518,
'86911': 4037,
'87485': 2009,
'87520': 2897,
'87869': 4761,
...})
def sumValues(x, y):
return (x + y)
sumRatings = movieRatings.reduceByKey(sumValues)
sumRatings.take(5)
[('3826', 22655.5),
('104', 75512.5),
('153', 112156.0),
('165', 144674.0),
('181', 5960.0)]
avgRatings02 = sumRatings.map(lambda x: (x[0], x[1]/countsByKey.get(x[0])))
avgRatings02.take(10)
[('3826', 2.5461339626882444),
('104', 3.3942778801636178),
('153', 2.9020622558025204),
('165', 3.5077587042963825),
('181', 1.9860046651116294),
('253', 3.5052233084147977),
('423', 3.008343508343508),
('494', 3.3739313292884723),
('762', 2.414987080103359),
('1396', 3.6338256484149856)]
movies = sc.textFile("e:/data/ml-latest/movies.csv")
movies.take(5)
['movieId,title,genres',
'1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy',
'2,Jumanji (1995),Adventure|Children|Fantasy',
'3,Grumpier Old Men (1995),Comedy|Romance',
'4,Waiting to Exhale (1995),Comedy|Drama|Romance']
movieHeader = movies.first()
movieInfo = movies.filter(lambda x: x != movieHeader)
movieInfo.take(100)
['1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy',
'2,Jumanji (1995),Adventure|Children|Fantasy',
'3,Grumpier Old Men (1995),Comedy|Romance',
'4,Waiting to Exhale (1995),Comedy|Drama|Romance',
'5,Father of the Bride Part II (1995),Comedy',
'6,Heat (1995),Action|Crime|Thriller',
'7,Sabrina (1995),Comedy|Romance',
'8,Tom and Huck (1995),Adventure|Children',
'9,Sudden Death (1995),Action',
'10,GoldenEye (1995),Action|Adventure|Thriller',
'11,"American President, The (1995)",Comedy|Drama|Romance',
'12,Dracula: Dead and Loving It (1995),Comedy|Horror',
'13,Balto (1995),Adventure|Animation|Children',
'14,Nixon (1995),Drama',
'15,Cutthroat Island (1995),Action|Adventure|Romance',
'16,Casino (1995),Crime|Drama',
'17,Sense and Sensibility (1995),Drama|Romance',
'18,Four Rooms (1995),Comedy',
'19,Ace Ventura: When Nature Calls (1995),Comedy',
'20,Money Train (1995),Action|Comedy|Crime|Drama|Thriller',
'21,Get Shorty (1995),Comedy|Crime|Thriller',
'22,Copycat (1995),Crime|Drama|Horror|Mystery|Thriller',
'23,Assassins (1995),Action|Crime|Thriller',
'24,Powder (1995),Drama|Sci-Fi',
'25,Leaving Las Vegas (1995),Drama|Romance',
'26,Othello (1995),Drama',
'27,Now and Then (1995),Children|Drama',
'28,Persuasion (1995),Drama|Romance',
'29,"City of Lost Children, The (Cité des enfants perdus, La) (1995)",Adventure|Drama|Fantasy|Mystery|Sci-Fi',
'30,Shanghai Triad (Yao a yao yao dao waipo qiao) (1995),Crime|Drama',
'31,Dangerous Minds (1995),Drama',
'32,Twelve Monkeys (a.k.a. 12 Monkeys) (1995),Mystery|Sci-Fi|Thriller',
'33,Wings of Courage (1995),Adventure|Romance|IMAX',
'34,Babe (1995),Children|Drama',
'35,Carrington (1995),Drama|Romance',
'36,Dead Man Walking (1995),Crime|Drama',
'37,Across the Sea of Time (1995),Documentary|IMAX',
'38,It Takes Two (1995),Children|Comedy',
'39,Clueless (1995),Comedy|Romance',
'40,"Cry, the Beloved Country (1995)",Drama',
'41,Richard III (1995),Drama|War',
'42,Dead Presidents (1995),Action|Crime|Drama',
'43,Restoration (1995),Drama',
'44,Mortal Kombat (1995),Action|Adventure|Fantasy',
'45,To Die For (1995),Comedy|Drama|Thriller',
'46,How to Make an American Quilt (1995),Drama|Romance',
'47,Seven (a.k.a. Se7en) (1995),Mystery|Thriller',
'48,Pocahontas (1995),Animation|Children|Drama|Musical|Romance',
'49,When Night Is Falling (1995),Drama|Romance',
'50,"Usual Suspects, The (1995)",Crime|Mystery|Thriller',
'51,Guardian Angel (1994),Action|Drama|Thriller',
'52,Mighty Aphrodite (1995),Comedy|Drama|Romance',
'53,Lamerica (1994),Adventure|Drama',
'54,"Big Green, The (1995)",Children|Comedy',
'55,Georgia (1995),Drama',
'56,Kids of the Round Table (1995),Adventure|Children|Comedy|Fantasy',
'57,Home for the Holidays (1995),Drama',
'58,"Postman, The (Postino, Il) (1994)",Comedy|Drama|Romance',
'59,"Confessional, The (Confessionnal, Le) (1995)",Drama|Mystery',
'60,"Indian in the Cupboard, The (1995)",Adventure|Children|Fantasy',
'61,Eye for an Eye (1996),Drama|Thriller',
"62,Mr. Holland's Opus (1995),Drama",
"63,Don't Be a Menace to South Central While Drinking Your Juice in the Hood (1996),Comedy|Crime",
'64,Two if by Sea (1996),Comedy|Romance',
'65,Bio-Dome (1996),Comedy',
'66,Lawnmower Man 2: Beyond Cyberspace (1996),Action|Sci-Fi|Thriller',
'67,Two Bits (1995),Drama',
'68,French Twist (Gazon maudit) (1995),Comedy|Romance',
'69,Friday (1995),Comedy',
'70,From Dusk Till Dawn (1996),Action|Comedy|Horror|Thriller',
'71,Fair Game (1995),Action',
'72,Kicking and Screaming (1995),Comedy|Drama',
'73,"Misérables, Les (1995)",Drama|War',
'74,Bed of Roses (1996),Drama|Romance',
'75,Big Bully (1996),Comedy|Drama',
'76,Screamers (1995),Action|Sci-Fi|Thriller',
'77,Nico Icon (1995),Documentary',
'78,"Crossing Guard, The (1995)",Action|Crime|Drama|Thriller',
'79,"Juror, The (1996)",Drama|Thriller',
'80,"White Balloon, The (Badkonake sefid) (1995)",Children|Drama',
"81,Things to Do in Denver When You're Dead (1995),Crime|Drama|Romance",
"82,Antonia's Line (Antonia) (1995),Comedy|Drama",
'83,Once Upon a Time... When We Were Colored (1995),Drama|Romance',
'84,Last Summer in the Hamptons (1995),Comedy|Drama',
'85,Angels and Insects (1995),Drama|Romance',
'86,White Squall (1996),Action|Adventure|Drama',
'87,Dunston Checks In (1996),Children|Comedy',
'88,Black Sheep (1996),Comedy',
'89,Nick of Time (1995),Action|Thriller',
'90,The Journey of August King (1995),Drama',
'92,Mary Reilly (1996),Drama|Horror|Thriller',
'93,Vampire in Brooklyn (1995),Comedy|Horror|Romance',
'94,Beautiful Girls (1996),Comedy|Drama|Romance',
'95,Broken Arrow (1996),Action|Adventure|Thriller',
'96,In the Bleak Midwinter (1995),Comedy|Drama',
'97,"Hate (Haine, La) (1995)",Crime|Drama',
'98,Shopping (1994),Action|Thriller',
'99,Heidi Fleiss: Hollywood Madam (1995),Documentary',
'100,City Hall (1996),Drama|Thriller',
'101,Bottle Rocket (1996),Adventure|Comedy|Crime|Romance']
movieTitles = movieInfo.map(lambda line: (line.split(",",1)[0], line.rsplit(",",1)[0].split(",",1)[1]))
movieTitles.take(20)
[('1', 'Toy Story (1995)'),
('2', 'Jumanji (1995)'),
('3', 'Grumpier Old Men (1995)'),
('4', 'Waiting to Exhale (1995)'),
('5', 'Father of the Bride Part II (1995)'),
('6', 'Heat (1995)'),
('7', 'Sabrina (1995)'),
('8', 'Tom and Huck (1995)'),
('9', 'Sudden Death (1995)'),
('10', 'GoldenEye (1995)'),
('11', '"American President, The (1995)"'),
('12', 'Dracula: Dead and Loving It (1995)'),
('13', 'Balto (1995)'),
('14', 'Nixon (1995)'),
('15', 'Cutthroat Island (1995)'),
('16', 'Casino (1995)'),
('17', 'Sense and Sensibility (1995)'),
('18', 'Four Rooms (1995)'),
('19', 'Ace Ventura: When Nature Calls (1995)'),
('20', 'Money Train (1995)'),
('21', 'Get Shorty (1995)'),
('22', 'Copycat (1995)'),
('23', 'Assassins (1995)'),
('24', 'Powder (1995)'),
('25', 'Leaving Las Vegas (1995)'),
('26', 'Othello (1995)'),
('27', 'Now and Then (1995)'),
('28', 'Persuasion (1995)'),
('29', '"City of Lost Children, The (Cité des enfants perdus, La) (1995)"'),
('30', 'Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)'),
('31', 'Dangerous Minds (1995)'),
('32', 'Twelve Monkeys (a.k.a. 12 Monkeys) (1995)'),
('33', 'Wings of Courage (1995)'),
('34', 'Babe (1995)'),
('35', 'Carrington (1995)'),
('36', 'Dead Man Walking (1995)'),
('37', 'Across the Sea of Time (1995)'),
('38', 'It Takes Two (1995)'),
('39', 'Clueless (1995)'),
('40', '"Cry, the Beloved Country (1995)"'),
('41', 'Richard III (1995)'),
('42', 'Dead Presidents (1995)'),
('43', 'Restoration (1995)'),
('44', 'Mortal Kombat (1995)'),
('45', 'To Die For (1995)'),
('46', 'How to Make an American Quilt (1995)'),
('47', 'Seven (a.k.a. Se7en) (1995)'),
('48', 'Pocahontas (1995)'),
('49', 'When Night Is Falling (1995)'),
('50', '"Usual Suspects, The (1995)"'),
('51', 'Guardian Angel (1994)'),
('52', 'Mighty Aphrodite (1995)'),
('53', 'Lamerica (1994)'),
('54', '"Big Green, The (1995)"'),
('55', 'Georgia (1995)'),
('56', 'Kids of the Round Table (1995)'),
('57', 'Home for the Holidays (1995)'),
('58', '"Postman, The (Postino, Il) (1994)"'),
('59', '"Confessional, The (Confessionnal, Le) (1995)"'),
('60', '"Indian in the Cupboard, The (1995)"'),
('61', 'Eye for an Eye (1996)'),
('62', "Mr. Holland's Opus (1995)"),
('63',
"Don't Be a Menace to South Central While Drinking Your Juice in the Hood (1996)"),
('64', 'Two if by Sea (1996)'),
('65', 'Bio-Dome (1996)'),
('66', 'Lawnmower Man 2: Beyond Cyberspace (1996)'),
('67', 'Two Bits (1995)'),
('68', 'French Twist (Gazon maudit) (1995)'),
('69', 'Friday (1995)'),
('70', 'From Dusk Till Dawn (1996)'),
('71', 'Fair Game (1995)'),
('72', 'Kicking and Screaming (1995)'),
('73', '"Misérables, Les (1995)"'),
('74', 'Bed of Roses (1996)'),
('75', 'Big Bully (1996)'),
('76', 'Screamers (1995)'),
('77', 'Nico Icon (1995)'),
('78', '"Crossing Guard, The (1995)"'),
('79', '"Juror, The (1996)"'),
('80', '"White Balloon, The (Badkonake sefid) (1995)"'),
('81', "Things to Do in Denver When You're Dead (1995)"),
('82', "Antonia's Line (Antonia) (1995)"),
('83', 'Once Upon a Time... When We Were Colored (1995)'),
('84', 'Last Summer in the Hamptons (1995)'),
('85', 'Angels and Insects (1995)'),
('86', 'White Squall (1996)'),
('87', 'Dunston Checks In (1996)'),
('88', 'Black Sheep (1996)'),
('89', 'Nick of Time (1995)'),
('90', 'The Journey of August King (1995)'),
('92', 'Mary Reilly (1996)'),
('93', 'Vampire in Brooklyn (1995)'),
('94', 'Beautiful Girls (1996)'),
('95', 'Broken Arrow (1996)'),
('96', 'In the Bleak Midwinter (1995)'),
('97', '"Hate (Haine, La) (1995)"'),
('98', 'Shopping (1994)'),
('99', 'Heidi Fleiss: Hollywood Madam (1995)'),
('100', 'City Hall (1996)'),
('101', 'Bottle Rocket (1996)')]
s = '29,"City of Lost Children, The (Cité des enfants perdus, La) (1995)",Adventure|Drama|Fantasy|Mystery|Sci-Fi'
print(s.split(","))
print(s.rsplit(",",1)[0])
print(s.rsplit(",",1)[0].split(",",1)[1])
['29', '"City of Lost Children', ' The (Cité des enfants perdus', ' La) (1995)"', 'Adventure|Drama|Fantasy|Mystery|Sci-Fi']
29,"City of Lost Children, The (Cité des enfants perdus, La) (1995)"
"City of Lost Children, The (Cité des enfants perdus, La) (1995)"
augmentedRatings = avgRatings01.join(movieTitles)
augmentedRatings.take(10)
[('854',
(3.722689075630252, '"Ballad of Narayama, The (Narayama Bushiko) (1958)"')),
('8591', (3.1783018867924526, '"Philadelphia Experiment, The (1984)"')),
('5902', (3.833290751149719, 'Adaptation (2002)')),
('1938', (3.859350237717908, '"Lost Weekend, The (1945)"')),
('2099', (3.5872438863185723, 'Song of the South (1946)')),
('2713', (2.5252918287937742, 'Lake Placid (1999)')),
('5833', (3.3345864661654137, 'Dog Soldiers (2002)')),
('4899', (2.359262229350441, 'Black Knight (2001)')),
('509', (3.6844782352626226, '"Piano, The (1993)"')),
('106022', (3.402390438247012, 'Toy Story of Terror (2013)'))]
augmentedRatings.takeOrdered(50, key = lambda x: -x[1][0])
[('134633', (5.0, 'Tony 10 (2012)')),
('156307', (5.0, 'Pardon My Sarong (1942)')),
('92783', (5.0, 'Latin Music USA (2009)')),
('161468', (5.0, 'Nutcracker Fantasy (1979)')),
('136872', (5.0, 'Zapatlela (1993)')),
('183261', (5.0, 'Sexting in Suburbia (2012)')),
('165787', (5.0, 'Lonesome Dove Church (2014)')),
('158370', (5.0, 'Muzungu (1999)')),
('185637', (5.0, 'Crítico (2008)')),
('184151', (5.0, 'VeggieTales: Celery Night Fever (2014)')),
('134757', (5.0, 'The Last Hunt (1956)')),
('124174', (5.0, 'Until They Sail (1957)')),
('182453', (5.0, 'Pure Fucking Mayhem (2008)')),
('137980', (5.0, "The Fall of '55 (2006)")),
('129478', (5.0, 'A Blank on the Map (1971)')),
('160624', (5.0, 'Big Man - An Unusual Insurance (1988)')),
('130558', (5.0, 'Shark Kill (1976)')),
('159125', (5.0, "Anybody's Son Will Do")),
('187793', (5.0, 'The Bride (1973)')),
('136992', (5.0, "When He Didn't Come Home (1998)")),
('179183', (5.0, 'Dragon Lee Vs. The 5 Brothers (1978)')),
('118906', (5.0, 'Turbulence 3: Heavy Metal (2001)')),
('123541', (5.0, 'Geordie (1955)')),
('178845', (5.0, 'The Lost City of Cecil B. DeMille')),
('193385', (5.0, 'And Love Has Vanished (1961)')),
('186299', (5.0, 'Thirty Souls (2017)')),
('169554', (5.0, 'Trevor Moore: High In Church (2015)')),
('137064', (5.0, 'Twisted Desire (1996)')),
('185263', (5.0, 'Deaf child (2017)')),
('143717', (5.0, 'Meet Market (2008)')),
('167708', (5.0, 'Good Fortune (2009)')),
('177183', (5.0, 'The Besieged Fortress (2006)')),
('190707', (5.0, '1968 (2018)')),
('149879', (5.0, 'Aschenputtel (2010)')),
('148084', (5.0, 'Emmanuelle in Soho (1981)')),
('139887', (5.0, 'But Always (2014)')),
('137006', (5.0, 'Poison (2000)')),
('185265', (5.0, 'Mandy (1952)')),
('133970', (5.0, 'Pony Express (1953)')),
('166231', (5.0, 'Die Salzprinzessin (2015)')),
('180409', (5.0, '1984 Revolution (2011)')),
('166229', (5.0, 'Allerleirauh (2012)')),
('166576', (5.0, 'Voracious (2012)')),
('187223', (5.0, 'Night Key (1937)')),
('121051', (5.0, 'A Matter of Time (1976)')),
('156309', (5.0, "It Ain't Hay (1943)")),
('173319', (5.0, 'Warning: This Drug May Kill You (2017)')),
('171621', (5.0, 'Warm Summer Rain (1989)')),
('178417', (5.0, 'A Lustful Man (1961)')),
('124903', (5.0, 'Styx (2001)'))]
Challenge 1:#
Report the average ratings for movies with ratings higher than 3.75 and the number of ratings total at least 1000.
Challenge 2:#
Study the genome tags and answer the following question.
How do user perceive and distinguish sub-genres? For example, what is the difference between action and action comedy?