The Golden Era of Movies

<p><strong>Golden Era of Movies</strong></p><p>You’ve joined a movie analytics team investigating audience preferences, genre dynamics, and the true value of highly-rated films on Letterboxd. Using the Letterboxd Movie Ratings dataset, your goal is to clean and transform movie data, uncover hidden audience trends, and identify which films punch above their popularity level.</p><p>Here are four questions your team lead wants you to answer:</p><ol><li><p>Which genre is among the<strong><em> </em>top 5 most popular genres</strong> as well as among the<strong><em> </em>top 5 best rated genres</strong>?</p></li><li><p>Categorize movies into three groups based on their runtime: "Short Film" (runtime &lt; 60 minutes), "Standard" (runtime &lt;150 minutes), and "Epic" (runtime &gt;= 150 minutes). Choose your favorite three movie genres and <strong>compare their runtime based on the runtime</strong> <strong>categories</strong>.</p></li><li><p>For each genre, calculate the <strong>average rating in each decade</strong>. Compare how the ratings changed over time for the genres <em>Action</em>, <em>Crime</em>, <em>Romance</em>, and <em>Music</em>.</p></li><li><p>Identify the "<strong>Hidden Gems</strong>" in our dataset: Movies with a high rating by a reliable group of viewers but typically overlooked by the mainstream audience. You want to reward high ratings and penalize high popularity. Calculate a custom <strong>Hidden Gem Score (HGS)</strong> for each movie. Which <strong>25 movies rank highest</strong> on this custom index?<br><strong><em>Note:</em> </strong>If you struggle to come up with a HGS, find inspiration from document analysis! TF-IDF uses a logarithmic penalty to prevent incredibly common words from drowning out the unique ones.</p></li></ol><p>---</p><p><strong>Solution:</strong></p><p>We start by accessing the cleaned Letterboxd dataset using a <em>CSV Reader</em> node. With the <em>Expression </em>node we create a “decade” column: We divide the ‘’year_released’’ column values by 10, apply a floor function to that value, multiply it with 10, and add ”s” at the end.</p><p><strong><em>Note:</em></strong>&nbsp; Instead of <em>Rule Engine</em>, use the mighty <em>Expression </em>node to create the decade column here for efficiency.</p><p>As many movies fall into multiple genres, we use a <em>Cell Splitter</em> node to split them into unique values, followed by an <em>Ungroup </em>node so that each movie-genre pair occupies its own row. A <em>Column Renamer</em> node is used to quickly clean up and rename the “genres_SplitResultList”&nbsp; column to "genre".&nbsp;</p><p>From here, the preprocessed data splits into four branches.</p><ul><li><p><strong>First branch:</strong> Top 5 Most Popular vs. Top 5 Best Rated Genres. We use a <em>GroupBy </em>node to aggregate the data by “genre” and calculate the average for “rating” and “popularity”. Use two parallel <em>Top k Row Filter</em> nodes, one keeps the top 5 best rated genres, while the other keeps the top 5 most popular genres. Both filtered tables connect to a <em>Reference Row Filter</em> node, which identifies the overlapping genres that rank in the top 5 for both criteria.</p></li><li><p><strong>Second branch:</strong> Comparing Runtimes Across Genres. An <em>Expression </em>node categorizes each movie into runtime groups based on duration: "Short Film" (&lt; 60 minutes), "Standard" (&lt; 150 minutes), and "Epic". A <em>Pivot </em>node then calculates the average runtime of movies by genre and runtime category. We use a <em>Row Filter</em> node to narrow our focus to specific genres (“Documentary”, “History,” and “Crime”), assign different colors to each genre using a <em>Color Manager</em> node and lastly visualize output with a <em>Bar Chart</em> node for easier comparison.</p></li><li><p><strong>Third branch:</strong> Average Rating of Genres Over Time. To analyze rating trends, a <em>Pivot </em>node calculates the average movie rating grouped by decade and genre. We apply a <em>Column Filter</em> node and choose “Action”, “Crime”, “Romance”, and “Music” columns. Lastly, we connect the aggregated data to a <em>Line Plot</em> node to visualize how audience ratings have changed over decades.</p></li><li><p><strong>Fourth branch:</strong> Hidden Gems. To uncover overlooked masterpieces, i.e., movies that are highly rated yet not extremely popular, the original dataset is connected to a <em>Row Filter</em> node that filters out low-quality or obscure entries. We filter the data as follows:</p><ul><li><p>“vote_count” between 100 and 2000: To ensure that the movies have a wide enough audience and at the same time are no Hollywood legacies,&nbsp;</p></li><li><p>“vote_average” higher than 7.5: To ensure we are generally looking at highly rated movies, and&nbsp;</p></li><li><p>popularity less than 40: To ensure that massive blockbusters will not dominate the chart.&nbsp;<strong><em>Note:</em></strong> Here, popularity refers to a real-time hype meter measuring how many people are actively talking about, searching for, and logging the movie right now.&nbsp;</p></li></ul></li></ul><p>After filtering the data, we use an <em>Expression </em>node to apply a custom formula to calculate the <strong>Hidden Gem Score </strong>(HGS). Note that there’s no official formula for discovering hidden gems. We have used a formula inspired by TF-IDF from document analysis that penalizes high popularity and rewards high ratings:&nbsp;$["vote_average"]/(log10($["popularity"]+10))</p><p>Lastly, we use a <em>Top k Row Filter</em> node to pick the top 25 and visualize the resulting table with <em>Table View</em> node.</p>