Twitter's Open Source Algorithm

Since taking over Twitter, the tech mogul has been teasing the release of "the algorithm," leading many to expect a comprehensive unveiling of the platform's promotion strategies. However, the release turned out to be more of a charade than a grand reveal. This post delves into Twitter's recommendation system, emphasizing that the model pipeline training has only been released, rather than providing insight into why specific recommendations appear on users' timelines, as many had expected.

The Deception

Musk's advocacy for open-sourcing the algorithm may have led some users to believe that it would reveal the weights or reasons for promoting specific Tweets. Twitter's recent open-source release of their recommendation system, on the other hand, only focused on the model training pipeline, leaving out the specific weights or criteria used to determine which Tweets are promoted. This distinction is critical because, while the release provides insight into the algorithm's overall structure and process, it does not provide complete transparency into the specific factors influencing the prominence of individual Tweets on the platform.

What has been made Public?

Twitter's recommendation system is a four-stage process that is similar to the one used at Shaped. The system works by gathering the most relevant Tweets from various sources (candidate sourcing), ranking each Tweet using a machine learning model, applying heuristics and filters, and finally building and serving the For You timeline via the Home Mixer. Although the released system provides valuable insights for developers on how to build a recommender system, it does not provide Twitter's internal data, which disappointed users who were looking for more transparency.

1. Sourcing Candidates

In-Network and Out-of-Network sources are used to find candidates. The In-Network source focuses on the most relevant and recent Tweets from users you follow, whereas Out-of-Network sources use Social Graph and Embedding Spaces to find relevant Tweets outside your network. Embedding Spaces computes numerical representations of users' interests and Tweets' content to establish content similarity, while Social Graph analyses engagements of people you follow or those with similar interests.

With the release of this component, some of the system's features have been revealed, as have rumours about Musk receiving special treatment for his tweets after taking over the company.

2. Ordering

The Ranking stage in Twitter's recommendation system is critical because it assigns a score to each Tweet based on its predicted relevance to the user. This process is carried out with the help of a neural network with approximately 48 million parameters that is constantly trained on Tweet interactions to optimize for positive engagements such as Likes, Retweets, and Replies. The model considers thousands of features and generates ten labels for each Tweet, representing the likelihood of various types of engagement.

The labels' scores are then used to rank the Tweets in order of relevance. The system ensures that the content displayed on a user's timeline is tailored to their interests by effectively ranking Tweets, resulting in a personalized and engaging experience on the platform.

3. Filtering

Various heuristics and filters are applied to the curated content in the Filtering stage of Twitter's recommendation system to improve the user experience by implementing product features that cater to individual preferences. These filters contribute to a more balanced and diverse feed, ensuring that users see a personalized and engaging timeline. Visibility Filtering, which removes Tweets from accounts a user has blocked or muted; Author Diversity, which prevents too many consecutive Tweets from the same author; and Content Balance, which ensures a fair distribution of In-Network and Out-of-Network Tweets, are some examples of filters used at this stage.

4. Preparing and Serving

The Home Mixer is in charge of combining these Tweets with non-Tweet content like Ads, Follow Recommendations, and Onboarding prompts. This combination ensures a diverse and engaging platform user experience. Once the Home Mixer has compiled the appropriate mix of content, it sends the finalized For You timeline to users' devices for display, providing them with a curated and personalized feed that corresponds to their interests and preferences.

Aftermath

While Twitter's open-source release of their recommendation system provides insight into the underlying mechanics of content curation, it is important to note that the weights that determine specific content appearances in the pipeline have not been disclosed, despite Musk's promises. This recommender system is critical in providing a more engaging and personalized experience for Twitter users, and its open-source release represents a step towards greater transparency.

With Hocalwire's social media auto-posting feature, any news items and articles that match the specified rule will automatically post on the selected social media sites.

So that you can concentrate on producing high-quality content and expanding your revenue opportunities, you need to understand how we create technically sound websites that take care of the SEO and indexing requirements. If you're searching for one, schedule a demo of Hocalwire CMS by booking a time.