Telegram turned my media consumption habits and preferences upside down with the rollout and popularization of ‘channels’. Channels are essentially just curated message boards, where only one party can post on the main board, while others might be able to comment/interact with it in a different way. Channels reporting news are extremely popular on Telegram, getting millions of subscribers. Consuming news through snippets that someone carefully tailored to your political bubble’s status quo is just much more fun than reading long articles. Also it drives engagement growth.

Despite now reading news snippets, you still find yourself spending just as much time as you would were you to read the original article. That is because news happens all the time, and with a shorter pipeline from ‘spawned’ to ‘reported’, more and more things are now being posted that are not so newsworthy. All the news channels I am subscribed to produce more than 300 snippets per day combined, which is almost as much as Kanye posts on Twitter. Some of the problems it creates:

• You don’t have time to critically assess what you read
• You clutter your mind with bullshit unimportant things
• You waste time
• …

Conceptual Model

Heavily inspired by newsminimalist and my own laziness, I created a Telegram bot that takes news posts from channels you select, analyzes their importance and spits out only the highest-performing ones. A rough conceptual model:

In words: through Telegram’s frontend, the user selects the channels they want to subscribe to. These channels are stored in a single table in a local SQLite database. Every morning, a Cron job activates, pulling all the posts you missed the previous day. Those posts are cleaned and turned into arrays of numbers, which are then classified as either worthy or unworthy of your attention. Finally, the posts are sent back to the user.

The project turned out to be a bit larger than I expected, so I am dedicating two posts to it. In this one, I will talk about data collection, preprocessing and the machine learning behind it.

The Data

I needed a lot of training data, so I chose six news channels that had consistently delivered news in a formal or semi-formal format. The channels are ‘Раньше всех. Ну почти.’, ‘РИА Новости’, ‘Varlamov News’, ‘Медуза — LIVE’, ‘BRIEFLY’, and ‘Ньюсач/Двач’. From each channel, I extracted 10,000 most recent posts with the following functions:

from telethon import TelegramClient
import asyncio

async def fetch_messages(channel, limit):
    """ Fetch messages from single channel."""
    try:
        message_list = []
        async for message in client.iter_messages(channel, limit):
            message_list.append(message.text)
        print(f'Finished fetching messages from {channel}')
        return channel, message_list
    except Exception as e:
        print(f'Error fetching messages from {channel}: {e}')
        return channel, []

async def main():
    """Fetching messages from all channels."""
    try: 
        post_history = {}
        for channel in channels: 
            dialogs = await client.get_dialogs()
            channel, messages = await fetch_messages(channel, limit)
            if messages: 
                post_history[channel] = messages
        with open('./data/post_history.json', 'w') as f:
            json.dump(post_history, f, ensure_ascii=False)
    except Exception as e: 
        print(f'Error: {e}')

Afterwards I cleaned them, removing links, deleting markdown symbols,etc., and was left with around 50,000 posts.

Scoring 50,000 News Articles on Importance

I left data labeling to LLMs, once again drawing heavy inspiration from newsminimalist. I used the following prompt to generate multiple scores for each snippet:

Instruction:

Analyze the following Russian news snippet and provide scores for each of the following dimensions. Please respond in the format: “Dimension: Score” (e.g., “Scale: 8”). Ensure your response is concise and only includes the dimension name followed by a score.

Dimensions:

1. Scale: How broadly the event affects humanity.
2. Impact: How strong the immediate effect is.
3. Novelty: How unique and unexpected is the event.
4. Potential: How likely it is to shape the future.
5. Legacy: How likely it is to be considered a turning point in history or a major milestone.
6. Positivity: How positive is the event.

   Input:

   [Insert Russian news snippet here]

   Response Format:

   Scale: [Score]
   Impact: [Score]
   Novelty: [Score]
   Potential: [Score]
   Legacy: [Score]
   Positivity: [Score]

Final score weighs the factors and is calculated with the following weights and function:

coefs = {
    "Scale": 0.25,
    "Impact": 0.25,
    "Novelty": 0.05,
    "Potential": 0.25,
    "Legacy": 0.15,
    "Positivity": 0.05
}

def calculate_scores(responses):
    final_scores = []
    for score in responses: 
        all_scores = re.findall(r'\d', score)
        all_scores= [int(s) for s in all_scores]
        final_score = round(sum([coef * all_scores[i] for i, coef in enumerate(coefs.values())]),1)
        final_scores.append(final_score)
    return final_scores

These weights reflect what I personally expect from my newsletter, so you might find them somewhat arbitrary. That is, however, not the main problem with my approach. To radically decrease investments into this inherently net-negative project, I opted for a smaller, cheaper LLM: ChatGPT-4o mini. This later comes back to bit me in the ass, as I will mention. That said, I did conduct multiple sanity checks to ensure the scores made any sense - and they did! Here are a couple of examples (translated from Russian to English):

ChatGPT-4o successfully ranked both commercials low, while boosting the score of the important snippet. Seeing these scores, one might think we are close to AGI, since even a low-cost model understands that Slovakia’s politics is unimportant. Despite many good hits, the model, just like an average Internet user, lacked critical thinking and ranked especially arrogant commercials, lies, and random things highly:

Below you can find the distribution of the scores. Something like a normal distribution with a mean of around 6.5 can be observed. I believe the average is higher than what newsminimalist reports because a cheaper model is much more impressionable and, as I already mentioned, can fall for simple tricks.

Bad scoring practices create the shit in, shit out problem: if the scores lack intrinsic insight, no model will be able to work with them effectively. This does not have much impact on my project much since I am just happy to be here [and showcase my skills].

Quick Dive Into the Data

Below is a wordcloud I created from the entirety of the data that I gathered. I lemmatized the text before to get a more general view of the corpus. For the English speakers: most popular words were “year”, “say”, “also”, “Russia”, “Trump”, which makes sense.

To lemmatize means to find the ‘lemma’ or root form of a word.

Now let’s take a look into the sizes of posts. The distribution is heavily skewed to the right, with most posts having around 20 words in them. A small bump at around 300 is caused by the channel ‘BRIEFLY’, since they group snippets together and post larger messages. There is a moderate correlation between post size and post score, with a Pearson correlation coefficient of 0.37. I believe it is caused by posts from ‘BRIEFLY’ getting ranked highly since they contain more information in a single post.

Pearson correlation is a measure of a linear relationship between two continuous variables.

Translating Human Language Into Numbers

Collecting scores only gives us the response variable, and we still need to use something as a predictor. We cannot, unfortunately, use documents as they are for model training - we need to translate them from human language into a format that machines understand, i.e., numbers. I applied two methods of calculating document embeddings: training own Doc2Vec model and using a pre-trained BERT model. I trained my own Doc2Vec model and used the pre-trained BERT model rubert-tiny-turbo to generate embeddings. I chose it because of its outstanding size/performance ratio. Benchmark results can be found here.

Doc2Vec is an extension of Word2Vec, which embeds words in a vector space using a shallow neural network. Doc2Vec we act as if all documents have another word-like vector which contributes to all training predictions and it a doc-vector.

BERT is a language model that understands text by considering the context of each word in both directions. It uses the transformer architecture, which applies attention mechanisms to capture relationships between words regardless of their position in a sentence.

Training a Model to Predict Importance

Essentially, the problem I need to tackle is classifying news snippets as important or unimportant. Therefore, to keep things simple, I decided to choose a threshold value of 6 and fit a logistic regression model to the data. Doc2Vec requires two important hyperparameters: vector size and epochs (i.e., how many times the model goes through the corpus). To find the optimal values, I fitted a logistic regression for multiple combinations, with the best one achieving a balanced accuracy of 71% and an F1 score of 0.65. Embeddings generated by a pre-trained BERT model showed much better performance, with accuracy of almost 80% and F1 score of 0.75. F1 score is calculated as follows:

Doc2Vec (test data):

rubert-tiny-turbo:

Creating the Bot

The Telegram bot that we need to make is rather simple, we just need to account for various situations (e.g., a user wants to resubscribe to a channel, no news marked as important, etc.) Full code can be found here.

Closing Thoughts

In the future, I would like to re-do this project with proper data labeling and model training. I believe that a ‘great content filter’ would be valuable and could be implemented in various ways across different platforms. If you feel like participating, either through labor, funding or in any other way, feel free to reach out.

A little over a decade ago, a new genre emerged from the depths of the Russian rap scene. As of now, it has no name (to my knowledge) and can take many forms, though it usually appears as rap. The genre’s distinctive feature is highly conceptual, linguistically experimental, neologism-heavy lyrics. Examples of artists include Ezhemesyachnye, Husky, makulatura, Krovostok and Pasha Technique. I think a name like “solyanka” can stick. Solyanka is a common dish, or rather, a culinary disaster of one, in post-Soviet countries, whose name became a phraseological unit meaning ‘a mix of everything’. In this context, solyanka is used as a metaphor for content-rich. One could argue solyanka the dish is as abstract in its contents as the music of the aforementioned bands.

Apple Music was not able to reliably distinguish between rap and solyanka, often creating poorly assembled playlists for me. Classifying solyanka versus not solyanka is a rather thrilling challenge, but too narrow for a master’s thesis. Instead, I decided to explore whether modeling lyrics — the defining feature of solyanka — could improve the performance of content-based music recommendation systems in general.

Why It Matters

Predicting which songs a user might like based on the listening histories of similar users generally yields a better result than using only content-based methods (audio, metadata, etc.). However, when songs have few or no streams — which is common for solyanka — content-based methods are the only viable option for recommendations. This makes it essential to find effective ways to represent song content in a computer-friendly manner.

To achieve this, I created multiple variables to model the complexity of song lyrics. Four of these variables significantly improved recommendation quality, including two which could be directly linked to solyanka: MTLD and lexical sophistication.

• MTLD (Measure of Textual Lexical Diversity) is calculated relatively simply and measures the proportion of unique words in a song. It is widely used in linguistic assessments and has applications in monitoring language impairments. Interestingly, MTLD also proved useful in predicting songs a person might enjoy.
• Lexical sophistication is the percentage of words not found in the 5,000 most commonly used words. I used the same method that Grammarly employs to calculate its sophistication index.

You can find the list of all variables in my paper.

Why It’s Difficult (or Nearly Impossible)

Solyanka, however, is not characterized just by lyrical complexity. It is a genre deeply rooted in the Russian culture and mindset, both of which are arguably impossible to model. In music recommendation literature, this phenomenon is called the Semantic Gap.

“The semantic gap is the discrepancy between what can be extracted from music (i.e., semantic or audio features), and high-level human perception of these features”.

One way to bridge the semantic gap is by modeling as many features as possible. In an ideal scenario, a data-omniscient supercomputer could fully capture the intricacies of solyanka, eliminating the gap altogether.

Feature Importance and Weighting (Not Included in the Paper)

One major idea I had was to account for the varying importance of features for different users. While this is not groundbreaking, it becomes a challenging task when a user has only a few streamed songs. In my dataset, most users had under 38 unique streamed songs. With over 70 features and only 38 observations, calculating feature importance became a major challenge due to the curse of dimensionality and the risk of overfitting.

For song similarity calculations, I used weighted cosine similarity as implemented in scipy:

To determine weights for individual variables, I used a transformed version of their feature importances/coefficients, calculated through a classification task: predicting whether a user listened to a song more than the average number of times.

Finding the optimal weights for each variable posed another challenge. A naive approach like random search was infeasible: with just 10 variables and 10 possible values each, the search space ballooned to 10 billion combinations. Instead, I needed a more efficient approach based on available user data.

Below are the R implementations for linear SVM and XGBoost used to calculate user-specific feature weights:

calculate_user_weights <- function(user_song_profiles, user_utility_vector) {
  set.seed(7)
  data <- data.frame(cbind(user_song_profiles, user_utility_vector))
  colnames(data)[ncol(data)] <- "utility"
  data$utility <- ifelse(data$utility >= mean(data$utility), 1, 0)
  data$utility <- as.factor(data$utility)
  
  classifier = svm(formula = utility ~ ., 
                   data = data, 
                   type = 'C-classification', 
                   kernel = 'linear',
                   class.weights = "inverse")
  
  weights_vector <- t(t(classifier$coefs) %*% classifier$SV)
  
  set.seed(NULL)
  return(normalize(weights_vector))
}

calculate_user_weights <- function(user_song_profiles, user_utility_vector) {
   set.seed(7)
   data <- data.frame(cbind(user_song_profiles, user_utility_vector))
   colnames(data)[ncol(data)] <- "utility"
   data$utility <- ifelse(data$utility >= mean(data$utility), 1, 0)
   data$utility <- as.factor(data$utility)
   class_freq <- table(data$utility)
   num_samples <- nrow(data)
   num_classes <- length(unique(data$utility))

   data$weight <- num_samples / (class_freq[as.character(data$utility)] * num_classes)
   data$weight <- ifelse(data$weight == max(data$weight), data$weight*10, data$weight)

   weights <- data$weight

   xgb <- caret::train(
     utility ~ .,
     data = data[, -ncol(data)],
     method = "xgbTree",
     weights = data$weight,
     trControl = trainControl(method = "cv", number = 5),  Using 5-fold cross-validation
     tuneGrid = expand.grid(
       nrounds = 1000,
       max_depth = 1,
       eta = 0.025,
       gamma = 0,
       colsample_bytree = 0.2,
       min_child_weight = 0,
       subsample = 0.7
     )
   )

   predictor <- Predictor$new(
     xgb,
     data = data[, -ncol(data)],
     predict.fun = function(object, newdata) {
       predict(object, newdata)
     }
   )

   imp <- FeatureImp$new(predictor, loss = "ce")

   weights_vector <- as.matrix(imp$results$importance)
   rownames(weights_vector) <- imp$results$feature

   return(weights_vector)
}

Results

Using SVM, the model showed significant overfitting, as shown in the MAP@500 scores:

While the weighted model performed worse on training data, it improved test set precision by 6%, suggesting potential benefits in real-world scenarios.

A large overfitting of the model can be observed in the results above with the large differences in MAP@500 between train and test. Moreover, while weighted model performed worse on train data, it managed to improve the precision by 6% on test data.

The data displayed is random!

The dashboard below was created using Python and the Streamlit open-source framework. Data is pulled from multiple sources, including the TikTok API and TONIC API, and is subsequently cleaned and transformed to fit the required format. The last column represents advice of whether to continue with a campaign or not based on the results of a machine learning model.

The dashboard is meant to display most relevant and recent data, improving reaction time of the buyer team. Therefore, it includes an ‘Update dashboard’ button that buyers can use to fetch new data outside of a predefined schedule.

The dashboard below was created using Evidence’s native scripting language and displays the performance of marketing assets. Data is fetched using ClickFlare API as well as with Selenium from theoptimizer.io. It is afterwards cleaned and queried using DuckDB dialect of SQL.

The dashboard is designed to reduce the time needed to analyze creative assets while conveniently displaying their performance over various periods.

In the thesis, I use listening history data from thousands of users across about 40,000 songs to show that certain linguistic features can improve content-based music recommendation systems. Below, you can find the full version of my work, and the code is posted on github.

In the next post, I will break down my thesis in simpler terms and share some extra thoughts that make it into the paper.

On mobile, click here to view the PDF.