Viewing a single comment thread. View all comments

JustBrowsing1989z t1_iy0vasf wrote

How does it work exactly?

Is it just based on keywords or is it actually accessing the internet to understand the queries?

For example, if I ask for "songs that played during fight scenes in movies", what does the AI system actually do?

7

notspoon OP t1_iy0yn0v wrote

High level overview:

The sentence embeddings are calculated using a Bidirectional Encoder Representation Transformer (BERT) model. There's a pre-trained model for this network trained on over 1 billion sentences from the internet that is publicly available, (thanks Microsoft) . The model transforms your description into a 784-long list of numbers (a vector) that represents the meaning of your sentence.

The model runs off a dataset of musical metadata for 35,000 songs. This metadata is very rich, it has a lot of useful columns like the genres, subgenres, and descriptions of tracks. The numerical data is binned into categorical values like "obscure" mapping popularity between 0 and 10, "highly danceable" mapping danceability between 80 and 100, etc. The text data is modified into a coherent sentence: "this song's main genres are _____. this song is from the 80s. this name of this song is lovefool by the cardigans. etc"

Each feature for each song in our metadata dataset is now a big paragraph that describes the song overall. The paragraph is split up into sentences, and the embedding of each sentence is found. The final embedding for each song is then calculated by taking the mean all sentence embeddings from the big paragraph.

To make your playlist, all that has to be done is compare the embedding of your query all 35,000 embeddings in the dataset and return the 100 most similar queries, using the cosine similarity distance metric. Thank god we have computers.

Once the 100 most similar tracks are found, your playlist is made by sending the Spotify track IDs through their API, and the link is generated for you.

38

JustBrowsing1989z t1_iy10hru wrote

Thanks for that! Seems very complex

And who provides that dataset? Is it Spotify? Or does it come from the labels or artists?

3

ryan__fm t1_iy3q0st wrote

Not OP but the data points he mentioned are available via Spotify's public API, so that would be my guess.

2

JustBrowsing1989z t1_iya6y9c wrote

That's fascinating stuff!

Amazing they can detect all that programmatically

1

thenerdyn00b t1_iy2k6rc wrote

That's awesome... Well I had the same idea from the site Spotify www.everynoise.com Generating the genre cloud through embedding, and comparing it through cosine similarity..

2