In early 2017 I wrote a silly Python script called Wall of Sound that chose desktop wallpaper for you from Flickr which was based on the lyrics of the currently playing song. Why? Well, why not? Music has always inspired visual art[1], and while machines can’t really interpret art, perhaps a few algorithms can help the listener funnel artistic appreciation from one medium to another.
The process is simple: first get the lyrics for the currently playing song on your computer, and from that extract key words which can be sent to Flickr to retrieve Creative Commons licensed images that are associated with those words. We sort the search results by interestingness, a metric derived from Flickr’s proprietary algorithm based on user interaction with each others’ photos. The selected top images have been found to yield much better quality than simply randomly selecting pictures[2]. We then grab one of the top few results that rank highest for interestingness, download it, and set that as the desktop wallpaper.
I haven’t used that script in a while, so when I tried running it again it didn’t work. For one thing, I used the IBM Alchemy web API to get keywords from song lyrics, and that service has since been discontinued. Another problem is scraping the web for lyrics using a Python library isn’t all that reliable. Also, Flickr itself seems to be littered with creepy screenshots of weird 3D figures from the Second Life video game, and while your mileage may vary on what constitutes a good image, I certainly didn’t want those pictures to adorn my desktop. Finally, I’d like to add Spotify support to the program. I have an extensive CD collection (support your favorite musicians, buy CDs!) but computers seem to be eschewing CD drives these days and I find it harder to play them, hence the reason I use Spotify.
So I set out to solve all those problems until my silly wallpaper hack worked again. If you want to get straight to the code, the link is here.
Getting Keywords From Song Lyrics
The first step in our process is to get the lyrics of a song which we can then extract keywords from. I signed up for the Genius API, backed by a company that was founded around analytics of song lyrics (but aims to be much more). The API is quite powerful, and the concept of song annotations (where lyrics are discussed in a crowdsourced format much like Wikipedia) is really appealing to me, but for the time being all I needed was just the lyrics, and the API works very well for that purpose. Is it perfect? Not really. Very rarely, you get the wrong song, but it works very well almost all the time, and that is good enough for me.
Next up is keyword extraction. Since IBM Alchemy has been retired, there does exist a replacement service called Watson Natural Language Understanding which I decided not to use for two reasons. First, it was overkill, and more importantly, the less we rely on third-party web API’s the better.
To get some phrases from any given song, we turn to a Python library called RAKE which stands for Rapid Automatic Keyword Extraction. It’s a Python implementation of a keyword extraction algorithm described by Rose et al. in 2010 [3]. The gist of the algorithm is that it first removes stopwords from a text, then ranks phrases based on both word frequency and co-occurrence. The RAKE library lets us specify the maximum length of candidate phrases (or n-grams where n is the number of words in a phrase) and also the maximum number of phrases we want to extract. Therefore I set those values as run-time tunable variables, so we can play around with the n-gram value and maximum number of phrases to see different results.
Whenever a song has no lyrics, the Wall of Sound script resorts to just using the song title as search parameters.
Filtering Flickr Results
For the most part, searching Flickr for interesting pictures is pretty straightforward. Flickr seems to have lost some of its luster in the past couple of years though, but there is still a really good pool of quality images out there. I restricted the search to Creative Commons only to avoid any possible thorny issues of downloading copyrighted images without permission because the Creative Commons licenses all permit copying by default.
One unexpected problem that cropped up was Second Life screenshots uploaded to Flickr were coming up in the search results. They aren’t photos, but instead game screen captures of weird, poorly-rendered 3D characters, and unless you fancy a creepy trip to Uncanny Valley, you don’t really want these pictures on your desktop. The API doesn’t seem to be able to exclude them from the search results, so I had to get creative.
The search is done in two stages, the first will be fed keywords found by RAKE into the Flickr search API with the requested results sorted by interestingness and licensed with Creative Commons. Then we take the set of results and check each of the image dimensions to see if they’re large enough (using tunable width and height values). If the image passes the dimensions test, we then make another Flickr query for the image to retrieve the EXIF data. If the EXIF data has a populated ‘camera‘ field, we know this has to be a real photograph and not a screenshot. Then the image is added to the set of candidate photos, and if it’s not an empty set, a random photo selected from that set of photos is chosen for the desktop wallpaper.
Spotify Support
The initial version of Wall of Sound used a simple BASH script that got song titles from the local mpd music playing daemon. A fork of my code on Github by Luca Coviello added Spotify support (as well as OS X support) with a ZSH script, but I wanted to take a slightly different approach.
I first reworked the Wall of Sound main program as a web service using Flask. A client could send a song title and artist to it, and it’ll get an URL of a Flickr image as a result. That way, the client is free to extract the currently playing song from any program, and set the wallpaper in any way it wants independently of the song analysis and Flickr search part. It could even run on a network and serve multiple clients on different operating systems.
Next, I would write a Linux Spotify-based client which would get the song title from Spotify, but how best to implement this? One way would be to get a Spotify API key and repeatedly poll the Spotify server to see if a song is playing, or if it has changed. This, however, would rely on yet another external API key and makes too many network requests. Therefore I thought up a hack; since the Spotify client sends a message to the desktop manager whenever a new song plays, why not snoop DBUS on Linux to see when that happens and grab the song title and artist off that message? The resulting Python code works even when you play Spotify on another device, as long as the Spotify client is open on the desktop machine.
Sample Runs
Now for the fun part! I ran the program using the same selection of songs as I had done in my blog post from 2017, and some new ones. Let’s see what came up.
First, I tried Here Comes The Sun, the classic song of hopeful optimism written by George Harrison from The Beatles’ hit album Abbey Road from 1969. The keywords chosen were “slowly melting”, and we got this:
The phrase “slowly melting” appears in the description of the image, and indeed it was ice and snow. It’s interesting how we got there from just that phrase, and I love the wildcat in the photo.
Next up is a song called Keep Running by Gemma Hayes from her 2011 album, Let It Break. The song repeatedly lists a few place names like London, Tokyo, and Seoul and I figured they would come up in the list of phrases but nope, what I got was “turn around” and it gave me the following image.
The phrase here is “the turn”, which is not quite “turn around” and yet the word association just worked, and kind of reflects the mood of the song.
The next song is Fire Engine by Aoife O’Donovan, from her debut solo album from 2013, Fossils. The Genius API did not give me any lyrics to this song, so the program just fed the title into Flickr’s search and we get… a literal fire engine.
This is where it really pays off sorting the results by interestingness; we got a well-composed shot of a cool retro fire engine instead of any old image that fits the bill.
Next I tried Stairway To Heaven, the hit Led Zeppelin song from their 1971 album, Led Zeppelin IV. The phrase chosen was “still time” and the image returned was the following.
Admittedly, this is rather random since we had a very short phrase to go by but the various mentions of gold in the song (though not in the search parameters) makes me think this still kind of works, even if it’s a fortuitous coincidence.
Next, I decided to try some poetry set to songs. Irish poet Thomas Moore’s Last Rose Of Summer from 1805 was fed into the program by way of Eleanor McEvoy’s recordings of Moore’s song-poems in her 2017 album, The Thomas Moore Project. The phrase chosen was “left blooming alone” gave me a picture of a rose.
The title of the picture by Tim Green is “left blooming alone”, and it may even be a direct reference to the Moore poem, so that was a really good match.
Next I tried an old Jimi Hendrix favorite, Little Wing, from the 1967 album Axis: Bold as Love by the Jimi Hendrix Experience. The chosen phrase was “little wing”, which is from the lyrics itself, not the title.
Again, a rather literal choice but it’s a pretty bird so I’m not complaining.
There were instances when the program failed, such as when I played David Bowie’s Beauty And The Beast, originally from his album Heroes in 1977, but the version I tried playing on Spotify was the extended 2017 remaster. The full title was “Beauty and the Beast – Extended Version; 2017 Remaster” and feeding those words into Genius gave me, I kid you not, a very long excerpt from James Joyce’s Ulysses. This is a bizarre result, but the original Heroes is not on Spotify, and you can only play the remastered version.
The program also doesn’t work very well for certain older music from the Baroque/Classical/Romantic eras, with a lot of it not having any lyrics and with titles that just aren’t Flickr search friendly. Feed it Mozart’s Symphony No. 40 in G Minor, K. 550: I. Molto allegro and Flickr returns nothing. However, something like Bach’s Toccata and Fugue in D Minor which crops up all the time in pop culture does get you some results.
A picture of an organ! It’s kind of funny, but then again Toccata and Fugue in D Minor is frequently associated with organs, and spooky things (a repeated run gave me a picture of creepy Halloween masks).
Conclusion
The updated Wall of Sound program is a decent improvement over the original. Does it always give a relevant picture? No, but the results, when we get some, are almost always interesting, and it’s a ton of fun to play with. The code is available at my Github page, so you can try it yourself. The version described in this blog entry is 0.3-beta, but the main branch may have changed depending on when you read this. If you do try it, let me know what you think!
References
- J. Arn, “How Music Motivated Artists from Matisse to Kandinsky to Reinvent Painting”, Artsy, 2019. [Online]. Available: https://www.artsy.net/article/artsy-editorial-music-motivated-artists-matisse-kandinsky-reinvent-painting. [Accessed: 28- Sep- 2021].
- K. Lerman and L. Jones, “Social Browsing on Flickr”, in Proceedings of the First International Conference on Weblogs and Social Media (ICWSM 2007), Boulder, Colorado, USA, 2007.
- S. Rose, D. Engel, N. Cramer and W. Cowley, “Automatic Keyword Extraction from Individual Documents”, in Text Mining: Applications and Theory, 1st ed., Wiley, 2010, pp. 1-20.