Chris Mendez in Radio, NodeJS, For Developers, Tools for Software Engineers, Audio, Python, Ruby

Radio: Preparing WAV files for cloud-based radio

For the past few months, my team at USC Radio Labs has been working to solve a pretty interesting problem; how to convert 60k+ audio files from WAV format to something more internet-ready. To provide a little context, in the 1990's, many radio stations started moving towards automating their music programming. The benefits were clear: no skips, no repeated songs, no "dead air" and for music directors, a way to plan and strategize their music curation.

If you remember, the 1990's was the decade of Microsoft Windows. Everyone was making Windows applications and therefore, most of the high-end broadcast software was built on Windows using wav as their default file format.

For the most part, wav format is fantastic. It's like the perfect wrapper for storing music regardless of its encoding. For example, if you've ever worked with a zip file, wav is very similar to that. You can do much more with wav than just package a song.

The only problem is that wav is terrible for internet distribution. There are two clear problems. The first is that wav files are generally uncompressed and therefore large. A 3 minute song can average 40Mb. The second problem is that WAV isn't designed to store metadata information. So there's no way no add song title, composer, genre, year and all of those other goodies –unless you start gettting unecissarily creative [1].

Going back to my team, as we begin exploring the idea of building a "cloud-based radio station", we thought it would be important to learn how to convert our existing catalog into a new system that better prepares us for the future.

The goal was to convert wav to another uncompressed, lossless format, that is still high-quality, yet capable of handling ID3 data and open source.

The rest of this article will explain:

  • How we converted wav file to flac
  • How we automated the process of finding metadata using our source material.
  • How we injected that metadata into flac.
  • How we created Mp4 renditions.

Hello FLAC

FLAC is a technology that offers most of our key benefits including:

  • Lossless audio compression.
  • Slightly smaller file sizes.
  • ID3 metadata is a first-class citizen.
  • License free
  • Mac and PC friendly
  • Supported by a strong open source community
  • Good suite of command-line tools

Step 1 - Creating a Process

From WAV to FLAC, a love story

One of my first tasks is figuring out how to convert data from WAV to FLAC. Thankfully, there are many options available but I chose a tool that would allow me to create a scriptable, automation-friendly process.

As a starting point, I'm going to use ffmpeg to convert my WAV file to FLAC format.

The only problem is that beets doesn't do well with WAV files. Therefore, our first step is to use ffmpeg to convert a WAV file to FLAC. Unfortunately, there is very little WAV support on beets so we'll need to run this command on its own.

Once you have a FLAC file, you can run some fingerprinting.


Since I'm on a Mac, I installed ffmpeg command-line through Homebrew package manager to help with the conversion.

In order to use the script below, you must first have Rake and Homebrew installed.


Create a file titled Rakefile and paste this code.

namespace :audio do
  desc %Q{ Install ffmpeg encoder. }
  task :install do
    # https://gist.github.com/predakanga/2376835
    sh %{ brew install ffmpeg boost taglib }    
  end
  
  desc %Q{ Encode WAV to FLAC }
  task :convert, [:input, :output] do |task, args|
    input = args.input
    output = args.output
    sh %{ ffmpeg -i #{input} -c:a flac #{output} }
  end
end

Download and install ffmpeg.

rake ffmpeg:install

Convert a WAV file to FLAC.

rake ffmpeg:convert[/path/to/file.wav,/path/to/file.flac]

Here are a few other examples showing you how to convert files to different sample rates


Step 2 - Injecting ID3 data

Now that my WAV files are in FLAC format, I now want to figure out a way to find the metadata and inject it into the file.

Considering that I have 60k files, the only plausible way to automate this is to use the type of fingerprinting technology you see on products like Shazam.

Ideally, I also want to somehow scour the Internet and find the album art for each song. Here's how we can do both using beets.

Getting Started

So here's the deal, Music Fingerprinting is an amazing technology with incredible benefits but it requires a little bit of configuration.

Since I work in radio, it totally makes sense for me to learn the intricacies of creating a Shazam-like tool but for most people, the barrier might be a little too high. Therefore, I've created a script that automagically installs the necessary software for you.

You can choose to install the necessary libraries yourself by visiting the docs

Add these new namespaces to your Rakefile.

namespace :fingerprint do  
  desc %Q{ Start Fingerprinting }
  task :install => [:dependencies] do
    puts "Great! We're ready to get started."
  end
  
  desc %Q{ Install Fingerprinting Dependencies. }
  task :dependencies do
    # Install Chromaprint's tool fpcalc
    Rake::Task["fpcalc:install"].invoke("~/Desktop/fpcalc.tar.gz")
    # Install flac library
    sh %{ brew install flac }
    # Install pyacoustic so that we can interact with fpcalc from pythong
    sh %{ pip install pyacoustid }
    # Install beets
    sh %{ pip install beets[chroma] }
  end
end 

namespace :fpcalc do  
  desc %Q{ Download fpcalc. }
  task :download do |task, args|
    source  = "https://github.com/acoustid/chromaprint/releases/download/v1.4.2/chromaprint-fpcalc-1.4.2-macos-x86_64.tar.gz"
    dest    = args.dest
    # Remove any previous downloads
    sh %{ rm -f #{dest} }
    # Download it as a new filename
    sh %{ curl -L #{source} -o #{dest} }
  end

  desc %Q{ Install fpcalc. }
  task :install, [:dest] => [:download] do |task, args|
    dest = args.dest
    # Unpack store the executable in /usr/local/bin
    sh %{ tar -xzvf #{dest} -C #{Dir.home}/Desktop --strip-components=1 }
    # Move fpcalc to /usr/loca/bin
    sh %{ mv fpcalc /usr/local/bin/fpcalc }
    # Double check your work.
    sh %{ which fpcalc }
    # Remove the desktop download
    sh %{ rm -f #{dest} }
  end
end

Now all you have to do is open up Terminal and run this command.

The command will install flac, pyacoustid, and fpcalc.

rake fingerprint:install
  • fpcalc is an open source fingerprinting technology is available within a command-line package called fpcalc. If you're on a mac, download fpcalc library and copy the executable file to your /usr/local/bin.
  • We need pyacoustid package to interact with the Chromaprint library from Python.
  • beets is a command-line tool for python that can automatically catalog your collection, transcode audio files, check for duplicates and help you fingerprint songs through chromaprint. We are also including chroma plugin.

Configuring Beets

Now that we've installed all the necessary packages, we now need to configure the beets application.

You will first need to configure beets so that you can install plugins.

Create a new config.yaml.

beet config -e

Here is an example of a config file. The file is located in ~/.config/beets/config.yaml.

# Copy cleaned-up music into that empty folder using beets’
directory: ~/Desktop/my_beets_app/my_music_folder
# Location of Beets music database
library:   ~/Desktop/musiclib.blb

import:
  # If no data is available, just move on
  quiet_fallback: skip
  # Do not copy songs to another directory
  copy: no
  # Inject ID3 data into the song if found.
  write: yes

# Use the Chroma plugin
plugins: chroma

chroma:
  # When you type beets import, the auto fingerprinting will begin.
  auto: yes

# You can do multiple processes at once. This is somewhat CPU intensive.
threaded: yes
# Show the command line
verbose: yes

Download a WAV file

For this example, let's use a Creative Commons WAV file from Nine Inch Nails titled "999,999".

Music Fingerprinting - Finding a Song's info based on it's "sound"

Now that our development environment is ready. Let's try and fingerprint a WAV file that contains no metadata.

Acoustic fingerprinting is a technique for identifying songs from the way they –sound– rather from their existing metadata.



Step - Import Music

Single track

Import a single track and include a log file.

beet import -A -C -s -l my_log_file.txt  my_music_folder/album/02.flac 
  • -A do not autotag anything, just import the files.
  • -C do not copy files to a new directory.
  • -s is the singleton method we should use when we're trying to access a single file.

Source

Entire Albums

Import a single track and include a log file.

beet import -A -C -l my_log_file.txt  my_music_folder/album/02.flac 

Step - Beet Queries

Beet Queries

List out the music you've imported.

beet list

Resources


  1. Resource Interchange File Format (RIFF) ↩︎