The youth sports industry is booming right now (in good and bad ways, but that's a different story), and throughout it you're seeing technology that was once inapproachably expensive become a part of home setups and how young athletes train.

The edges of this space feel very much like the world in which Square first released its "little white card reader." Consumers suddenly had a computer in their pocket, and Jack and team saw the chance to use that to revolutionize how businesses could get paid.

s for me, we put a batting cage in the backyard, and hitting sessions now involve a Rapsodo Pro 2.0 launch monitor and a Blast Motion bat sensor, along with a couple phones and iPads. I couldn't help trying to make those pieces fit together.

For those not in the weeds of all of this, the Rapsodo Pro 2.0 is an example of a launch monitor. It contains camera and radar to observe the flight of the baseball. For pitching, that gives you amazing data on release points, spin rates, and velocity. For batting, it tracks similar details as the ball leaves the bat after contact. This tech is much more common in the golf world, where launch monitors are at the core of simulator setups. It's the same idea for baseball. The launch monitor is observing exit velocity, exit direction, launch angle, spin characteristics, etc. These are the outputs of the baseball swing.

The Blast Motion bat sensor is a tiny device that you slip onto the knob of the bat. It is an "inertial measurement unit," meaning that it can understand how it moves through space. Using clever math, it turns that into an understanding of the path of a baseball bat from the batting stance to and through the point of contact. From Blast Motion you get what I think of as the inputs to a baseball swing: bat speed, attack angle, rotational acceleration, and a variety of metrics on angles and planes.

My basic question starting out was how you could marry these input and output ideas to get a more complete view of a swing. Rapsodo and Blast both have apps and websites, so could I access both data sets and match them up to create one view? This also presented an excuse to dip my toe back into iOS development, first more manually and more recently with a heavy dose of AI.

In a CageStats swing view, you can see how inputs connect to outputs, as well as use the data we've recorded to visualize a trajectory path for the ball. I let Claude learn the physics here, not me.

This visual represents sort of the core of what I first set out to see: how does the input bat data connect to the output launch data? MLB StatCast gives a number of definitions that become useful here: the Ideal Attack Angle range is 5-20°; the Launch Angle Sweet-Spot is 8-32°. From a coaching perspective, the idea that you're trying to teach is that the best swing matches the trajectory of the incoming pitch, which will naturally be coming down some as it travels toward the plate. Matching that trajectory gives you the most time where the bat and the ball are on the same plane, and the most chance of direct solid contact on the hit.

Via the combined data, you can compute a really interesting metric, the Squared-Up Rate. From StatCast, this is "How much exit velocity was obtained compared to the maximum possible exit velocity available, given the speed of the swing and pitch." So pitch speed (Rapsodo) and bat speed (Blast) give you a theoretical max exit velocity, and you can look at actual exit velocity (Rapsodo) to figure out what percentage of the potential was achieved on the swing.

As for the boys, they tolerate my interests. When they have friends over to hit they enjoy watching each others' calculated distances in the Rapsodo app. They nod respectfully when I show them session rates for attack and launch angles after the fact. They pay attention when I scrub through videos comparing how their hips fire on a good swing vs a great swing.

SwingScrubber is a little app I built to take a video of a hitting session, allow the easy tagging of swing contact points, and then a way to compare two synchronized swings.

Honestly, I pretend this is all for them, but I played through high school and can totally see myself joining an old-man league in a decade when we're empty nesters. Don't ask how many more swings I've taken out there compared to them.

This is obviously just a tiny taste of the data-based approach to training pioneered by folks like Driveline, but it's also very much in the vein of what they're doing. The Rapsodo is still a relatively expensive piece of kit, but it's not actually that crazy in a world where youth two-piece composite bats at $500 a pop are lining every dugout fence, and it's not going to be that long until you get the same data off of an iPhone.

Plenty of apps—including Blast's—are out there trying to use the iPhone camera to act as a launch monitor. They aren't nearly as good as the Rapsodo yet, but the sensors will continue to get better.

The real explosion in this space is around AI, doing automated analysis of video to produce coaching tips, cut highlight packages from game film, and so forth. You name it, someone's out playing with it right now. SwingAI from WinReality (who bought Blast Motion in 2025), is a great example, though personally I'm not sure I'll ever be able to convince it that my Torso Load is great enough.

WinReality's TrainVR is also an amazing example of what becomes possible as VR headsets advance. Right now the Meta Quest 3s it uses is ok, not great, but I love the idea that this can give my 11 year old a way to train the recognition of what a curveball looks like before he'll ever see a high quantity of them in a game.

So if I start posting here about sports tech, it's not that I've quit moving money as a day job. I've just picked up a hobby, and am so interested to see where this moment takes us and what kind of innovation comes next.

I didn't even have a good way to deploy an update to change the sidebar to show that I've spent the last six years working for Square and Cash App rather than public radio.

Definitely time for a refresh. I'd been messing around with using a Docker container to get my ancient Ruby running on this M1 iMac, so I decided to shoot for ECS (Elastic Container Service), using Amazon-provisioned database and Redis.

Fair note here: All of this is overkill for this blog, which should really be hosted as a static site somewhere. We'll cross that bridge later. I figured getting it running correctly was step one to doing any other changes. Also, as you can see in the screen above, Amazon is getting set to retire the setup my old instance was running in and therefore putting a cap on how long I can be lazy.

I was pretty lost on exactly where the industry had moved here going in. I knew I could get the app running in a Docker container and that would take care of a lot of my environment issues, but I've spend the last six years just deploying apps onto the tooling that platform teams give me, not trying to keep up with developments in AWS.

I knew ECS was a way to run containers, but also saw mentions of Fargate and didn't understand the difference. TLDR: Fargate is just a way to deploy services into ECS without having to know about the EC2 instance underneath.

I struggled for a while to just get a handle on which AWS pieces I was going to need. This blog is a Rails app with a MySQL database and a Redis instance for caching. Traditionally it used a local filesystem to store image assets.

There are lots of "how to deploy a Rails app in ECS" blog posts, but this series of posts probably helped me walk through the concepts the best.

I needed to:

1. Build my image
2. Get it to a container registry
3. Have a database
4. Build out an ECS cluster with a task using my container
5. Have a load balancer point to the containers
6. Lots of other stuff

On the application side, I needed to get off that local storage and get assets hosted on S3 instead. Conveniently, I had apparently started that effort years ago and had an S3 bucket with a bunch of image files in it. Inconveniently, I had left it half-done and without any real notes on what was or wasn't working. Images on this site use an engine called AssetHost, which I wrote at SCPR and released as open source on Github. In my second stint at SCPR I had ported AssetHost to use S3 for a backend (we actually used a locally-hosted S3-compatible interface through Riak CS, but that's a different story), but hadn't ever fully merged that back to the main repo.

In the end, it's debatable whether any of this pre-work was actually helpful: it left me thinking these changes were closer to complete than they were, which lured me away from just making the smallest possible updates to get things running.

I couldn't ever figure out how to get Ruby 2.1.5 to build on M1 and/or modern OSX, but I used Docker Compose to get the app running in a container by taking Ubuntu 14.4 and compiling Ruby 2.1.5 inside of it. I built my own, but my Dockerfile for the ruby container looked very similar to this one.

My eventual Dockerfile for the app looked like this:

FROM ewr/ruby-2.1.5:0.0.1
RUN mkdir -p /app
RUN mkdir -p /rails
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN bundle install --jobs 20 --retry 5
RUN mkdir -p /puma/pids
RUN mkdir -p /puma/sockets
COPY . ./
RUN RAILS_ENV=production DATABASE_URL=nulldb://user:pass@127.0.0.1/dbname bundle exec rake assets:precompile
EXPOSE 3000
CMD ["bundle", "exec", "puma", "-C", "config/docker_puma.rb", "-p", "3000"]

Note on the asset compilation: In Rails 3 you could just add config.assets.initialize_on_precompile = false to not load the Rails environment when compiling. That didn't work for me, because a piece of the AssetHost javascript wants to know where the AssetHost engine is mounted to hardcode some paths. I needed the app to start up, but without any DB at this stage in the build. nulldb let me get there. Later I think I refactored away the need to initialize the app entirely, but I haven't revisited the step yet.

Eventually I got the app running in Docker, using s3 for asset storage, and everything was looking up.

Then I had to figure out the AWS stuff.

I made a first pass in February just clicking buttons in the AWS console, following the blog posts I linked above. They helped me understand some pieces, but I didn't get anything working.

This month, in the effort that finally stuck, I started from this Fargate / Terraform tutorial and actually got somewhere.

I've stuck my terraform code up on Github. I extended the tutorial to add RDS, Redis, a worker task, and some Route 53 bits around domain validation for the SSL certificate.

So what's not done yet?

I have no clue how health checks work, and am probably doing wrong things there.

I still have to manually build images and trigger a deploy. I need to hook into something from Github.

There's no reason a worker should be hanging around all the time for the once every few weeks I upload an image here... That worker should be a lambda, and my Resque queue should just be SQS.

I should stick Cloudflare or something in front of the images, rather than serving them directly.

And, as mentioned before, this should all just be statically generated.

But still... Progress, right?

A lot has happened in between my last posts in 2015 and now. The kid who had just been born in this 2014 post is now seven and in first grade. He's got a brother, Brent, who is five. We've moved houses twice, and since early 2016 I've been working for Square, where I took a job to do Ruby code and somehow quickly ended up in payments. Turns out, it's a fun space, and six years later I'm still learning it as I help teams on Cash App's Financial Platform figure out how to move money around the world.

I've been loosely thinking about bringing this site back to life since mid-2021, when I picked up an M1 iMac as my first non-work computer in a bit. My 2012 Macbook Pro had gotten claimed by Brent when the boys suddenly needed technology for virtual school in the fall of 2020. Copying things over to the new machine made me a bit nostalgic, so I figured I would just get the old code for this site running and pick off some changes that had been on my todo list for a decade or so...

Turns out, trying to resurrect old Ruby code on a different architecture isn't straight-forward. This blog is running a version of Ruby on Rails that came out in mid-2013, running on a Ruby version that came out in late 2014. The Chef cookbooks to provision the AWS EC2 instance I'm hosting this on were put together in late 2013.

It's a bit amazing that it all still works in production, but also kind of terrifying to get working again to migrate to something more modern. My goal would be to switch the blog content here over to a static site generator, but I still want the nicety of a web UI for composing and for uploading image assets. I've debated between just starting from the database content and trying to adapt it to some more modern platform, or trying to get the current Rails code up to date and then just adding a static output generator. I haven't gotten far on either approach yet, so TBD.

Step one: figure out how to deploy some slightly updated templates to fix my bio and some copyright dates.

Somewhere along the way I want to do the same thing to blogdowntown, the site I ran for years about Downtown Los Angeles. The site belongs to Southern California Public Radio today, but it's fallen off their radar and hasn't been online for the last few years. I have a database dump and I think enough image assets to restore most of the archives, but I just need to get enough code running to figure out a game plan.

So, sometime in the next six years or so?

That all works great, but there's been one nagging case I've wanted to clean up: We do use Comcast for television, and I like their Xfinity Stream app for listening to news while I'm getting ready in the morning. Because of using the AT&T Fiber link, though, the Xfinity app thinks I'm out-of-home, which limits my ability to watch in-progress recordings.

To fix that, I need to set the USG to route Xfinity traffic over the Comcast link, but leave other traffic doing its normal failover setup that prefers AT&T. That's not an especially hard ask, but it's been sitting on the todo list for a while.

When we moved in mid-2019 I turned on Comcast internet because it was the quickest to get going, but then quickly moved to using AT&T Fiber for our primary connection. I kept the Comcast internet active, though, because in this day and age redundant Internet connections seems

Update (2025-01-04): I've since switched to using DirecTV Stream for TV, but Irv emailed to note that Comcast has changed its IP detection ranges. He suggested adding 69.252.0.0 - 69.252.127.255 and 68.87.64.0 - 68.87.127.255, so I've updated the directions below. I also in the time since this post switched to a Gateway Pro, which has a Policy-Based Routing UI. Not sure whether that's for all the hardware now...

Original post continues:

Full confession: I actually did this once already. I followed various online tutorials and set up the firewall rule on the router, which worked great. But with how Unifi works, making hot updates on the running gateway only applies until a reboot. I didn't take the final step of configuring it all to be saved for future provisioning, so along the way my changes disappeared and it took me a while to come back to them and do it correctly.

I guess that's part of why I'm writing this up... If I ever lose my config again, this can be a guide for my future self to get it set back up.

Also, how useful is it really to have a dedicated backup internet connection? 99% of days it sits there entirely unused, but it has saved me a handful of times. Just this month AT&T went out for most of a day at a time where virtual school was going on. It was great to realize that a bit into the outage when AT&T texted me, having not even noticed the actual point of outage as the connection flipped over. I get paid good money to do work on a computer. Paying $50/mon as a backup seems justified.

The Goal

When you're using the Xfinity Stream app or the web interface, the logic does a check to see whether you're at home. The web app, for instance, hits https://xtvapi.cloudtv.comcast.net/inhome-restrictions/. When you're out-of-home (my AT&T connection), you get this:

{
    "_type": "InHomeStatusRestrictions",
    "_links": {
        "self": {
            "href": "../inhome-restrictions/"
        }
    },
    "inHomeStatus": "out-of-home",
    "currentRestrictions": [
        "out-of-home"
    ]
}

That causes the app to refuse to play certain content where licensing doesn't allow out-of-home streaming.

Could you man-in-the-middle that request somehow and just remove the restrictions? Maybe? I didn't try. Since I've got the Comcast connection, I just want to send Xfinity requests there. I have no clue if other parts of the stack enforce similar in-home constraints.

The basic idea here is to use "policy-based routing" based on the destination IPs, and target the correct interface on the USG accordingly.

Step 1: What IPs are we routing?

So what IPs do we need to route? If we run dig we see that there are a number of IP addresses and CNAMEs behind the hostname.

~: dig xtvapi.cloudtv.comcast.net

; <<>> DiG 9.10.6 <<>> xtvapi.cloudtv.comcast.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40875
;; flags: qr rd ra; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;xtvapi.cloudtv.comcast.net.    IN      A

;; ANSWER SECTION:
xtvapi.cloudtv.comcast.net. 73  IN      CNAME   xfinityapi-production.r53.aae.comcast.net.
xfinityapi-production.r53.aae.comcast.net. 45 IN CNAME xfinityapi-tango-production-active.r53.aae.comcast.net.
xfinityapi-tango-production-active.r53.aae.comcast.net. 45 IN CNAME xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net.
xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net. 45 IN CNAME xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net.
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.47
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.162
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.250
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.32
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.219.99
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.180
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.223
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.68

;; Query time: 18 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Jan 03 14:52:24 EST 2022
;; MSG SIZE  rcvd: 401

Instead of trying to target as few IPs as possible, my goal was to just end up with a range of Comcast IPs that was unlikely to break as they changed internal infrastructure. To arrive at that, I ran a whois 96.115.218.47. That gave us a good set of ranges:

# whois.arin.net

NetRange:       96.64.0.0 - 96.124.255.255
CIDR:           96.120.0.0/14, 96.112.0.0/13, 96.124.0.0/16, 96.64.0.0/11, 96.96.0.0/12
NetName:        CABLE-1
NetHandle:      NET-96-64-0-0-1
Parent:         NET96 (NET-96-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS7922
Organization:   Comcast Cable Communications, LLC (CCCS)
RegDate:        2008-02-21
Updated:        2021-01-25
Ref:            https://rdap.arin.net/registry/ip/96.64.0.0

For better or worse, I figured 96.120.0.0/14, 96.112.0.0/13, 96.124.0.0/16, 96.64.0.0/11, 96.96.0.0/12 was a good set to route. Update: And we later added 69.240.0.0/12, 68.80.0.0/13.

Step 2: Setting up the routing on the USG

There are a number of good guides to USG policy-based routing and how to customize the config.gateway.json required to persist the changes.

So why can't you just set up static routes in the Unifi UI? Apparently if you have load balancing turned on the load balance rules are applied before the static routing rules, making them unhelpful here.

To apply the routing you'll need to SSH to the USG. Settings -> Site -> Device Authentication for settings in the older UI, or Settings -> System -> Application Configuration -> Device SSH Authentication in the "New UI". You can add an SSH key to make logins easier. Once that's applied you'll SSH to the USG.

The basic premise here is that we'll set up an address group with the Comcast IP ranges, a static routing table that routes to the Comcast interface, and then a firewall rule that says "use the static routing table if the destination IP is in the address group."

I haven't figured out a great way to get the Comcast gateway address automatically in the rule (there are discussions on next-hop-interface, but caveats that it isn't a good idea). For the moment I'm content to grab it and hardcode it. On the USG:

admin@ubnt:~$ show load-balance status  
Group wan_failover
  interface   : eth0
  carrier     : up
  status      : active
  gateway     : 172.16.0.1
  route table : 201
  weight      : 100%
  flows
      WAN Out : 349000
      WAN In  : 8
    Local Out : 1502

  interface   : eth2
  carrier     : up
  status      : failover
  gateway     : 66.56.48.1
  route table : 202
  weight      : 0%
  flows
      WAN Out : 53345
      WAN In  : 0
    Local Out : 397

eth2 is my Comcast interface (wan2 in the UI), so the gateway IP I need is 66.56.48.1.

Now, to apply:

configure
set protocols static table 5 route 0.0.0.0/0 next-hop 66.56.48.1
set firewall group address-group comcast 96.120.0.0/14
set firewall group address-group comcast 96.124.0.0/16
set firewall group address-group comcast 96.96.0.0/12
set firewall group address-group comcast 96.112.0.0/13
set firewall group address-group comcast 96.64.0.0/11
set firewall group address-group comcast 69.240.0.0/12
set firewall group address-group comcast 68.80.0.0/13
set firewall modify LOAD_BALANCE rule 2500 action modify
set firewall modify LOAD_BALANCE rule 2500 modify table 5
set firewall modify LOAD_BALANCE rule 2500 destination group address-group comcast
set firewall modify LOAD_BALANCE rule 2500 protocol all
commit
exit

Once it's committed, if you run traceroute xtvapi.cloudtv.comcast.net you should see it take a Comcast hop from the router instead of the AT&T hop.

Step 3: Persist the Config

Those commands on the USG just make the changes in memory. Now we need to ship the config to wherever we're running the Unifi management console. In my case, that's a Cloud Key Gen2. Following the guide I linked above, you'll dump the JSON config on the USG, strip it down to the bits we've done here, and then load that onto the Cloud Key as effectively an override file for the config input through the UI.

On the USG, you'll dump config using mca-ctrl -t dump-cfg > config.json. Then just get that JSON to a desktop.

From here I'm positive there's some magic jq incantation that could extract just the JSON leaf nodes we care about, but I couldn't figure it out.

JSON sections that matter to you (at least as of the time of writing... No clue how stable the config format is here):

• firewall.group.address-group.comcast
• firewall.modify.LOAD_BALANCE.rule.2500
• protocols.static.table.5

Strip the JSON file down and then transfer it to the device running the management console. For me, running a Cloud Key, I needed to make it /srv/unifi/data/sites/default/config.gateway.json and chown unifi:unifi.

Once you've got the file in place, you can go into the management console, find the USG, Settings -> Manage -> Trigger Provision.

If you formatted the file correctly, the USG will cleanly load the config. If not, it'll get stuck in a startup loop. If you hit that case just move the config.gateway.json to something like config.gateway.json.tmp to let it start clean while you debug.

Podcasts are the hot topic in the audio world right now, but they present a big challenge for those of us trying to make sense out of the numbers. Some of that challenge is intractable right now—podcast clients simply don't give us any information about when a downloaded file is played—but some of it is just about putting a better wrapper around old-school download stats.

At KPCC we're using two free tools—Logstash and Elasticsearch—to make it much quicker and easier to keep track of podcast downloads.

A few things to consider

Before you jump into setting up tools, it's important to understand a few characteristics of podcast downloads that you need to think about for metrics:

One download doesn't equal one request

Many podcast clients will use byte range requests to fetch a file in multiple requests. Some clients probe for the size first and then try to get it all via one followup request, while others simply divide the download into chunks, using one request per chunk. In the server's access log, these requests are typically represented via a status code of 206, or Partial Content. Other clients use a single request, and their response is given a status code of 200.

Either way, we want to make sure we count a single download one time.

Some requests may not equal downloads

In an ideal world, podcasters would be able to track and understand the number of times actual people listen to their content. Unfortunately, that's not how podcast clients work. They download files, which may or may not ever be listened to by the user. That's a reality we just sort of have to live with right now.

But not every request to a podcast file is even a client downloading audio. Some requests are from bots, others are from clients that like to inspect the beginning of a file but don't actually follow through with a download.

Broadly, we end up with specific requests we want to exclude (maybe via an IP blacklist), and requests that are too small to count.

Concepts

I'll get into the specifics, but basically there are two important concepts to our approach.

Session UUID

In order to easily deduplicate the pieces that make up one request, we inject a session UUID query parameter during the client's first request for audio content. We do this via a 302 Moved Temporarily response, which asks the browser to use the UUID URL for its download, but to try the original URL again next time it wants to access the same content.

If you aren't using a session or request ID of some sort, you end up needing to do the same deduplication manually during analysis by processing all your requests and looking for some unique combination of IP address, request path, user agent and time. You'll arrive in largely the same place, but the analysis script has to work much harder.

Elasticsearch Cardinality Aggregation

Once we've processed requests with the UUID injected above, we're able to then leverage Elasticsearch's cardinality aggregation to reliably estimate a number of unique sessions, without having to load all the session IDs into memory and count them.

Importantly, aggregations are something that Elasticsearch can do well across a cluster, meaning that your query can still perform well even as your number of requests and sessions scales.

The Specifics

nginx

The only important bit here is that you somehow inject the UUID. There are a million different ways you could do that. At KPCC, we use an old in-house tool called Podroller behind nginx for our audio delivery. It integrates with another old in-house system that serves up transcoded prerolls, which isn't relevant to this conversation. Because Podroller was sitting in the right place in our delivery pipeline, though, it was easy to add a step that takes any request that comes in without a uuid query param, generates one, adds it to the URL and redirects the user there.

If your audio delivery is via CDN, the redirect step may just need to be a light script that runs on a local server and redirects the altered URL to the CDN.

Logstash

We use logstash-forwarder to ship our nginx log files from the media server to our central Logstash server.

Once they get there, we use patterns matching our NGINX log file format and a Logstash filter to break each request into a number of fields:

After breaking the request log into all its fields (including, importantly, fields for the query parameters), Logstash outputs the results into Elasticsearch.

Elasticsearch

Here's where the fun happens. Now that all our data is in Elasticsearch, it's time for analysis.

I wrote a little Node.js tool called scpr-media-tools that we use for extracting Podcast stats. It ends up creating a rather lengthy JSON query body:

So what is it doing? It's filtering for:

• Requests that come from our audio delivery host and are marked as "podcasts" with a via query parameter. The latter part wouldn't be necessary if podcast data is all you were piping into the system.
• Requests that involved sending more than 8kb, which for us, this ended up being a suitable floor to filter out most of the non-requests mentioned above.
• Requests that match the day we're looking for, since our script grabs stats one day at a time.
• Requests that did not come from an IP address that we've blacklisted as a non-human scraper.

By saying size: 0, we're asking Elasticsearch not to return any actual data. Instead, we're asking for it to just return the output of two aggregations, one nested inside the other.

The show aggregation uses a context query parameter that we tag onto our URLs to break downloads out by program. If you've instead got well-named directories or files, you could pull them out via a Logstash filter and do the same thing here.

For each program, we then use the afore-linked cardinality aggregation to produce an estimated number of unique session UUIDs, using the uuid query parameter. The precision threshold tells Elasticsearch to manually count result counts under the threshold, but to do estimation above. Increasing that number will increase memory use, but reduce estimation error.

Once you've got your data to this point, you can also use Kibana to visualize it and build nifty dashboards:

Eric Richardson / KPCC

Example podcasts dashboard using Kibana.

We have a couple different metrics pipelines for different types of audio delivery, but Elasticsearch aggregations have become the common denominator in all of them.

Today some of that thinking actually gets into the hands of users. We just pushed a completely-rewritten version of KPCC's iPhone app that lets you rewind to the start of the currently-live program regardless of when you start listening.

Right at the tail end of my first stint at the station, we switched KPCC's online streams over to StreamMachine, a server package that I had written to replace our aging Shoutcast processes. It's done a great job serving our online listeners over that period, but until today it hasn't had a chance to show off its party trick: letting listeners step back in time and explore "nearly live" radio.

In March of 2012 I wrote here about "taking radio beyond the play button" and a little bit about the potential for new listening experiences we could enable by treating our "recently live" stream as a big buffer that you could use in different ways.

StreamMachine has had that buffer for three years now, but no one really ever got a chance to take advantage of it. The Knight Foundation didn't choose to take us up on the grant proposal we submitted, and other priorities left StreamMachine dutifully serving up plenty of live connections, but none that wandered back in time.

With today's launch—credit for which should go to @seandillingham, Ben Hochberg and @meeker5—we're finally taking a first step into those capabilities. The app lets listeners play the live stream or rewind to the start of the current show, along with the usual complement of on-demand podcasts, etc. You can't yet seek to arbitrary positions or explore previous programs, but you can finally solve the problem of tuning in only to realize you missed the top of the show.

Under the covers is another very cool change: the app and StreamMachine are now using Apple's HTTP Live Streaming (HLS) protocol instead of Shoutcast for pulling the audio stream. Where Shoutcast is one long connection that needs to stay active, HLS uses a stream of short-lived connections that grab segmented chunks of media content. That's way more user-friendly in a mobile world where listeners hop from WIFI to 3G/LTE and back with regularity.

There's lots that I'll want to write about the process of implementing HLS and its implications for things like listener analytics, but that will be a different post.

A nifty byproduct of the switch: you can now hit pause, run into the store, and hit play again afterward to pick up right where you left off. In the Shoutcast world, that would have required that the app keep sucking data while you weren't listening, but HLS lets us just go back to pulling segments as needed.

As today's launch trickles down to users, we know we'll run into lots of edges that could work just a little bit better, but hopefully we'll also get listeners excited about new ideas in how a radio stream fits into their daily life.

On the 20th, Kathy and I became the very proud parents of a very little boy who shares the name Robert with one grandfather, two great grandfathers and a great-great grandfather.

And then yesterday I ended the month with my last day at Emcien, where I've been working for the last two-and-a-half years since moving to Atlanta. On Monday I (virtually) head back to my last job with KPCC, where I'll get to dive back into a lot of cool ideas about the future of radio.

During the time since I left, we've continued to slowly push forward on StreamMachine, the streaming audio server that I had written for the station. I'm very excited to get to ramp that work up and dive back into all the other cool things that we can do to deliver great content to listeners in this ever-more-connected world.

I'm using Chef for configuration management, since that's what I use day-to-day at Emcien. My little one-server deployment gave me an excuse to try out their hosted product, as compared to the open-source server we're running.

A handful of off-the-shelf recipes do the initial setup:

• hostname: Set the server's host name based on the node name and the "ewr.is" domain.
• users::sysadmins: Create my user account from information in a data bag.
• sudo: Grant sudo rights to the proper user groups.
• ntp: Make sure we're running the NTP daemon to sync the server clock.
• nodejs: Install NodeJS from packages.
• redis::server: Install Redis server.
• percona::server: Install the Percona MySQL server.

After that, a few of my own cookbooks take over.

• I put together a quick nginx_passenger cookbook to deploy NGINX and Phusion Passenger via the .deb packages that Phusion recently started building.

• A lifeguard cookbook installs the lifeguard process runner that I wrote for Emcien. A lifeguard_service resource then offers a quick way to install new services that will restart via the Capistrano-friendly touch tmp/restart.txt.

• The cleverly-named ewr cookbook ties the two together, illustrating how this site gets configured:

include_recipe "nginx_passenger"

# -- Pre-reqs -- #

package "imagemagick"
package "libimage-exiftool-perl"

Run the nginx_passenger::default recipe, then install two packages the site needs to run.

# -- Create a User and Directory -- #

ewr_app "ewr" do
  action      :create
  with_cap    true
  ruby        "1.9.1"
  env({RAILS_ENV:"production"})
end

dir = "#{node.ewr.app_path}/ewr"

Use a little app resource to create a user and home directory, install Ruby 1.9.3 (1.9.1 is the ABI version, which packages still use) and set our environment.

# -- Make asset_images directory -- #

directory "/data/ewr/asset_images" do
  action    :create
  owner     "ewr"
  recursive true
end

Create the directory that we need for storing images.

# -- Install the Site File -- #

nginx_passenger_site "ewr" do
  action      :install
  dir         "#{dir}/current"
  server      "ewr.is"
  ruby        "/usr/bin/ruby1.9.1"
  rails_env   "production"
end

Create an NGINX site configuration file pointing to the server we created.

# -- Install Lifeguard and Tasks -- #

lifeguard_service "ewr.is Resque Pool" do
  action  [:enable,:start]
  command "bundle exec resque-pool"
  user    "ewr"
  dir     "#{dir}/current"
  service "ewr-resque"
  path    "#{dir}/bin"
  env({RAILS_ENV:"production"})
end

Install a Lifeguard-managed Resque Pool instance, used for processing images.

The resulting code may not be useful to anyone else, but at least I don't have to be embarrassed when I log on to my own server.

Technical difficulties (see my post about falling behind) kept me from posting these favorite photos here earlier, but I think it is only right to eventually give them a public viewing.

At Emcien, my main role these days involves creating the processes we use to build our servers and infrastructure. That means a lot of building server images and a lot of thinking about how to make sure our environments are always running up to their potential. I spend a lot of my day writing recipes to make sure that we don't have to worry about our systems and whether a deploy was done correctly.

So imagine my surprise when I logged onto my long-neglected personal server yesterday and realized how much it was the antithesis of all those principles I stick to at work.

It's running Ubuntu 11.10, which hit end-of-life back in May. Services were cobbled together using a variety of screen sessions. Ruby, Node.js and MySQL are all ancient.

Obviously that's natural in many ways: you're more likely to put the work in when you're paid to do it than you are in your free time, and the rewards for doing the work are higher when you're talking about dozens of servers than they are when you're only managing one.

Still, there has to be some kind of a life lesson here, right? The things we do when we're in private eventually catch up to us?

As for me, I need to get going on building up some new server recipes.

It was an amazing ride while it lasted, though. What started as just me writing about my exploration of a neighborhood turned into a news source that at its peak was attracting 40,000 to 50,000 unique monthly visitors. That traffic was down more than half by the time KPCC pulled the plug.

In the end, the question I couldn't answer was the same one that has been haunting journalism for years: how do you make money on this?

As Kevin Roderick noted at LA Observed, we went through several models over the years. We tried the non-profit thing, but discovered that local businesses really don't care about a tax write-off—they care about whether you can bring people to their door. The print edition was by far the biggest shot at making a financial model, and I believe that it is one that could have succeeded given a little more runway and a little more business savvy.

That's the biggest lesson I took away from my time as an attempted entrepreneur: I can't do everything, at least not all at once. If I were doing it all over again I would have made sure to find a partner who could have lived and breathed the numbers side in the same way that I did the content side. When I was having to wear both hats, too often it was the money side of things that got put off in favor of the content and the product, and that's clearly not sustainable.

As I became less a part of blogdowntown after going to KPCC, I threw myself back into technology to fill the void of the writing and photography that I had come to so enjoy. It's been a fun rediscovery to realize how much I do enjoy building systems and software, but I definitely have a soft spot still for the world of journalism. Who knows, our paths may again cross one day...

The image above shows the two different views of my activity over the last year, with public activity on top and my private view on the bottom. Apparently it's been a private year: of my 1,189 commits, only 67 show up in public.

That says that while it's been a productive year in general, it hasn't been much of a year for the side-projects that tend to be my public contributions.

While that's not all bad—I've had plenty of chances to play with interesting technology at Emcien—there is something cool about developing side projects.

On the bright side, 25 of those public commits have been to StreamMachine in the past month. I launched the online streaming project with KPCC on May 1 of last year, just before I left the station to head east, and it has continued to reliably deliver audio to thousands of listeners every day since. Some of those commits are ones requested by the station, but a number of them are just about wanting to push forward on a project that inspired some big ideas.

Those ideas may have taken some time, but the reason I want to see them happen hasn't changed: no one else has implemented exactly what I'm looking for yet.

I set to work reading up on it yesterday, just trying to scope out what it would take to add support to StreamMachine. Turns out, not much, but I got to have some fun playing with binary in the process.

In the streaming world, AAC audio frames contain an Audio Data Transport Stream (ADTS) header. It tells you things like what encoding the audio is using and how much of it there is before the next frame header. Conveniently, that's very similar to MP3.

MP3 frame headers are four-bytes, starting with 11 sync bits that are all turned on (all 1's, in the binary world). The next 21 bits tell you things like the sample rate, whether the audio is mono or stereo, etc.

ADTS headers are either seven or nine bytes, depending on whether error protection is enabled. They start with 12 sync bits.

Laid out as zeros and ones, the ADTS header would be either 56 or 72 digits long. That's not how you interact with it in the programming world, though. Instead, bytes (eight bits) are represented as a hexadecimal pair. So instead of 01011100 you get 0x5C (both equal 92 -- either 4+8+16+64 in the binary or "five-sixteens plus 12" in the hex).

That's all well and good, but for something like this I still end up having to go back to the pen and paper and draw out my bits, as you can see above. It may be digital, but sometimes it still takes seeing it on paper to actually understand it.

I'm blessed to be able to walk to work most days, so most of the steps that Moves ends up tracking are on the 0.6 mile trek between our house and Emcien. That walk is pretty short, but includes a steep climb. Because of that, my walking speed is noticeably faster coming home from work than it is heading in: approximately 0.3 - 0.5mph different, according to Moves.

Even more noticeable is the difference in steps. Each day I take approximately 1300 steps on the way into the office, and 1100 steps on the way back. I'm covering about 15% more ground with each step coming down the hill.

I guess it makes sense, then, that I keep having to tell Moves that I wasn't running. Given the difference, it's a fairly understandable mistake.

It was the same in photography. In Lightroom I have 15,595 pictures from 2009, 14,685 from 2010, 9,125 from 2011 and 690 from 2012. Even that number doesn't quite tell the whole story, since 220 of those were taken in December.

It's funny, sometimes, how hard it is to get back into a habit once you've stepped away.

My time was plenty busy with other things. Along with moving across the country and buying a house, 2012 became my year to dive back into technology. Starting the year at KPCC that meant getting into Node.js and diving into ideas about where to take online radio. More recently it has meant getting a handle on dev-ops for Emcien, fleshing out a strategy for using Chef to provision and deploy our servers in the cloud.

Those first passions draw at you, though. I do miss the research, the writing, the working to get pictures, just the storytelling process in general.

No clue where that will lead in the future, but for now how about it just turns into a few more frequent updates here? Six years might just be long enough...

Housekeeping note: A few days ago I moved this site to the domain ewr.is (my middle name is William, to answer the obvious question). I just got tired of how long ericrichardson.com was to type. All the cool kids have short domains these days.

If you look at our map from back in June, Vinings wasn't really on it (and neither were townhouses), but over the summer my work relocated there from midtown. That took away the idea of looking for something near transit, and instead we ended up asking ourselves how you could really do better than somewhere in walking distance of the new office.

We decided you couldn't.

Friends helped us move our stuff from storage a week ago, so this weekend was the first we spent as homeowners in our new place. Even with buying new construction, there were still plenty of projects to get started on.

On Friday we stopped by a Lowes to pick up two Nest thermostats, replacing two basic programmable stats with a device that has a little higher geek cred. I had been eying them after how highly eecue spoke of his, and the combination of a price break after the introduction of the second version, a 10% off coupon and a Georgia tax-free weekend for energy purchases let me pick up the $250 device for a little under $200. Good thing, since I needed two (and really could use three—I didn't switch the stat on the third zone on our bottom level).

I got those up and running quickly on Saturday morning, and Sunday got to do the much less exciting task of installing towel racks and a new showerhead (1.75 gpm may be "eco-concious", but it just wasn't cutting it).

Vinings has given us a bit of the walkable lifestyle we missed from our time in downtown L.A. There are a number of good restaurant options in easy walking distance, along with yogurt, ice cream and a few shops. No good coffee, though... Still miss that.

We're starting to get used to our new routines, but won't really be able to get settled in until we find a house and can get the rest of our things out of the storage unit we dropped them off in when the moving truck arrived. Last weekend we put in some major miles toward that process, driving a big chunk of the city to get a feel for where we might be looking.

While we'll be working with my uncle to actually pick a place, our weekend's exploration was mostly powered by Zillow. After doing some initial research online, Kathy drove and I navigated as we wound our way around neighborhood after neighborhood doing drive-bys of homes in our price range.

We probably spent ten hours in the car between the two days, and then supplemented that with a heavy dose of research into schools on Saturday evening.

Zillow was a huge help, and we made good use of it across the web, iPhone and iPad platforms. In the process, though, I arrived at some thoughts about what I would like to see changed.

• On the web, Zillow will give you the schools for a listing, but it won't give them to you on the iPhone or iPad. While a multi-platform application has to build its user interface around the unique traits of each platform, feature parity is important in creating a consistant experience.

• Integration to link to or pull in school info from something like schooldigger would be a real time saver. Lots of flipping to a new tab and googling for "es name schooldigger". Taking a next step forward to show map overlays with elementary districts would be even better.

• Though I can't say I've done a huge amount of looking, I haven't found an app that will give me census information for the tract I'm in right now. Why isn't there an app that does the same sort of visualizations as the NY Times' census maps?

• Zillow lets you make a "Favorite Homes" list, but we really wanted the ability to do multiple lists breaking listings down by "Good Schools", "outside the perimeter", etc.

We'll be taking a break this weekend to host a friend, but after that Kathy and I will start the process of actually touring homes. We've got our target area in mind, so now it's just time to look for the right place to come on the market and make a move.

Both will live on, since the station was generous enough to allow me to open-source them. There's a thin line between a live open-source project and abandonware that just happens to have code online, though, so it remains to be seen whether either generates the kind of momentum needed to create a life beyond KPCC's uses.

The Vision

The idea behind AssetHost is a simple one: your CMS shouldn't have to know or care what kind of media you want to slap on a news story or blog post. It should just know what goes where and how to call the system that takes care of implementation. That system, AssetHost, is in turn based around the concept that regardless of media type, you're always going to need to create thumbnails and flat visual representations for things like homepages, section indexes and mobile devices.

The current generation of AssetHost does a good job of moving in that direction. We hooked the chooser and asset models into both Django and Rails apps with great success, and seamlessly integrated support for videos hosted on Brightcove.

But what if you took that idea to the next level?

A Future of Media Plugins

At the beginning of the year I wrote AssetGallery, the code that serves up the albums in my Photos section. It's a nifty bit of code, using AssetHost for image serving and turning out some galleries that look cool both on the desktop and on IOS devices.

What would be even cooler, though, is if AssetGallery was a plugin for AssetHost that created gallery assets out of image assets. Then you could go into the AssetHost admin UI, create a new gallery and use that gallery as an asset on a story without the CMS having to know or care that it is serving up a slideshow instead of a photo.

Or what if you had an "AssetMap" plugin that would take a geocoding and serve up a map as an asset, putting a nice interactive map UI on the story page and cutting thumbnail versions automatically to serve up in static contexts? That would sure seem a heck of a lot more useful than the map screenshots you too often see on breaking news stories.

The goal would be to have AssetHost act as a framework for these sorts of rich media, allowing plugins to be lightweight bits of code that just focus on their specific implementation.

Engines for Engines

What you basically end up with are engines that can be plugged into AssetHost in the same way that AssetHost can be plugged into your CMS.

Rails does a lot of the heavy-lifting on this already. The engine overhaul that came in Rails 3.1 is great, and both AssetHost and AssetGallery are written as mountable engines.

The trick that I'm still working to wrap my head around, though, is what an engine would need to do to export multiple routers. For instance, AssetGallery currently has a routes file that looks like:

AssetGallery::Engine.routes.draw do
  resources :sets do
    member do
      match '/:asset(/detail)' => "sets#show_asset", :as => :show_asset
      post :assets
    end
  end

  match '/' => 'home#index', :as => :home
end

That allows me to pull the gallery functionality into my site just by including one line in my site's routes file:

  mount AssetGallery::Engine => '/gallery', :as => :gallery

What I want for this AssetHost engine scheme, though, is to allow AssetGallery to also have a set of admin routes that it register with AssetHost, so that going to /assethost/galleries/ would give you an admin interface for galleries just like the one /assethost/assets/ would give you for other assets.

I have a feeling the answer lies somewhere in ActionDispatch::Routing::RouteSet, but exactly where has escaped my first few attempts.

Riders zipped by us heading north on Olive street, and then a few minutes later heading south on Hill street.

My photos from yesterday are just little snapshots compared to the ones I took for blogdowntown the last time the tour came through Downtown, but it was fun to get out for a few minutes with the camera nonetheless.

You can see larger versions of the photos over in the gallery section.

Yesterday I bought the plane tickets: our time as Angelenos ends at 1:10pm on June 15.

Moving back east is a thought that we had entertained for a while, since we don't have any family out here on the west coast. Once Kathy decided that she wasn't going to be returning to her school next year, the conversation about where we might want to be started in earnest.

We went back to South Carolina for Kathy's spring break, and while there decided that we were serious enough about the idea of Atlanta as a possible destination that I should start looking into companies who might be hiring. I sent off a few resumes, and ended up setting two interviews for a day that we would be passing through before flying back out west. Both turned into offers, and a week later I turned in my notice to the station and we began the process of planning a move.

Last Monday I started my new job with Emcien, a company that has a suite of applications built around really fast pattern matching in large data sets. While I won't be going anywhere near the math itself—I had to admit during my interview that my last math class was my junior year of high school—I'm really intrigued by the challenges that come in building the applications that allow users to make effective use of their data and the results that come out of it.

blogdowntown stays at KPCC, where it hopefully has a long life ahead of it. My day-to-day involvement in the site ended back in January, so our move won't create much of a change there. I'll continue to be involved with the open-source development of AssetHost and StreamMachine, the visual asset handling and audio streaming systems I wrote while there.

There's a lot that we'll miss here in L.A., but we're excited to head out on a new adventure. Unfortunately, that adventure starts with logistics: this week I get to start calling movers.

There have been a few hiccups—I don't think people with Roku boxes are very happy at the moment—but on the whole it's been a very successful launch.

While audio streaming provides a number of interesting challenges—how do you handle deployment of new app versions when your connections are of indefinite length, for instance—I'm really interested to play with ways of visualizing listener behavior and interactions.

For the StreamMachine launch I'm using a pair of really interesting open source packages put out recently by Square to enable real-time analysis of listening patterns. Cube is a datastore built on top of MongoDB that's designed specifically for time-series data. Cubism is a visualization plugin using d3 that then turns that data into interesting visuals.

Traditional logging for a listening stream doesn't make for a very good event. Saying that a listener completed a 45-minute streaming session at 6:55pm is great, but that data point needs to be spread out over the whole listening period to give a true picture.

But what if instead you logged every minute listened as it was listened? You would end up with a series of events taking place at the right time in the timeline, and you could then analyze them as they're still in progress.

The screenshot above shows the results, which gives a snapshot of all listening, a comparison to yesterday's stats, and a view of KPCC app listening vs some in-browser numbers. Refining the options and presentation is a work-in-progress, but already it's a fascinating live view of what's going out of our servers.

Only after I had all that and I tried to make an API request did I realize that Google doesn't support service accounts in Analytics yet, so all of that work was in vain. To try to keep it from going to waste, I figured I would post it here. Looking to authenticate a Google service account for a Ruby app? Hopefully this will save you a little time.

The JWT gem can be found here.

    require 'jwt'

    # -- Create JWT -- #

    # load in private key
    keydata = nil
    File.open(Rails.root.to_s+"/config/google_api_key.p12",:encoding => "binary") {|f| keydata = f.read(); }
    pk = OpenSSL::PKCS12.new(keydata,"notasecret")

    # generate the JWT
    jwt = JWT.encode({
      :iss    => "12345678@developer.gserviceaccount.com",
      :scope  => "https://www.googleapis.com/auth/someservice.readonly",
      :aud    => "https://accounts.google.com/o/oauth2/token",
      :exp    => (Time.now + 3600).to_i,
      :iat    => Time.now.to_i
    },pk.key,"RS256")

    # -- Convert into an OAUTH2 Token -- #

    uri = URI("https://accounts.google.com/o/oauth2/token")
    req = Net::HTTP::Post.new uri.path
    req.set_form_data(:grant_type => "assertion", :assertion_type => "http://oauth.net/grant_type/jwt/1.0/bearer", :assertion => jwt)

    conn = Net::HTTP.new uri.host, uri.port
    conn.use_ssl = true

    resp = conn.start do |http|
      http.request(req)
    end

    json = JSON.parse(resp.body)

    # token is at json['access_token']
    puts json

While we didn't get picked to receive the grant I wrote about a few weeks ago, I've been saying for several months now that I really wanted an app to let me grab a commute's worth of audio just before I walked out the door and onto the subway.

Last night, I finally got the app to the point where I could do just that. The server side of things has been in place since mid-February, but the app side of things kept running into the fact that I don't know Objective-C and have never put the time into getting up to speed on iOS app development.

That point kept being hammered home to me over the weekend as I again and again ran into variations of the EXC_BAD_ACCESS error as I tried to reference things that I was inadvertently failing to keep in memory.

Last night I finally refactored my way past that and got the app working.

Then this morning I pushed the "Grab New Audio!" button a few minutes before walking out the door and quickly had the last fifty minutes of live radio to listen to as I navigated the Red and Gold lines—a route that has both periods of no data connection and connectivity losses.

It may just be a proof-of-concept, but it worked like a champ for me.

Today I have a two terabyte drive that I use for backups, and 750 gigabytes inside my laptop. Another terabyte drive (the one pictured) is sitting on the desk, but not being used for anything in particular.

I still marvel at how cheap storage has gotten, and how quick it has gotten there.

My first few weeks have produced an interesting result, though: they've made me a reader again.

I've gone through four books since buying the device just a little over a month ago. I started out with Neal Stephenson's most recent release, Reamde, then circled back to some of the books that followed Orson Scott Card's Ender's Game, reading Speaker for the Dead, Ender's Shadow and Shadow of the Hegemon.

In that time no apps have become essential parts of my life, though I will admit to spending a few hours on Real Racing 2 HD (and have spent a few hours more than I will admit). I think the iPad makes an amazing web browser, and have been thoroughly impressed with the battery life.

Above all, though, I've just loved having a device that I can carry to catch up on news, do some emails, and sneak a few pages from a book every time I've got an extra minute. I've started carrying it instead of my laptop on my commute, which means that I can grab a few pages while waiting for the train and then a chapter more as I'm on the Gold Line. Sitting in Starbucks before I head into the office it's web and email, then maybe a few more pages at lunch if I'm headed off on my own.

I don't doubt that eventually other apps will find their way into my day-to-day, and I have a few ideas for ones that I could write to scratch some of my own itches, but in the meantime I'm enjoying this new return to books. Next up: Don Delillo's Underworld, a book that's been sitting on my shelf for the last few years. Here's to hoping this time I'll go ahead and read it through.

Once you've got all those pieces in place and have both applications happily running off the same data, though, you still need a way to route incoming requests to one or the other. That could be something that you handle in logic on the load balancer, but our setup instead left this as something to solve as the request came into the web server. Turns out, though, nginx made it all quite easy.

The Requirements

In order to declare our deployment a success, we needed the ability to gradually roll features and users from the existing Django site to the new Ruby on Rails frontend.

Some paths needed to be conditional based on a beta cookie, while others needed to only point to the Django site to pick up things that we hadn't yet implemented in Rails. Still others needed to always point to Rails, to allow ground-up development (such as [the new videos section) to exist only on the new site regardless of cookie status.

Nginx to the rescue!

It turns out that those requirements are exactly the sort of thing nginx is good at handling. While "www.scpr.org" is defined in a "server" block in the nginx config, deeper constructs such as conditionals have all the same rights to define settings related to how the request will be handled.

Early in the process of testing out whether this idea would work, I deployed nginx with Passenger to see whether Passenger's "experimental" wsgi support might work for hosting our Django app. It turned out to run quite well, which allowed me to know that all I needed to do in nginx was set root to either the Django or Rails apps and let Passenger handle the rest.

Defining a map

nginx has a nifty map construct that makes up the bulk of our conditional logic:

map $uri $test {
    ~^/$                1;
    ~^/assets/          2;
    default             0;
}

That takes $uri and tests it against the map, putting the result in $test. The tilde in front of the keys allows regular expressions to be used. Our simple test here will return one (can go either way, based on cookie status) when the user is visiting the homepage, two (Rails-only) if they're looking for something under /assets/, and zero (Django-only) if they're trying to get anything else. Our production map is a lot longer, but you get the idea.

Once we have that, we need to add the cookie test. This gets a little funkier. If $test2 contains the results of your cookie test, typically you would want to say something like:

if ( $test1 == 0 || ($test1 == 1 && $test2 == 0) ) {
    # django
} else {
    # rails
}

nginx, though, doesn't support testing multiple variables in a conditional. That meant we needed to combine the two:

# default to old site
set $site 0;
set $test2 "";

# possible match...  two tests are combined
if ($test = 1) {
   set $test2 A;
}

# map based on existence of a cookie
if ($cookie_scprbeta = "true") {
    # use new site
    set $test2 "${test2}B";
}

if ($test2 = "AB") {
     set $site 1;
}

if ($test = 2) {
     # always map to the new site
    set $site 1;
}

Default to site zero, but use site one if either a) we have a conditional match from the map and have the cookie set or b) we have a map match that is only on Rails.

From there we simply set our root based on the $site value:

if ($site = 0) {
        # run old site
        root /web/django_app/public/;
        passenger_min_instances 4;
}

if ($site = 1) {
        # run the new site
        root /web/rails_app/current/public/;
        rails_env "production";
        passenger_min_instances 4;
}

Ta-da! Seamless handoff in the same visit.

My family used to camp at the state park every year, so I've spent much time on that side of the road, but I can only remember once that we had ever visited Brookgreen, the sculpture gardens that the pair opened to the public in 1932. We paid a return visit this week and really enjoyed wandering around the former rice plantation.

A warm spring meant that even in the first week of April, many of the Brookgreen blooms were already past their prime.

Given our normal views in Los Angeles, though, all the green and color was still a sight to behold.

You can see more photos over in the Photos section.

It was a fortuitous coincidence, since I was looking for ideas on how best to implement logging inside StreamMachine, the rewindable streaming audio server I've been playing with at KPCC.

Bunyan's toolset isn't quite fleshed out at the moment, but the idea is that logging loosely-structured data as JSON is a heck of a lot more functional than squashing everything into strings.

So instead of a typical Shoutcast w3c log line:

96.10.155.166 - - [04/Apr/2012:07:11:50 -0700] "GET /kpcc HTTP/1.1" 200 219606 "-" "VLC/2.0.1 LibVLC/2.0.1" 85

StreamMachine can now log a connection as:

{
"name": "StreamMachine",
"hostname": "dev2.kpcc.org",
"pid": 18605,
"source": "kpcclive",
"level": 20,
"req": {
    "method": "GET",
    "url": "/kpcclive.mp3?socket=11516610901102892326",
    "headers": {
        "host": "scprdev.org:8000",
        "connection": "keep-alive",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.83 Safari/535.11",
        "accept": "*/*",
        "referer": "http://www.scpr.org/assets-flash/streammachine.swf",
        "accept-encoding": "gzip,deflate,sdch",
        "accept-language": "en-US,en;q=0.8",
        "accept-charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3"
    }
},
"listeners": 0,
"msg": "Connection end",
"time": "2012-04-05T11:49:38.871Z",
"v": 0
}

(Ignore that I'm not actually including seconds listened or bytes sent in that log line...)

It's wordy, but if the computer is the only one reading, who cares?

I've been writing a lot of log-parsing tools lately, taking in Shoutcast logs and running stats on listening habits like time-of-day and duration. A structured JSON object gives me a lot more lee-way when it comes to adding and removing log data than a w3c-formatted log does because I no longer have to remember that user agent is the third match in my regex results, etc. Calling log.req['user-agent'] is nice and simple.

Given the headaches that trying to make sense from log data already present—and always will—that's a good thing.

Turns out, that may have been the start of something interesting.

Over the last few months I've become a bit obsessed with the idea of where radio goes in this new Internet world, and particularly with the idea that the experience of listening to a digital stream has the potential to be worlds better than that offered by your radio. Your radio, after all, only offers you a play button.

Up to now, though, that's all we've offered Internet listeners as well. Connect, hit play, and go do something else.

That's not going to be good enough any more. Yesterday KPCC submitted a grant application to the Knight Foundation's News Challenge, seeking funds to build pieces of the next-generation of streaming radio infrastructure.

My Saturday morning is a great example of one of the problems we are trying to solve. My favorite NPR show is Car Talk (what that says about me as a news person, I'm not sure). Inevitably, though, I am useless when it comes to remembering to start listening at the top of the hour. Via NPR's iPhone app, I can find a station streaming Car Talk at pretty much any hour of the weekend, but none of them do me any good if I only remember to listen at :15 past.

What if I could take control of the process and rewind to the point where I meant to start listening, instead of just the point where I did start listening?

ESPN Radio's web player does something like this. While the UI leaves a bit to be desires, the idea is killer. Technically, it's not even really that difficult: just build a big buffer up on the server and build a UI around being able to access the stream at different points.

So far I've built a demo web player implementing this functionality, and continued to play around with StreamMachine as the server backend.

Just the beginning....

Rewinding is just the beginning when it comes to the potential for new functionality.

I get to the station every morning via two trains. It's a 40 - 45 minute trip. I would love to listen to the radio on my way in, but between the subway and connectivity dropouts on the Gold Line, it's just not practical. I don't want podcasts either: I want today's news, not yesterday's.

But what if our iPhone app had a function where I could tell it that I had a 45 minute trip and it would pump me the last 45 minutes of live radio? Over my home wifi, it would download in a couple minutes and I could commute listening to "nearly live" radio. Today's news, just a little delayed.

What if I just heard something awesome on the radio and I want to share it with my friends? What if I could hit a button and share that point in the stream to my Facebook timeline or Twitter followers?

All of a sudden, radio gets interesting and social.

What's next?

This project started as a spare-time pursuit on my evenings and weekends, but I have a feeling it's going to become a much larger part of my work hours over the next few months. If we do end up getting the Knight grant, that would come this summer and would certainly help accelerate the process of building out more player interfaces and integrating into the station's existing broadcast systems.

In the meantime, though, it's time to just keep playing.

There are plenty of ways to do caching. Page caches, partial caches, timestamp-based caches, versioned caches... they've all got some place in valid place in the caching arsenal. In Rails, the convention is to build sweepers that get fired when model objects are saved, giving a spot to handle expiration or rebuilding of caches. But what do you do when Django needs to be the one to expire your Rails caches?

Django's default cache is built around memcache and expiration times, with a version number prefix that can be incremented to change keys and start over when content changes.

That was the setup when I first got to the station, but expiration-based caches have a number of consequences that are less than ideal. First, unless you are keeping track of versioning and updating that key prefix, you're always going to have a delay before the cache times out and updates show up on the site. Keep that timeout too low, though, and you'll expire caches unnecessarily. That creates a situation where most page views are fast, but every time the cache expires some unlucky visitor is going to get burned as we rebuild the entire page.

Last summer we switched that setup out, putting in a new system that attempts to do a cleaner job of expiring caches only when the content in them changes. Taking advantage of Redis and its sets data type, the system stashes a copy of the cache key in a set for each piece of content inside. When that piece of content is updated, the cache system expires every cache in that object's set.

While that move was made well before the plan to implement Rails, it turns out the system didn't need much change to power both systems at once. Make sure the cache keys are stored in full form with any framework-specific prefixing, and all of a sudden a content update in Django can expire caches put in place by Rails.

Listening in to the broadcast

Going a step further, what if you want a content update to trigger a cache rebuild, rather than leaving that action to trigger when a poor user hits an uncached view?

Again, Redis comes to the rescue. The database includes a Pub/Sub messaging system that allows an easy abstraction between publishers and subscribers.

In Django, I set up a post_save action on our content that triggers a PUBLISH action with a JSON object containing the content's unique key and some commonly-needed bits of metadata (what is the content's publish status? what action is this: a save, a publish, an unpublish?).

Using the worker listener in Resque as a model, I then set up a Rails process to sit and listen on that same channel. When it sees a change that should update the contents of the homepage, it fires a process that rebuilds our Sphinx caches and then caches new views.

Getting Fancy... What if you want Rails to cache for Django?

Want to get super-fancy? Use that Rails worker to cache back to Django.

It's actually not that hard, thanks to RubyPython:

if Rails.env == "production"
  RubyPython.start(:python_exe => "/usr/local/python2.7.2/bin/python")
else
  RubyPython.start()      
end

@pickle = RubyPython.import("cPickle")

Now you have a Ruby interface to Python's Pickle, the important piece you need to write caches that Django will be happy to open:

# if we're passed a pickle object, also perform django headlines caching
if pickle
  (Rails.cache.instance_variable_get :@data).set(
    ':1:hsection:headlines',
    pickle.dumps( view.render(
        :partial => "home/cached/django/headlines", 
        :object => scored_content[:headlines]) 
    ),
    :raw => true
  )
end

Nifty.

Rails Code

The Rails side of our caching code is available in the redis-content-store gem on SCPR's Github account. It includes the underlying caching code and simple view helpers for cache_content and register_content.

It's not perfect—my next step is to put some work into how nested caches would keep track of their content objects correctly—but it's working well for our purposes.

Django Code

The Django code can be found in this Gist: