Policy Routing for Xfinity Traffic on a Dual-WAN Unifi Gateway

Sunday, January 16, 2022, at 09:32AM

By Eric Richardson

A message in the Comcast Xfinity Stream app shows content that can only play only an in-home internet connection.

Since mid-2019 I've had redundant Internet connections here at the house, using AT&T Fiber as our primary link but keeping a Comcast connection active just in case of outages. For a long time it was just set up warm, where I would need to go down to the basement and switch a cable to switch links. Somewhere along the way a coworker pointed out that my Unifi Security Gateway (USG) supported Dual-WAN failover, and I plugged them both in.

That all works great, but there's been one nagging case I've wanted to clean up: We do use Comcast for television, and I like their Xfinity Stream app for listening to news while I'm getting ready in the morning. Because of using the AT&T Fiber link, though, the Xfinity app thinks I'm out-of-home, which limits my ability to watch in-progress recordings.

To fix that, I need to set the USG to route Xfinity traffic over the Comcast link, but leave other traffic doing its normal failover setup that prefers AT&T. That's not an especially hard ask, but it's been sitting on the todo list for a while.

When we moved in mid-2019 I turned on Comcast internet because it was the quickest to get going, but then quickly moved to using AT&T Fiber for our primary connection. I kept the Comcast internet active, though, because in this day and age redundant Internet connections seems

Full confession: I actually did this once already. I followed various online tutorials and set up the firewall rule on the router, which worked great. But with how Unifi works, making hot updates on the running gateway only applies until a reboot. I didn't take the final step of configuring it all to be saved for future provisioning, so along the way my changes disappeared and it took me a while to come back to them and do it correctly.

I guess that's part of why I'm writing this up... If I ever lose my config again, this can be a guide for my future self to get it set back up.

Also, how useful is it really to have a dedicated backup internet connection? 99% of days it sits there entirely unused, but it has saved me a handful of times. Just this month AT&T went out for most of a day at a time where virtual school was going on. It was great to realize that a bit into the outage when AT&T texted me, having not even noticed the actual point of outage as the connection flipped over. I get paid good money to do work on a computer. Paying $50/mon as a backup seems justified.

The Goal

When you're using the Xfinity Stream app or the web interface, the logic does a check to see whether you're at home. The web app, for instance, hits https://xtvapi.cloudtv.comcast.net/inhome-restrictions/. When you're out-of-home (my AT&T connection), you get this:

{
    "_type": "InHomeStatusRestrictions",
    "_links": {
        "self": {
            "href": "../inhome-restrictions/"
        }
    },
    "inHomeStatus": "out-of-home",
    "currentRestrictions": [
        "out-of-home"
    ]
}

That causes the app to refuse to play certain content where licensing doesn't allow out-of-home streaming.

Could you man-in-the-middle that request somehow and just remove the restrictions? Maybe? I didn't try. Since I've got the Comcast connection, I just want to send Xfinity requests there. I have no clue if other parts of the stack enforce similar in-home constraints.

The basic idea here is to use "policy-based routing" based on the destination IPs, and target the correct interface on the USG accordingly.

Step 1: What IPs are we routing?

So what IPs do we need to route? If we run dig we see that there are a number of IP addresses and CNAMEs behind the hostname.

~: dig xtvapi.cloudtv.comcast.net

; <<>> DiG 9.10.6 <<>> xtvapi.cloudtv.comcast.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40875
;; flags: qr rd ra; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;xtvapi.cloudtv.comcast.net.    IN      A

;; ANSWER SECTION:
xtvapi.cloudtv.comcast.net. 73  IN      CNAME   xfinityapi-production.r53.aae.comcast.net.
xfinityapi-production.r53.aae.comcast.net. 45 IN CNAME xfinityapi-tango-production-active.r53.aae.comcast.net.
xfinityapi-tango-production-active.r53.aae.comcast.net. 45 IN CNAME xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net.
xfinityapi-tango-production-aws-us-east-2-active.r53.aae.comcast.net. 45 IN CNAME xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net.
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.47
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.162
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.250
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.32
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.219.99
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.180
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.223
xfinityapi-tango-production-aws-us-east-2-green.r53.aae.comcast.net. 45 IN A 96.115.218.68

;; Query time: 18 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Jan 03 14:52:24 EST 2022
;; MSG SIZE  rcvd: 401

Instead of trying to target as few IPs as possible, my goal was to just end up with a range of Comcast IPs that was unlikely to break as they changed internal infrastructure. To arrive at that, I ran a whois 96.115.218.47. That gave us a good set of ranges:

# whois.arin.net

NetRange:       96.64.0.0 - 96.124.255.255
CIDR:           96.120.0.0/14, 96.112.0.0/13, 96.124.0.0/16, 96.64.0.0/11, 96.96.0.0/12
NetName:        CABLE-1
NetHandle:      NET-96-64-0-0-1
Parent:         NET96 (NET-96-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS7922
Organization:   Comcast Cable Communications, LLC (CCCS)
RegDate:        2008-02-21
Updated:        2021-01-25
Ref:            https://rdap.arin.net/registry/ip/96.64.0.0

For better or worse, I figured 96.120.0.0/14, 96.112.0.0/13, 96.124.0.0/16, 96.64.0.0/11, 96.96.0.0/12 was a good set to route.

Step 2: Setting up the routing on the USG

There are a number of good guides to USG policy-based routing and how to customize the config.gateway.json required to persist the changes.

So why can't you just set up static routes in the Unifi UI? Apparently if you have load balancing turned on the load balance rules are applied before the static routing rules, making them unhelpful here.

To apply the routing you'll need to SSH to the USG. Settings -> Site -> Device Authentication for settings in the older UI, or Settings -> System -> Application Configuration -> Device SSH Authentication in the "New UI". You can add an SSH key to make logins easier. Once that's applied you'll SSH to the USG.

The basic premise here is that we'll set up an address group with the Comcast IP ranges, a static routing table that routes to the Comcast interface, and then a firewall rule that says "use the static routing table if the destination IP is in the address group."

I haven't figured out a great way to get the Comcast gateway address automatically in the rule (there are discussions on next-hop-interface, but caveats that it isn't a good idea). For the moment I'm content to grab it and hardcode it. On the USG:

admin@ubnt:~$ show load-balance status  
Group wan_failover
  interface   : eth0
  carrier     : up
  status      : active
  gateway     : 172.16.0.1
  route table : 201
  weight      : 100%
  flows
      WAN Out : 349000
      WAN In  : 8
    Local Out : 1502

  interface   : eth2
  carrier     : up
  status      : failover
  gateway     : 66.56.48.1
  route table : 202
  weight      : 0%
  flows
      WAN Out : 53345
      WAN In  : 0
    Local Out : 397

eth2 is my Comcast interface (wan2 in the UI), so the gateway IP I need is 66.56.48.1.

Now, to apply:

configure
set protocols static table 5 route 0.0.0.0/0 next-hop 66.56.48.1
set firewall group address-group comcast 96.120.0.0/14
set firewall group address-group comcast 96.124.0.0/16
set firewall group address-group comcast 96.96.0.0/12
set firewall group address-group comcast 96.112.0.0/13
set firewall group address-group comcast 96.64.0.0/11
set firewall modify LOAD_BALANCE rule 2500 action modify
set firewall modify LOAD_BALANCE rule 2500 modify table 5
set firewall modify LOAD_BALANCE rule 2500 destination group address-group comcast
set firewall modify LOAD_BALANCE rule 2500 protocol all
commit
exit

Once it's committed, if you run traceroute xtvapi.cloudtv.comcast.net you should see it take a Comcast hop from the router instead of the AT&T hop.

Step 3: Persist the Config

Those commands on the USG just make the changes in memory. Now we need to ship the config to wherever we're running the Unifi management console. In my case, that's a Cloud Key Gen2. Following the guide I linked above, you'll dump the JSON config on the USG, strip it down to the bits we've done here, and then load that onto the Cloud Key as effectively an override file for the config input through the UI.

On the USG, you'll dump config using mca-ctrl -t dump-cfg > config.json. Then just get that JSON to a desktop.

From here I'm positive there's some magic jq incantation that could extract just the JSON leaf nodes we care about, but I couldn't figure it out.

JSON sections that matter to you (at least as of the time of writing... No clue how stable the config format is here):

firewall.group.address-group.comcast
firewall.modify.LOAD_BALANCE.rule.2500
protocols.static.table.5

Strip the JSON file down and then transfer it to the device running the management console. For me, running a Cloud Key, I needed to make it /srv/unifi/data/sites/default/config.gateway.json and chown unifi:unifi.

Once you've got the file in place, you can go into the management console, find the USG, Settings -> Manage -> Trigger Provision.

If you formatted the file correctly, the USG will cleanly load the config. If not, it'll get stuck in a startup loop. If you hit that case just move the config.gateway.json to something like config.gateway.json.tmp to let it start clean while you debug.

Eric Richardson lives in Atlanta, Georgia, where he works on payments infrastructure for Cash App.

Past lives involved streaming audio for Southern California Public Radio, running a hyperlocal news site for Downtown Los Angeles, working on custom mapping code, and writing Perl at the Jet Propulsion Labratory.

Previously on this date...

2004
almost there
I'm sitting in the Detroit airport right now, taking full advantage advantage of the wireless access I paid $7 for. We're here about two hours...
always missing something...
I hate that last minute panic of trying to figure out if you've remembered everything you need to take on a trip. Normally I forget my phone charger...
back, but not for long
I have to be quick this morning, since it's 8:35, I leave for the airport in two hours, and I haven't packed... The trip to Vegas was good. Traffic...