Saturday night, prime time, Cowboys at Chargers. “Shortly before kickoff,” tweeted @SoFiStadium, “we experienced an outage with our internal broadcast system. We are resetting and expect all systems to be up and running shortly.” It’s a nightmare for any venue: minutes before a major event, the stadium scoreboard goes black.
It’s easy for angry fans and sports radio jocks to snark about something like this. But talk to anybody that works at a stadium, or builds the broadcast and production systems inside of them, and the reaction is more grounded.
“What if that happens here?”
Let’s be honest. If it can happen at SoFi, one of the best engineered and staffed facilities in the world, it can happen at your venue. A better question to ask is “What’s our plan? What do we do when something goes down?”
We’ve got a list to help you figure it out:
- Make A Plan
It sounds obvious, but the time to prepare is -before- things go wrong. “Start at your sign and work your way backwards” says CTI’s John Kvatek. Formerly University of Central Florida’s Associate Athletics Director, Multimedia and Creative, Kvatek adds, “If something fails, and it takes your board down, what can you do? You need to work through the whole chain.” It’s easy to switch out a camera. But if the switch or video router goes down? Can you fall back on a physical patch panel? Find out before you need to!
- Identify Single Points Of Failure. Eliminate Them If Possible.
Broadcast standards require redundancy and failover. If you’re generating millions of dollars per hour in advertising, you can’t afford to go off the air. At its most extreme, you’ll find live broadcast facilities with two completely separate electrical systems running to everything in the studio. Is this overkill for a small college? Absolutely. Is having a spare TriCaster or Broadcast Pix box on hand too much? Find how much ad-revenue you’ll lose if the game doesn’t stream, or how the coaching staff reacts when recruits can’t watch!
- Have Spares On Hand
Data centers often have a have a hot spare drive in the server, and a rack of extra drives ready to replace them. “If you have the budget, notes Kvatek, “configure critical components, and air gap them. Don’t leave them plugged in or on the network. Store them on a shelf, so they’re ready for an emergency.”
- Protect Your System From Power Anomalies!
Check your gear to see if it’s plugged into an Uninterruptible Power Supply, or UPS. If it’s not, get one. Even minor power fluctuations can cause your gear to bluescreen and reboot. A complex system might require every component to be restarted in the right order, and that can drastically increase your downtime. The best way to prevent that is by preventing lightning, blackouts, and brown outs from taking down your system in the first place. Plug everything into a UPS!
- How Do You Show The Clock And Score If The Main Display Goes Down???
The rules of most sports let the officials keep the time, at least if they’re on or near the playing surface. That will keep the game going, but if the players and fans can’t see the score or time? That creates a host of other issues, and you can expect an ugly response. You need a redundant way to keep clock and score feeds visible if the main display goes down!
- Document Everything!
Make sure your team documents the plan and the system. Build a book with checklists, procedures, and contact information. “Bob knows how to fix that” doesn’t help if Bob is out sick or has moved on to another job. And make sure there is a hard copy: if there’s a major power outage, you might not have access to the network or a computer!
7. What’s Next?
Looking to upgrade your stadium technology? Need some help working out your plan? Contact CTI!
