Sergey Nivens - Fotolia
Uptime is critical for online video service Hudl. To better manage the company's 1,000 servers and to dramatically cut staff time troubleshooting the system, Hudl implemented log analysis software.
Collegiate -- and, increasingly, high school -- athletics are bigtime businesses. So when teams invest in a particular service, they need to know it will work.
This is the problem and opportunity faced by Hudl, an online video storage and streaming service used by high school and college sports teams across the country. Hudl, based in Lincoln, Neb., lets team staff upload film from games and practices so that players and coaches can review it after the fact to see what the players are doing right and wrong on the field. Uptime is critical to the service's business model. If coaches can't access videos when they need them, the business would fail.
To ensure high uptime, Hudl has turned to log analysis software. Engineers use it to constantly monitor data streaming off servers to make sure everything is running smoothly and track down the source of any problems. They also use the data to track how customers use the service, which can streamline customer service. Representatives on the customer support team use log analytics to create a timeline of when videos were accessed or modified and by whom.
"If one coach on a team deletes some old video, but they don't tell the other coach, we can look it up and say, 'It looks like the other coach deleted it,' " said Jon Dokulil, vice president of engineering at Hudl.
Log analysis tool ensures uptime
The company uses a log analytics platform from software vendor Sumo Logic, based in Redwood City, Calif. The platform ingests data from Hudl's application servers, which mainly host customer-facing applications, and its database servers, which run a combination of MongoDB and SQL Server software.
Dokulil said Hudl has about 1,000 servers in total. This creates a lot of complexity. Having a system to analyze and parse all the data created by the servers is critical to ensure uptime.
"If you're trying to trace through something that touches 20 servers, it's nice to have one place to go," Dokulil said.
The Sumo Logic log analysis software aggregates all the data coming off the servers. Dokulil and his team have written around 500 searches that run against the server data every five minutes, and if anything falls outside certain predetermined ranges, the system will generate an alert. They've also created data dashboards that are routed to TVs set up throughout the office. These screens show customer service reps things such as how many calls, email messages and live chat requests they're receiving and how long it's taking them to respond. This can help service teams redirect their efforts if necessary.
For Dokulil, one of the best parts about Hudl's approach to log analytics is that it's built around a cloud system. The Sumo Logic platform is offered as a service, which Dokulil said saves the Hudl engineering staff huge amounts of time. They've been using the current system for about a year, but have been analyzing log data for about seven years. Prior to the current setup, they managed everything in-house.
As the company grew over the past seven years, so did the number of servers. This made managing the log analytics system an increasingly burdensome task. Dokulil said the staffer responsible for managing the old system would spend weeks at a time just troubleshooting.
"We're saving money from an infrastructure cost standpoint, but in my opinion, the biggest payoff is the guy ran the old system, we got all his time back," he said.
For security, analysts need to be able to search log files
Better log analytics needed for monitoring wireless networks
Learn to spot network attacks with log analytics