Facebook’s IT Upgrade Challenge

In fall 2009, Facebook was heading toward a data storage crisis.

Despite aggressively adding servers and data centers, the social networking giant couldn’t keep up with its own outrageous growth.

Between January and September, the number of Facebook users had more than doubled, to 300 million. Every new member meant more data to be stored: an average of 90 new pieces of content per month per person. On top of that, the company was constantly adding more and better features.

Hyper growth meant Facebook was busting out of a key data center that stored a quadrillion bytes – a petabyte – of information critical to maintaining operations for users and employees.

But outgrowing the data center wasn’t an option. Because that particular center stored a cluster of data-intensive distributed applications, going over capacity would have crippled the business.

No matter that a new center wasn’t in Facebook’s annual information-technology budget. The company needed it ASAP. Facebook’s capacity engineering and analysis group, led by manager Jason Taylor, estimated they had about four months to find the money, build the center, buy the technology to put in it and transfer the data before the existing center would max out.

At many companies, launching a project of that magnitude on short notice in the middle of a recession would have been impossible.

But Facebook completed the multimillion-dollar project on time and on budget – and according to the company, without a moment of downtime for users – in part because the IT and finance departments worked on it side-by-side from day one.

“It’s all about partnering,” says David Gardner, manager of the company’s group technical program, which led the project. “Facebook is constantly looking for innovative ways to deliver projects faster and more efficiently, and breaking down barriers between finance and IT does that,” he says.

The Benefits of Bringing IT, Finance Together
While Facebook is as tech-driven as they get, you don’t have to be a tech company to benefit from cross-functional partnerships between IT and finance.

At most companies, finance, operations and IT work separately, which makes it harder to justify major IT expenditures, because while the department may buy the equipment, operations pays the electric bills. But to get the most from IT investments, the groups need to work together, says Chris Mines, a senior vice president and research director at Forrester, the Cambridge, Massachusetts, technology researcher. “They should be joined at the hip.”

That means the finance team needs to understand how IT projects and goals meet overall business strategies, and IT needs to learn the language of the business. “When you take a holistic approach to IT decision making, it demystifies the department, and makes these projects less about the technology and more about the business,” Mines says.

At Facebook, Taylor’s capacity engineering and analysis team includes dedicated finance department employees and analysts who work with technical engineers from IT to define the costs and benefits of investments and upgrades, evaluate technology and align IT infrastructure costs with corporate strategy. “Having us all in one group makes us better able to solve problems that typically come with supporting major capital expenditures,” Taylor says. “That gives us a competitive advantage.”

In September 2009, that cross-functional dynamic enabled them to quickly create a business case for building and managing the new data center.

Following a standard upgrade protocol, analysts on Taylor’s team conducted a technical assessment of why IT needed to add capacity. Next they reviewed five top server brands and processors to see which offered the best long-term return on the investment. The difference in cost was critical because, for the project, Facebook would be buying several thousand servers and related equipment.

To determine which server offered the best per-dollar, per-watt performance, Facebook’s in-house lab tested models for reliability, performance and speed, and used that information to come up with projected power costs over three years. Because electricity is a big part of Facebook’s operating expenses, choosing servers that require less energy could dramatically reduce the total cost while increasing data center capacity. Data centers are limited by their power needs, so the less energy each server uses, the more servers a center can house. Low-energy servers also would reduce the center’s carbon footprint, an ongoing sustainability goal, Gardner says.

When the team analyzed three-year projected operating costs, the difference in server costs was dramatic, Taylor says. “All five vendors met our technical goals, but there was a 30 to 40 percent swing in total cost of ownership.”

Making the Case
After the technology review, the team combined those projected costs with costs for infrastructure and facility maintenance to create a three-year financial model that became part of their business case for expanding the data center.

The model included the projected costs of each server to demonstrate how energy efficiency and each server’s capacity would lead to a lower total cost – even though they commanded a higher up-front price. “We didn’t just go in saying ‘We want to buy these servers,’” Taylor says. “We had data to support our decision.”

Gardner projects that the new servers, which hold four times as much data as the previous generation while consuming less power, will save the company millions of dollars in energy costs over the next three years.

All told, it took the team four weeks to develop the business case for the new center, and two weeks to win approval. The project was financed through a fund Facebook maintains for special projects. In the weeks that followed, Taylor’s team bought the technology and contracted for construction of the facility. When the time arrived, they transferred the data using a private Facebook network that linked directly to the new center, an activity that took three weeks.

By the end of December 2009, less than four months after initially identifying the need, the team completed the project on time and on budget.

Fast forward to September 2010. Facebook ended that month with more than 500 million active users, and the now year-old data center is nearing capacity once again. According to Taylor’s projections, it will run out of space within six months. As a result, Facebook is preparing to build yet another data center, again with support from the finance department.

Taylor expects the new project to run as smoothly as the last one. “These projects are never about justifying our costs, they are about making the best decisions for the business,” he says. “We are all trying to solve the same problem: How do we grow the business in a cost effective manner? The partnership between my group and finance helps us do that.”