Azure datacenters: a look behind the scenes
Be it the storage of data, mail traffic or the use of software applications: cloud computing has become an integral part of our everyday lives. Companies as well as private households are increasingly relying on the cloud. Have you ever wondered which infrastructure supports it?
We took a look behind the scenes of Azure, Microsoft’s cloud computing platform. On the virtual “Inside Datacenter Tour – Azure Datacenter Operations”, we gained interesting insights that we would like to share with you.
The first mega datacenter built outside the US was DB3 near Dublin. Given the increasing demand for cloud services, more datacenters have been constructed since its completion in 2009. Thanks to drone images, what’s difficult to see from the ground becomes clear from a bird’s eye view: the facility is gigantic!
During construction, Microsoft attached great importance to efficiency, operationalized as Power Usage Effectiveness (PUE). To calculate the PUE, you divide the energy consumption of the entire facility (lighting, cooling,…) by the energy consumption of the IT equipment only (computers, storage,…). That’s how you determine how much energy actually flows into data processing.
Thanks to free air cooling, the PUE of DB3 is 1.25. DB4 and DB5, which were built later, have a PUE of 1.25. For comparison: the average is 1.76. Put differently, an average facility consumes 76 cents in addition to every euro that touches the server.
To cover the energy demand, Microsoft is constructing a power plant on site in Dublin. On the Amsterdam campus in the Netherlands, 60 wind turbines delivering 180 megawatts will be built. Under the ground, these datacenters host water retreatment plants to recycle water for cooling. Each plant has a capacity of a million liters.
In addition, air handling units on the roof of the datacenters are used for cooling. Fun fact: this cooling practice is based on a principle already know to the ancient Persians. Outside air is passed through fans to increase its volume. Then, the air flows over a surface of wet cooling material. This causes a temperature drop of six degrees Celsius while consuming relatively little water and mechanical energy.
Different ways of construction
In Quincy, Microsoft runs a datacenter consisting of preassembled components. You can think of them as datacenter containers equipped with up to 2,000 servers. Given their modular design, these datacenters can be set up faster and cheaper. For example, they support Hotmail, Skype or Outlook.
Of course, geographic and climatic conditions are also taken into consideration when designing data centers. In Singapore, for example, there is less space available. That’s why Microsoft chose a vertical design. Moreover, the climate is warmer and more humid, requiring more cooling. Keeping the PUE low is more difficult under these conditions.
Factors like demand, infrastructure and man power play a crucial role when choosing datacenter locations. Apropos: you won’t find network engineers running around with patch cables in the datacenters anymore. Software Defined Networking is used to centrally control data traffic without having to manually configure the switches that connect different network segments.
The server names sound like blockbuster titles: “Godzilla”, “Beast” or “Beast v2”. Microsoft has its own department specifying the architecture and design of servers. OEM vendors like HP or Dell then build the servers. When developing the “Gen 2 Blade” servers in 2011, Microsoft hoped that they would stick around for a long time and support different types of workloads.
However, it soon became clear that you needed a veritable “computing monster” to run databases in the cloud. That was the starting signal for “Godzilla”, a server with comparatively large random access memory (512 GiB) and Solid State Drives (SSD) for more storage capacity (9 x 800 GB). Further server generations, designed for different processing types, followed.
Within the scope of project Catapult, Microsoft studied the development of programmable hardware. Looking for more computing power and alternatives to CPU, they explored field-programmable gate arrays (FPGAs). It soon became apparent that these chip sets could deliver high performance given their speed, programmability and flexibility. For example, they were used to improve the search engine Bing.
Today, almost every server in the Azure datacenters contains an FPGA. For instance, the FPGA can function as a remote accelerator for distributed computing by placing it between the top-of-rack network switches of the datacenter and the network interface chip of the server. That’s how Microsoft routes the entire network traffic through the FPGA.
As far as the eye can see, server racks are lined up in the aisles of the datacenters. The units in the lower part of the racks host over one petabyte per box on conventional hard disks. The smaller units on top support 512 terabyte of SSD memory.
With Micron SCUTI-O, Microsoft uses the fastest SSD of the world. This allows to analyze huge amounts of data in real-time. Micron Scuti-O employs 3D XPoint, a memory technology that is based on changes of electrical resistance.
In a traditional network environment, you can address about 200 racks by one electrical switch. The goal of Microsoft’s project Sirius is to develop an optical, data-center-wide network.
In contrast to the hierarchy of electrical switches common today, this network would be completely flat and could comprise thousands of racks. These optical networks could provide better performance, be more reliable and cost less.
Datacenters consume a lot of energy. However, a study by Microsoft came to the result that Azure is up to 93% more energy efficient than local solutions. Moreover, when the use of renewable energy is taken into account, the cloud platform is up to 98% more carbon efficient.
For example, Microsoft obtains 315 megawatts from solar plants in Virginia, and 180 megawatts from wind turbines in the Netherlands. In Sweden, Microsoft will build zero-waste datacenters powered exclusively by renewable energy.
Microsoft plans to be carbon negative by 2030. In addition, they want to simplify the supply chain by producing the energy on site wherever possible. The shorter the supply chain, the less energy loss and potential points of failure there will be. This allows to provide more data with less resources.
In the Stark pilot project, fuel cells in units sitting on top of the server racks transform natural gas to energy. The energy is then directly fed to the server racks. As the fuel cells produce CO2 in addition to water, researchers started working on hydrogen fuel cells.
Being the second largest network of the world, Azure unfortunately also attracts malicious traffic. Each day, they analyze over six trillion thread signals. In distributed denial of service (DDoS) attacks, for example, many requests from different sources deliberately block applications.
To prevent this, a program constantly compares the actual traffic utilization with predefined critical thresholds. As soon as these thresholds are exceeded, it automatically initiates a DDoS mitigation. The malicious data traffic is removed, the normal one continues to flow.
A multilayer security system ensures the security of the data centers. Access requires passing a number of checkpoints, including a multi-factor biometric entry point. This video gives an overview.
Microsoft destroys discarded devices directly on site and retains the corresponding reports. Non-recyclable materials are avoided. According to the principles of isolation and segregation, no service management takes place in the data centers. Operation facilities managing the software are located elsewhere.
Cooling presents a particular challenge. The energy demand of the servers will probably continue to increase. Liquid cooling could provide a remedy since it can handle higher heat fluxes better than air. Research in this field is still ongoing.
Among others, Microsoft is experimenting with two-phase immersion. This means that you immerse the server in a non-corrosive, non-toxic fluid with a low boiling point. The liquid evaporates, rises, cools and condenses. They only lost three percent of liquid per year.
In another current research project, Microsoft studies the feasibility of underwater datacenters. A few weeks ago, they successfully completed the second phase of project Natick. It turned out that underwater datacenters do make sense from an economic and logistic point of view. For two years, a twelve meter long cylinder containing more than 800 servers had lied at the bottom of the Scottish sea.
The failure rate was eight times lower than on land. This is because the atmosphere of dry nitrogen in the underwater datacenter protects against corrosion. Another reason is that there were no people working and bumping on the components.
Since underwater datacenters don’t require freshwater for cooling, they are considered sustainable. In addition, they consume renewable energy. With project Natick V3, the next step is to sink twelve cylinders off the coast.
The demand for long-term data storage in the cloud is growing. To meet the demand in the future, Microsoft’s project Silica investigates how to store data in quartz glass by using ultrafast laser optics. A small piece of glass can store hundreds of terabytes of data.
Machine learning algorithms can decode the data encoded by the laser. The glass is extremely resistant, for example, you could boil or demagnetize it. This enables safe and long-term data storage.
Huge facilities, renewable energy supply, efficient hardware, massive storage capacity, outstanding energy efficiency, strict security measures and state-of-the-art projects – on our look behind the scenes of Azure datacenters we encountered many superlatives.
More on the cloud
Bella Diekmann works for the editorial office of agiles. She studied Spanish and French and has a PhD in linguistics. Besides her passion for languages, she is very interested in current IT topics. Blogging for agiles allows her to combine her love of writing with her interest in computer science and informatics.