Simulated production of a Higgs event in ATLAS

ATLAS Computing and Muon Calibration Center

SuperComputing 2014

SuperComputing 2014AGLT2Caltech University of VictoriaUniversity of Michigan

One Server, 100 Gbps over the WAN for ATLAS/LHC

Intelligent Software Driven Dynamic Hybrid Networks With Terabit/sec Science Data Flows

Motivations - 100 Gbps Demo

High energy physics needs to deal with extensive amounts of data for its globally distributed collaborations.

This type of distributed data-intensive science is no longer unique; many areas of science and engineering are evolving to be data-intensive, collaborative and broadly distributed.

How can we support fast, efficient access to large-scale data to expedite “time-to-discovery”?

System Design - 100 Gbps Demo

Goal: A single server capable of delivering ~100 Gigabits per second of disk read and write performance over the the Wide Area Network (WAN)

For large data centers: The system can be used as a fast WAN transfer cache.

For smaller data centers: The system can deliver push and pull data at 100G using a single 4U server. Small sites can’t afford to deploy many servers to source and sink data over the wide-area network. A single server is far easier to deploy and maintain.

100Gbps Demonstration Overview

SC14 Demo Diagram

The demonstration used a 100GE Dedicated Link from the University of Victoria to the Caltech and University of Michigan booths on the SC14 show floor.

Local Network Read from Dell R920 Using IO read peaks at 96 Gbps

Throughput Graph

In the Local Area Network, the system is capable of transferring 2 Terabytes of Data roughly every 2 minutes.

SanDisk Fusion ioMemory

SanDisk Fusion ioMemory Cards

SanDisk FusionIO 2.6 TB ioDrive2 Duo Solid State Storage Cards.

SanDisk has produced a press release with additional information and comments as well as links to more information on the Fusion ioMemory product.

“SanDisk is thrilled to be working with the teams at the University of Michigan and University of Victoria to help fuel their success by providing fast, cost-effective and highly scalable flash solutions to increase data access,” said Sumit Sadana, executive vice president and chief strategy officer, SanDisk. “By utilizing flash technology, the researchers can cost effectively transfer massive amounts of data over long distances, ultimately enabling them to reach new discoveries faster.”

Similarly configured Dell R920 systems were installed at University of Victoria, the Caltech booth, and the University of Michigan booth.

Victoria Setup

Victoria Setup

Configuration at the University of Victoria. Can power over 100G data transfer in only 10 Rack Units.

CFP2 Optics

Closeup of new smaller form factor CFP2 100G optics

Intelligent Software Driven Dynamic Hybrid Networks With Terabit/sec Science Data Flows

SC14 Network Complete Overview

During the conference a team led by the California Institute of Technology (Caltech), the University of Victoria, and the University of Michigan, together with teams from SPRACE Sao Paulo, FIU, Vanderbilt and other partners worked together to smash the previous records for data transfers using the software defined networking (SDN). This work was recently published in NDM14 under the title: Towards managed terabit/s scientific data flows.

Locally at AGLT2 we hosted a Dell MD3460 storage system linked through a Juniper EX9208 to participate in this demo.

Dell MD3460 Storage