ESnet Renews, Upgrades Transatlantic Network Connections

ESnet_Final_Logos_All_Wordmark_SpellOut_RGBThree years after ESnet first deployed its own transatlantic networking connection, the project is now being upgraded to four 100 gigabits-per-second links. These links gives researchers at America’s national laboratories and universities ultra-fast access to scientific data from the Large Hadron Collider (LHC) and other research sites in Europe.

The original configuration that went into service in December 2014 consisted of three 100 Gbps and one 40 Gbps links. Since December 2014, the LHC traffic being carried by ESnet alone has grown nearly 1600%, from 1.7 Petabytes per month in January 2015, to nearly 30 Petabytes per month in August 2017.

The four new connections link peering points in New York City and London, Boston and Amsterdam, New York and London, and Washington, D.C. and CERN in Switzerland. The contracts are with three different telecom carriers.

“Our initial approach was to build in redundancy in terms of both infrastructure and vendors and the past three years proved the validity of that idea,” said ESnet Director Inder Monga. “So, we stuck with those design principles while upgrading the fourth link to 100G.”

Overall goals of the new agreements accomplished:

  • Increase in overall capacity to meet projected demand
  • Reduction in the overall cost
  • Increase in the diversity of the cable systems providing ESnet circuits, and
  • Maintain as much R&E network community transatlantic cable diversity as possible, including that of the Advanced North Atlantic Collaboration.

Another new component is a collaboration with Indiana University funded by the National Science Foundation with its Networks for European, American and African Research (NEAAR) award within the International Research Network Connections (IRNC) program. The goal of NEAAR is to make science data from Africa, such as that collected by the Square Kilometer Array, and Europe, like data from CERN’s Large Hadron Collider, available to a broader research community.

With the upgrade, the total transatlantic capacity for Research and Education networks  is now 800 Gbps, continuing the close collaboration between the seven partners providing transatlantic connectivity under the broader umbrella of the Global Network Architecture Initiative (GNA).

ESnet’s Science DMZ Design Could Help Transfer, Protect Medical Research Data

As medicine becomes more data-intensive, Medical Science DMZ eyed as secure solution

Like other sciences, medical research is generating increasingly large datasets as doctors track health trends, the spread of diseases, genetic causes of illness and the like. Effectively using this data for efforts ranging from stopping the spread of deadly viruses to creating precision medicine treatments for individuals will be greatly accelerated by the secure sharing of the data, while also protecting individual privacy.

In a paper published Friday, Oct. 6 by the Journal of the American Medical Informatics Association, a group of researchers led by Sean Peisert of the Department of Energy’s (DOE) Lawrence Berkeley National Laboratory (Berkeley Lab) wrote that the Science DMZ architecture developed for moving large data sets quick and securely could be adapted to meet the needs of the medical research community.

“You can’t just take the medical data from one site and drop it straight in to another site because of the policy constraints on that data,” said Eli Dart, a network engineer at the Department of Energy’s Energy Sciences Network (ESnet) who is a co-author of the paper. “But as members of a society, our health could benefit if the medical science community can become more productive in terms of accessing relevant data.”

Read the full story.

Medical Science DMZ
Schematic showing components of the Medical Science DMZ.

ESnet Congratulates the LIGO Visionaries on their 2017 Nobel Prize in Physics

ESnet congratulates Barry Barish and Kip Thorne of Caltech and Rainer Weiss of MIT on receiving the 2017 Nobel Prize in Physics for their vision and leadership of the LIGO Laboratory. Their discovery of gravitational waves, made just two years ago, culminates decades of effort. ESnet is proud to have played a role in supporting this achievement.

LIGO’s Hanford facility in Washington was an early adopter of ESnet’s OSCARS, the On-Demand Secure Circuits and Advance Reservation System, for guaranteed bandwidth services in 2005 for early development. In fact, the project was one of the very first users of the OSCARS service.

Last year, ESnet upgraded the Hanford LIGO sites network connection to Seattle with a dedicated 10 Gbps link, which complemented a shared 10 Gbps link to Boise. The Hanford site consistently moves about 400 megabits of data per second to Caltech in Southern California.

ESnet provides 10 Gbps connectivity to the Hanford LIGO Observatory in southeast Washington, linking the site to Caltech and the international research community.

You can see the real-time data transfer rates and other details of this connection on the MyESnet portal.

Lastly, we are also working with Caltech to improve end-to-end bandwidth at the campus as part of theascr-funded SENSE (SDN for End-to-end Networked Science at the Exascale) project. By improving scientific workflows and end-site driven intelligent services to increase data throughput, the project will help LIGO in using high throughput data transfer methods.

Again, congratulations to our LIGO colleagues and we look forward to continuing to support your research mission.

Read how Berkeley Lab’s distributed computing experts developed software to help the LIGO manage the distribution of data from the experiment.

For a great explanation of the LIGO project, read this NASA Jet Propulsion Laboratory blog.

ESnet, Internet2 Renew Critical Exchange Point Contract with NYSERNet


The Department of Energy’s Energy Sciences Network (ESnet) and Internet2 — two of the nation’s leading research and education networks — today announced the renewal of an agreement to remain anchor tenants at one of the world’s most critical Internet exchange points operated by the New York State Education and Research Network (NYSERNet).

Located at 32 Avenue of the Americas in Manhattan, NYSERNet’s “32AofA” global network exchange is a well-known international hub where the world’s leading research and education networks connect to content, data and telecom providers to seamlessly exchange traffic among their networks.

“One network alone cannot connect every scientist in the lab, every student in a classroom or every researcher in the field. By creating a rich interconnected fabric of networks, we are able to bring together the best ideas, minds and scientific resources no matter where in the world they may be,” said Inder Monga, director of ESnet and the Scientific Networking Division of Lawrence Berkeley National Laboratory. “This is how discovery in the era of big data will take place. We appreciate our continuing partnership with NYSERNet and Internet2 to provide this truly critical network connection.”

Read the full story.

ESnet’s Mariam Kiran Earns DOE Early Career Award

Mariam Kiran
Mariam Kiran

Mariam Kiran, a research scientist in the Energy Sciences Network’s (ESnet’s) Advanced Network Technologies Group, has received a 2017 Early Career Research Program award from the Department of Energy’s (DOE’s) Office of Science. Now in its eighth year, the award supports exceptional researchers during critical stages of their formative work by funding their research for five years.

Kiran will use her award to advance the state of the art in network research. She will employ methods from machine-learning and parallel computing to work with network research, to optimize traffic and path allocation.

“Networking is an interesting field utilizing multiple hardware and software skills. For example, configuring links involve understanding current network topologies, as well as, anticipating traffic demands and user requirements. ESnet has already been at a forefront of networking research with advanced monitoring tools and network expertise,” Kiran said. “However, as networks grow and become more complex, we have to find new methods to automate some or all of current network tasks. These include anticipating problems in advance and automating the ‘fixes’ to maintain a healthy network environment.”

Mariam Kiran’s demo on “InDI: Intent-based User-defined Service Deployment over Multi-Domain SDN applications” was one of the more popular sessions in the DOE booth at SC16.

Read the full story.

Patrick Dorn to Lead ESnet’s Network Engineering Group


Patrick Dorn, a network engineer who joined ESnet in 2011, has been named the new leader of ESnet’s Network Engineering Group. He has held the job in an acting capacity since last September.

During his time with ESnet, Dorn spent a year at CERN in Switzerland, working to establish high speed links between CERN and the U.S. research community.

Before joining ESnet, Dorn was a senior network engineer at the National Center for Supercomputing Applications in Urbana-Champaign, Illinois. At NCSA he held both technical and management roles.

While at NCSA, Dorn served as the SC)08 conferencechair for SCinet, the high-speed network that provides wired and wireless connectivity for the thousands of attendees. SCinet is entirely volunteer-driven and takes more than a year to plan and then deploy.

SLAC, AIC and Zettar Move Petabyte Datasets at Unprecedented Speed via ESnet

Twice a year, ESnet staff meet with managers and researchers associated with each of the DOE Office of Science program offices to look toward the future of networking requirements and then take the planning steps to keep networking capabilities out in front of those demands.

Network engineers and researchers at DOE national labs take a similar forward-looking approach. Earlier this year, DOE’s SLAC National Accelerator Laboratory (SLAC) teamed up with AIC and Zettar and tapped into ESnet’s 100G backbone network to repeatedly transfer 1-petabyte files in 1.4 days over a 5,000-mile portion of ESnet’s production network. Even with the transfer bandwidth capped at 80Gbps, the milestone demo resulted in transfer rates five times faster than other technologies. The demo data accounted for a third of all ESnet traffic during the tests. Les Cottrell from SLAC presented the results at the ESnet Site Coordinators meeting (ESCC) held at Lawrence Berkeley National Laboratory in May 2017.


The test loop ran from 5,000-mile loop that goes from Department of Energy’s SLAC National Accelerator Laboratory (SLAC) in Menlo Park, Calif. across the country to Atlanta and then back to SLAC. The data transfers are part of the experiment to handle expected amounts of data generated by experiments at SLAC’s planned Linear Coherent Light Source II ( LCLS-II).

“Collaborations like this provide the networking community with an opportunity to use a production network for testing new technologies and seeing how they perform in a real-world scenario,” said ESnet Director Inder Monga. “At the same time, ESnet also gets to learn about leading-edge products as part of our future planning process.”

Read the AICCI/Zettar news release.

Read the story in insideHPC.

ESnet’s John Paul Jones Caps 33-Year Career at National Labs

John Jones in his signature blue beret.

Soon after John Paul Jones moved from Idaho to California in 1983, he and his wife visited the Berkeley Hat Company, where he bought a royal blue beret. Since then, during his 33+ years at Lawrence Livermore and Lawrence Berkeley national labs, the flat blue hat has become part of Jones’ persona.

But when he retires from ESnet at the end of June 2017, Jones said he may also think about hanging up that hat. Around the house, he said, he usually wears his blue and gold Golden State Warriors cap.

In 1995, the Department of Energy made the decision to move ESnet and NERSC from Livermore to Lawrence Berkeley National Laboratory. Jones knew people who were part of the ESnet team at Livermore and it piqued his interest when ESnet’s then-manager Jim Leighton called him in to talk about joining the group.

“He unrolled this big network map and showed it to me,” Jones recalled. “I said, ‘What!? Oh yeah – I am definitely in!’”

When ESnet made the move in 1996, Jones joined the group that configured, installed, maintained and did troubleshooting on the routers that powered the national network.

Read more about JP’s life and career.

How Brian Tierney’s “Aha moment” Turned into a 28-year Career at Berkeley Lab, ESnet

Brian Tierney

As he prepares to retire this month after more than 28 years at Lawrence Berkeley National Laboratory, Brian Tierney, head of ESnet’s Advanced Network Technologies Group, still remembers the exact moment when he knew where his career path would lead.

“I met Bill Johnston at San Francisco State and on the very first day of his Computer Graphics class, he told us ‘Anybody who gets and A in my class gets an internship in my group,’” Tierney recalled. “A light bulb went off and I knew I was going to get an A. I literally thought “That’s what I might do for the next 30 years.’”

He started in Johnston’s Graphics Group as a graduate student assistant in 1988 and a year later Tierney had become a career staff member.

Among the key projects Tierney has either contributed to are perfSONAR, the network performance toolkit, and, a collection of tips and tools for, well, faster data transfers.

An iconic photo of Brian circa 1998

Read more about Brian’s career and retirement plans.

Kennedy High IT Students Spend a Week at the Lab Making Connections

As ESnet’s Sowmya Balasubramanian (left) and LaRue Moore (right) look on, Kennedy IT Academy students Michelle and Luis evaluate the performance of network nodes.

Ten students from the IT Academy at Richmond’s Kennedy High School spent the first week of their summer vacation getting hands-on experience in high-speed networking and getting first-hand advice on planning their future.

The students and IT Academy lead teacher LaRue Moore participated in the June 12-16 pilot workshop introducing them to networking for science. The five-day workshop include a 30-minute instructional presentations followed by 30 minutes of hands-on work, a sequence developed by Sowmya Balasubramanian of ESnet. Topics included configuring IP addresses, tracing packets, assessing network performance and locating bottlenecks.

On the last day, students were given the assignment: You are a network administrator. You have five Raspberry Pis that serve as data transfer nodes. They are connected to a switch that can process at 1000 megabits/second. The Raspberry Pi themselves can transfer at 100 megabits/second. A user wants to use one of the data transfer nodes and has approached you for help in finding the best node. You need to run tests to find which is the best node.

Working in teams, the students measured the round-trip time for each node and then balance speed against packet loss to determine which performed best. Students then presented their findings to the group, as they would make a recommendation to an IT expert.

“This was extremely valuable for our students and now I want to see how we can scale it up to 40 students,” Moore said. “This week has given them both more knowledge and more confidence.”

Read the full story.

View a photo feature of the workshop.