Network Matters: Updates from DOE's Energy Sciences Network
perfSONAR is a set of tools that help engineers actively monitor & troubleshoot multi-domain network issues. It provides the ability to pinpoint and solve service problems immediately to restore the highest levels of network performance
ESnet, Network Startup Resource Center Combine Expertise to Spread the Word
For members of the established research and education (R&E) networking community, attending conferences or sitting in on workshop sessions is the normal way to learn about the latest equipment, architecture, tools and technologies.
But for network engineers striving to establish basic R&E infrastructure where bandwidth and other resources are scarce, the University of Oregon’s Network Startup Resource Center (NSRC) is often the primary information conduit. NSRC staff travel to emerging nations in Africa, Asia-Pacific, Middle East and South America where they hold intensive hands-on training courses combined with direct engineering assistance to bring institutions up to speed.
And for the second time in a year, ESnet and the NSRC have produced and released a library of short explanatory videos to help network engineers around the world gain basic knowledge, set up basic systems and drill down into areas of specific interest. In December, 15 videos detailing the Science DMZ network architecture were posted, covering the background and structure, specific designs, and techniques and technology.
“The goal is to make the information more accessible to networking staff, in the U.S. and particularly in emerging economic areas where institutions are trying to bootstrap a research network,” said ESnet Network Engineer Eli Dart, who developed the Science DMZ concept with Brent Draney of the National Energy Research Scientific Computing Center (NERSC). Both ESnet and NERSC are DOE Office of Science User Facilities managed by Lawrence Berkeley National Laboratory.
Created in 1986, the U.S. Department of Energy’s (DOE’s) Energy Sciences Network (ESnet) is a high-performance network built to support unclassified science research. ESnet connects more than 40 DOE research sites—including the entire National Laboratory system, supercomputing facilities and major scientific instruments—as well as hundreds of other science networks around the world and the Internet.
Step 9: Why just rent fiber? Pick up your own dark fiber network at a bargain price for future expansion. In the meantime, boost your bandwidth to 100G for everyone. (2012)
Step 10: Here’s a cool idea, come up with a new network design so that scientists moving REALLY BIG DATASETS can safely avoid institutional firewalls, call it the Science DMZ, and get research moving faster at universities around the country. (2012)
Step 12: 100G is fast, but it’s time to get ready for 400G. To pave the way, ESnet installs a production 400G network between facilities in Berkeley and Oakland, Calif., and even provides a 400G testbed so network engineers can get up to speed on the technology. (2015)
Step 13: Celebrate 30 years as a research and education network leader, but keep looking forward to the next level. (2016)
A joint effort between ESnet, Internet2, Indiana University, and GEANT, the pan-European research network, perfSONAR is a tool for end-to-end monitoring and troubleshooting of multi-domain network performance. In January 2014, perfSONAR reached a milestone with 1,000 instances of the diagnostic software installed on networking hosts around the U.S. and in 13 other countries. perfSONAR provides network engineers with the ability to test and measure network performance, as well as to archive data in order to pinpoint and solve service problems that may span multiple networks and international boundaries.
At the workshop, Tierney will give an introduction to perfSONAR and present a session on debugging using the software. Zurawski will talk about maintaining a perfSONAR node, describe some user case studies and success stories, discuss “Pulling it All Together – perfSONAR as a Regional Asset” and conclude with “perfSONAR at 10 Years: Cleaning Networks & Disrupting Operation.”
Since it was first released about five years ago, the perfSONAR network measurement toolkit has provided the research and education networking community with tools for end-to-end monitoring and troubleshooting of multi-domain network performance. And over the years, this ability to diagnose network problems has become increasingly important as research is increasingly collaborative and dependent on sharing large data sets.
The latest release of perfSONAR, version 3.4, gives network engineers access to more data about network performance as well as increased security protections. The new version will be discussed in a session called “perfSONAR 3.4: Not Just another Incremental Update” at the Technical Exchange conference being held from Oct. 19-25 in Indianapolis. perfSONAR is developed by a collaboration between the Department of Energy’s ESnet, Internet2, Indiana University and GEANT, the pan-European research network.
perfSONAR provides network engineers with the ability to test and measure network performance, as well as to archive data in order to pinpoint and solve performance problems that may span multiple networks and international boundaries.
Berkeley Lab staff from ESnet, NERSC and the IT Division will be among the presenters at the 2014 Technology Exchange, a leading technical event in the global research and education networking community. The annual meeting is co-organized by ESnet and Internet2. The conference will be held Oct. 27-30 in Indianapolis.
Among the topics to be addressed by Berkeley Lab staff are ESnet’s recently announced 100 gigabits-per-second connections to Europe, the newest release of the perSONAR network measurement software library, Science DMZs and cyber security.
The annual meeting brings together a wide range of technical experts to address the challenges facing the research and education networking community as it supports data-intensive research. The conference will be hosted this year by Indiana University.
The perfSONAR-PS project celebrated a milestone by surpassing 1000 deployed software instances in December of 2013. The perfSONAR software is designed to assist network operators and end users with the task of monitoring end-to-end performance, and assisting with debugging tasks in the event that problems arise. As the number of deployments grows, the software becomes more effective by offering better coverage across more network paths, including ESnet and the connectivity to Department of Energy and NSF funded resources. perfSONAR-PS is a joint effort between ESnet, Fermilab, Georgia Tech, Indiana University, Internet2, SLAC, and the University of Delaware.
Need to troubleshoot some network performance problems? perfSONAR-PS is an open source development effort to create a colletion of easy-to-use and easy-to-install perfSONAR network performance monitoring services and tools. Version 3.2.1 of the pS-Performance Toolkit is now available for download. This update contains: new throughput graphs, new delay/loss graphs, scheduled traceroute tests and numerous bug fixes. Release notes are available to denote all changes since the 3.2 release. Please visit the pS Performance Toolkit page for more information.
Debugging network performance problems on long multi-domain paths that cross the Atlantic has just become easier for ESnet users. The GÉANT network, a pan-European communications infrastructure serving Europe’s research and education community, has improved the perfSONAR performance monitoring software architecture and toolset it co-developed with ESnet, enhancing the ability of users in the U.S. and Europe to spot red flags in network performance.
PerfSONAR (or as GÉANT calls the European implementation, perfSONAR MDM or Multi-Domain Monitoring) software, enables users to pinpoint trouble spots and bolster end-to-end network performance across network domains—and now across the Atlantic. This improvement to perfSONAR will provide ESnet users enhanced visibility and interoperability, speed the identification of performance issues and help prevent delays caused by engineers working across multiple time zones. European and U.S. users can now monitor network performance via the same intuitive interface using consistent data formats. Specifically, it allows ESnet users to run tests from 8 of the more than 70 ESnet measurement points across the country, including Fermilab in Chicago, IL, Brookhaven National Laboratory in Upton, NY, and Lawrence Berkeley National Laboratory in Berkeley, CA to measurement points distributed across the GÉANT European network and vice versa.
This combination of ESnet and European perfSONAR measurement points provides a comprehensive end-to-end view of network performance, invaluable for global data exchange in projects such as CERN’s Large Hadron Collider.
Is your network up to the data demands of the LHC?
This spring ESnet achieved something akin to global presence, figuratively with our network, and in-person at conferences as we traded ideas with the technical community, such as the limitations of bandwidth on demand, and how to compose services that are easy for end users to understand and use. In May Steve Cotter, Bill Johnston, and Inder Monga were invited speakers at the TERENA Networking Conference in Prague.
Inder Monga followed with a presentation on “Network Service Interface: Concepts and Architecture,” that discussed the motivation, concepts and architecture in the upcoming Open Grid Forum standard that has the promise to enable researchers simple abstract constructs to dynamically create and manage their communication infrastructure to serve their science. During the talk he explained some of the differentiating attributes of the protocol: Recursive and flexible request and response framework, abstraction of physical topology into a service layer representation—and declared that composable services are the next logical step in network design. The key to dealing with complex infrastructure is to abstract it into objects the users can understand, but that is just the beginning. A composable services model contains essential elements like abstracted technical requirements in a language that all users can understand, failsafe backups, service changes that are transparent, transport efficiency and monitoring for “soft” failures. He pointed out that a Topology Service would be the next target for standardization once the Connection Service was fully specified.
Steve Cotter talked about meeting user expectations in “Fighting a Culture of ‘Bad is Good Enough,” asserting that bandwidth on demand on its own is inadequate to meet the growing needs of science. In ESnet’s surveys scientists report that while the technology is often there, they don’t know how to access it or how to make it work. The result is that poor network performance is often the norm at various sites and scientists are left to fend for themselves without technical assistance. Frustrated, many simply give up attempting to send data via the network and instead use ‘sneakernet’. But it doesn’t have to be this way. Cotter cited the LHC as one example of investment of time and commitment to do networking right. “For us as a community to succeed, we need to provide intuitive services to researchers, and documentation and assistance to make it easy for them.” said Cotter, before he launched into a run-down of new ESnet tools and ventures.
Now that OSCARS version 0.6 is code-complete, ESnet is taking offers to help test the code. ESnet is also working with its sites to build secure, dedicated enclaves on the perimeters of networks, dubbed Science DMZs, which are fully instrumented with perfSONAR. Separating the campus science traffic from converged network services like VOIP makes it easier to debug and improves performance across the WAN. To make it easier to test and troubleshoot infrastructure, ESnet has created a community knowledge base, http://fasterdata.es.net that regularly receives more than 2500 hits a week. ESnet is also developing a multi-function web portal called MyESnet that it will launch at ESCC/Joint Techs in a few weeks. MyESnet will have lots of tools and new features for the scientist and networker, including: traffic flow visualizations, high-level information about ‘site health’, the ESnet maintenance calendar, a discussion forum and idea repository, as well as one-stop shop where users will be able to log in with Shibboleth or OpenID, initiate perfSONAR tests, and open trouble tickets.
Going beyond just bandwidth on demand
Three weeks later at the NORDUnet conference in Reykjavik, Iceland, Inder Monga discussed the ins and outs of developing composable network services on demand. Given new developments in network virtualization, co-scheduling, cloud services and 100G bandwidth, the network is playing an ever larger role in providing scientists new services.
Incidentally, Inder used high-speed networking to accomplish the enviable feat of being two places at once without violating any laws of physics. Upon landing in Iceland, Inder promptly presented a talk on green networking from Iceland for the conference on Green and Sustainable ICT in Delhi, India.
Designing “greener” networks is one of ESnet’s key priorities, and something you will be hearing more about from ESnet in the future.