During SC18, ESnet staff will be working with other research organizations and vendors to support a series of demos at the forefront of networking, including the use of new hardware and software. These demos include:
SENSE: In collaboration with two universities and three national laboratories, ESnet will demonstrate its Software-defined Network for End-to-end Networked Science at Exascale research project.
FAUCET: ESnet is collaborating with the University of Waikato in New Zealand and the faucet Foundation on a very large, never-before-done deployment of the Faucet controller, which was created to bring the benefits of software-defined networking to a typical enterprise network and has been deployed in various settings.
Monon400: As part of the Network Research Exhibition (NRE) at SC18, ESnet is collaborating with Indiana University (IU), Ciena and Internet2 to power Monon400, the fastest network ever built for research and education.
This post is written by ESnet Director, Inder Monga, and shared through a personal lens, other perspectives may vary
I wanted to share a remarkable effort happening in the field of networking that is going to be shown in production at the Supercomputing 2018 as part of SCinet. Most of the people working in the SC booths (or even at SCinet) may never realize the role they are playing in moving the field of networking to the next level. Let me share some history [and you can skip the Background section to go straight to the main essence of the blog], and talk about the international, collaborative effort at play here.
Background: my software-driven journey
This is being shared by the perspective of the writer’s journey, other perspectives may vary
From the early 2000’s, the dream that excited me was building software layer that would manage networks and expose the network black-box as a set of ‘knobs and dials’ that applications could use to request and customize for their purposes. A small group of us at Nortel Networks came up with an idea of CO2 aka Content over Optics in 2001 – see the slide below.
This excitement was very new and led to my most productive set of patents filed during that time. Just as the concept was taking hold, an innovative research and education network, SURFnet, decided to deploy this concept on their newly built network, and the name evolved from CO2 to DRAC aka Dynamic Resource Allocation Controller.
Independently, ESnet was on a similar journey and started working on a concept called OSCARS aka ‘On-demand Secure Circuits and Reservation System’. As the community coalesced on the same direction, multiple initiatives like OSCARS, UCLP, DRAC, MANTICORE and others started merging towards an aligned software-driven network journey for the WAN. In the meantime, the Openflow effort from Stanford driven by an enterprise-based architecture, Ethane, captured the hearts of many, and the ‘Software-Defined Network’ (SDN) wave was born.
I am going to skip discussing the twists and turns of the evolution of SDN in industry, where there have been many recent articles on whether OpenFlow is dead. In fact there is a grassroots and under-the-radar SDN/Openflow effort that started in New Zealand called Faucet that proves these concepts are still alive and well, and are poised to thrive! In fact this work recently won the award in the New Zealand Open Source Software Project Award.
Ever since I was invited to join the Faucet Foundation Board, I was insistent that I ‘eat my own dog food’. With the help of Josh Bailey and Simeon Miteff, I began running a faucet controlled switch (OpenFlow is not dead) in my office as my only bridge into the Lab’s network and the Internet, i.e. if faucet did not work, I could not get connected! I am proud to say that other than one small self-inflicted upgrading glitch, the system has been working seamlessly over the past year no matter what I do or use in the office. You can see the couple of raspberry PIs running the SDN controller working with a commercial off-the-shelf switch in the image below.
[I wish it was cleaner picture, but I had to get everything together to click a photograph without my desk in the way.]
Then my colleague, Nick Buraglio, and I decided, why not wire our satellite branch office with faucet based networking than buy traditional switches? With Josh (Bailey)’s help, Nick has now wired the lives of eight of ESnet’s key employees to be dependent on faucet and he has been having a blast (no downtime there either)! Read more about his journey on his own private blog: http://www.forwardingplane.net/2018/11/faucet-enterprise-openflow-in-production/
Faucet at SCinet, SC18
I may have led you on and will keep the suspense on for a bit longer….Faucet is being showcased in SCinet as production SDN controller managing a portion of the booth networks. SC18 is in two weeks and I encourage all of you to stop by SCinet to see it working! Reach out to the faucet champions – Josh Bailey, Brad Cowie, Richard Nelson, Nick Buraglio and Kate Mace who can talk you through the intricacies of the faucet deployment there. Poseidon, as you see printed on one of the raspberry pi’s in the image above, is also being featured at SC as it integrates learning and security with faucet.
More details on this topic will be available after SC is over. In the meantime, do check out Nick’s blog for the engineering details. An image from the recent multi-vendor plugfest at SCinet staging in Dallas to – yes, vendors support this.
Look forward to seeing you all in Dallas as all of us continue on this journey
“Applications want more instrumentation from the network, and so do network engineers,” said Monga. “As the era of ‘gut-feel networking’ passes to analytics-driven networking, more and more data about networks, including the constituent flows, will need to be being tracked and retrieved. With networks becoming an effective sensor, new methods are being proposed to manage the streaming telemetry.”
Upgrades include a programmable API, graphical interface and ability to make multi-point reservations
The Energy Sciences Network’s (ESnet’s) award-winning OSCARS (On-demand Secure Circuits and Reservation System) has just been upgraded. The tool, now called OSCARS 1.0, allows network engineers to make multi-point bandwidth reservations and features a new programmable API and graphical user interface.
OSCARS is a software service that creates dedicated bandwidth channels for scientists who need to move massive, time-critical data sets around the world. The previous iteration, called OSCARS version 0.6, was finalized and released in 2012. In 2013, it was recognized as one of R&D Magazine’s top 100 technologies.
“OSCARS 1.0 brings a lot of engineering improvements to the tool,” said ESnet Engineer Evangelos Chaniotakis, who led the upgrade. “We simplified the architecture so that engineers can connect more tools. We also made improvements to pathfinder so that users can choose between different flavors of path finding. Now instead of just looking for the shortest path, users can make requests like ‘find me the path with the least amount of devices,’ which ensures resiliency.”
According to Chaniotakis, the most visible change to the tool is the user interface, which now has an almost fully graphical view of the network and the connections on it. Before, users would see this information in a spreadsheet. He notes that with a few clicks of the mouse, OSCAR 1.0 users can get a pretty complex network setup with information about quality of service, network constrains, and scheduling, among other data. OSCARS 1.0 will also continue to support the MSI protocol for international collaborations.
“We’ve deployed OSCARS 1.0 at a few selected sites, and it is considerably faster. I was able set up hundreds of connections with this updated version in the same time it took me to do 10 connections with the previous version,” says John MacAuley, an ESnet Software Architect, who helped test OSCARS 1.0.
The tool also allows engineers to automatically provision jobs on the network, which is especially useful for long-haul, day-to-day tasks for large experiments like the Large Hadron Collider, as well as research projects that require complicated network topologies. Chaniotakis notes that this automatic provisioning is less burdensome for network engineers who used to manually set up these tasks. It will also become a crucial tool as ESnet continues to grow.
Using a 5,000-mile network loop operated by ESnet, researchers at the Department of Energy’s SLAC National Accelerator Laboratory (SLAC) and Zettar Inc. recently transferred 1 petabyte in 29 hours, with encryption and checksumming, beating last year’s record by 5 hours, an almost 15 percent improvement.
The project is aimed at achieving the high data transfer rates needed to accommodate the amount of data to be generated by the Linac Coherent Light Source II (LCLS II), which is expected to come online in 2020. The LCLS is the world’s first hard X-ray free-electron laser (XFEL) and its strobe-like pulses are just a few millionths of a billionth of a second long, and a billion times brighter than previous X-ray sources. LCLS II will provide a major jump in capability – moving from 120 pulses per second to 1 million pulses per second. Scientists use LCLS to take crisp pictures of atomic motions, watch chemical reactions unfold, probe the properties of materials and explore fundamental processes in living things.
The increased capability is expected to generate data transfers of multiple terabits per second– as the experimental results are sent from SLAC to Department of Energy’s (DOE) supercomputing facilities for analysis. As the DOE’s dedicated network user facility for scientific research, ESnet carries data between universities and DOE’s national laboratories and national user facilities along a multi-100 gigabits-per-second (Gbps) backbone network.
This screen shot from MyESnet shows the southern links of the network dark red, indicating data transfers exceeding 50 Gbps during the 5,000-mile transfer conducted by SLAC and Zettar Inc.
As ESnet’s new chief security officer, Adam Slagell will head the facility’s security team and generate a security strategy for the network. He will also join the organization’s senior leadership team and provide valuable input into the growing network’s strategic direction.
Slagell comes to ESnet from the National Center for Supercomputing Applications (NCSA) where he spent the last 15 years of his career and was promoted several times. After joining NCSA in 2003 as a security engineer, he was promoted to Senior Security Engineer in 2007, then to Senior Security Engineer and Chief Information Security Officer in 2012, Assistant Director of the Cybersecurity Directorate and Chief Information Security Officer in 2013, and Cybersecurity and Networking Division Director and Chief Information Security Officer in 2016.
During this time at NCSA, he also served as principal investigator and co-principal investigator on a number of grants. He was co-PI for the Bro Project in collaboration with researchers in UC Berkeley’s International Computer Science Institute and Berkeley Lab’s Computational Research Division and served as the first chair of the Bro Project’s leadership team. He was also security operations co-lead and the security officer for the XSEDE federation, a single virtual system that scientists can use to interactively share computing resources, data, and expertise; and developed the policies and procedures for the first HIPAA enclave on the University of Illinois at Urbana-Champaign campus.
“What brings me to ESnet is a chance to do something new and still be close to open scientific research,” said Slagell. “For most of my career I’ve been focused on data center security, but at ESnet we’re looking at security on a wide area network. We are providing high-bandwidth services to a whole bunch of customers around the world and we have a responsibility to protect those resources.”
Born and raised in Northern Illinois, Slagell received his masters degree in computer science from the University of Illinois at Urbana-Champaign in 2003, a masters and bachelors degree in mathematics from Northern Illinois University in 2000 and 1999, respectively.
“I was really ahead of the game in mathematics,” said Slagell. “I was taking college-level math classes in high school and in college I got into cryptography primarily though independent study. This eventually led me to computer security, which I loved because it allowed me to do pure math while staying close to computers, which is something I’ve always been interested in.”
Slagell will be working remotely from Illinois. He is an active power-lifter and biker. He rides his bike 10 miles every day to get to work, even in Illinois winter and summer. In his spare time, he is active in a charity that his wife co-founded called CU Able, which is a support network for families of individuals with disabilities.
Sponsored by the DOE’s Office of Advanced Scientific Computing Research, the event will bring together experts from national labs, industry, government, and academia to identify research challenges and opportunities in the development and deployment of quantum networks to support distributed quantum information science (QIS) activities. The workshop will cover a spectrum of quantum communications networks and subsystems, including quantum interconnects, quantum local area networks, quantum metropolitan networks, and quantum wide area networks.
“While we are busy designing and developing our next-generation network ESnet6, we need to look ahead to the emerging quantum technologies that will impact our science users to ensure that we have a deep understanding of the technological issues and are able to invest in the appropriate infrastructure when needed,” said Monga, who is on the organizing committee for the QNet workshop and will co-lead a discussion on Operations and Control for Quantum Networks with Dr. Ben Yoo from the University of California, Davis and Dr. Cees de Laat from the University of Amsterdam, an affiliate of Berkeley Lab and a member of the QuSoft Research Center on Quantum Software.
Summer student interns have a lot to offer Berkeley Lab: they are motivated, energetic and can bring fresh perspectives to old problems. They also represent the next generation of researchers and engineers. To capture the best talent, ESnet has built an internship program that operates as a student-to-staff pipeline, channeling not only talent but diversity into Berkeley Lab, said Patricia Giuntoli, area lead for Networking and Systems at ESnet.
“Diversity is important in order to have unique and different views. Sometimes solving a problem or getting to a new place can be challenging, and having people with different experiences and views can help create unique solutions,” said Giuntoli.
This summer, four students have been working at ESnet. Three of them—Hocine Mahtout, from the Bordeaux Graduate School of Engineering; Kunal Singh, from UC Berkeley; and Aaron Jia, who is also attending UC Berkeley—are working with ESnet Research Scientist Mariam Kiran on a project that employs machine learning and parallel computing to optimize network traffic and path allocation. In 2017, Kiran received an Early Career Award to support this work.
Jia and Singh are developing algorithms to predict network traffic and deep-learning methods to model network traffic. This information can be used to prepare for intense server traffic and help designate server workload. When they are done with the research, they will each write a paper about their findings and present their findings at SC18 in Dallas, TX.
Mahtout is also working on a separate project with Kiran. A few years ago, Kiran started working on iNDIRA, a software package that translates a user command written in English (not a programming language) into network commands. After seeing Kiran talk about iNDIRA in a presentation, Mahtout independently contacted Kiran to discuss ways they might work together. They later launched a second version of the project called EVIAN that incorporates machine learning to improve future interactions with users.
We did not intend on creating a new version of iNDIRA, but because Mahtout was so excited about working on it we now have a second version of iNDIRA coming out,” Kiran said. “As an intern you see the environment and the types of science happening here and making a difference to everyday life, and it’s so exciting that it makes you want to be part of it all.”
Meanwhile, the fourth student—John Christman, a senior from University of Nevada, Reno whose father, John Christman Sr., recently retired from ESnet after 29 years—is working with ESnet Network Engineer Nick Buraglio to improve ESnet security and traffic analysis software. Christman, who also worked at ESnet last summer doing network traffic analysis, is working with the monitoring group to improve ESnet’s networks by getting new servers and networks up and running and securing them from cybersecurity attacks.
“John has been tasked with working on the prototyping team for a next-generation management network design,” said Nick Buraglio, ESnet network engineer and Christman’s mentor. “John has excelled at both the tasks of leading pieces of a larger project as well as operating as a high performing team member. He has gained experience with completing deliverables that are critical pieces of a larger project.”
At the University of Nevada Reno, Christman is majoring in Information Systems, which is similar to computer science except geared towards use in enterprise. He plans to get his masters degree in this subject at UC Berkeley.
“I’ve learned an incredible amount while I’ve been here. From hearing presentations from world-class scientist to gaining hands-on experience, this summer has been filled with new insights on cutting-edge information,” he said. “My expectations have not only been met but exceeded. I look forward to coming in everyday, as each day brings something new and exciting. I’m truly blessed to be interning here this summer.”
One of ESnet’s goals with the summer student program is to give students the opportunity to gain a working knowledge of ESnet that might lead them to join the ESnet team once they graduate. Several ESnet employees, including Kiran, Sowmya Balasubramanian and Scott Richmond, began their careers at the lab as interns.
“My goal was to educate myself and gain practical experience in network operations so I could leverage that into a career in industry and the laboratory was a well-respected place that was accepting interns,” said Richmond, who is now a network engineer at ESnet. “After spending nearly a year at ESnet as a student assistant, the training and education I received from my mentors and colleagues in software, hardware and networking allowed me to transition to a full-time career as a Computer Systems Engineer with ESnet, working in the Operations and Deployment group.”
Some of this summer’s interns may just follow suit.
“This internship has gotten me thinking about what I want to do in the future,” said Jia. “I had never had formal work experience before, and this gave me a great idea of what it’s like.”
Last week, several ESnet staff members organized and participated in the second annual National Research Platform (NRP) workshop. Held August 6-7, 2018, at Montana State University in Bozeman, Montana, the workshop brought together representatives from more than 100 key institutions that are working to create a nationwide science data network.
Co-sponsored by a National Science Foundation (NSF) award to the California Institute for Telecommunications and Information Technology (Calit2) at the University of California, San Diego (UCSD), CENIC, ESnet and Internet2 and hosted by Montana State University, the workshop was designed to identify best strategies for enabling high-performance data transfer via interoperable Science DMZs and data transfer nodes at a national scale.