Vote for ESnet5 – a good steward of Petabytes

It has almost been a year since we turned 25, and transferred a “whole universe of data” at Supercomputing 2011 – and that was over a single 100G link between NERSC and Seattle. Now we are close to the end of building out the fifth generation of our network, ESnet5.

In order to minimize the downtime for the sites, we are building ESnet5 parallel to ESnet4, with just a configuration-driven switch of traffic from one network to the other. Since the scientific community we serve depends on the network to be up, it’s important to have assurance that the transition is not disruptive in anyway. The question we have heard over and over again from some of our users – when you switch the ESnet4 production traffic to ESnet5, how confident are you that the whole network will work, and not collapse?

In this blog post, I’d like to introduce an innovative testing concept the ESnet network engineering team (with special kudos to Chris Tracy) developed and implemented to address this very problem.

The goal of our testing was to ensure that the entire set of backbone network ports would perform solidly at full 100 Gbps saturation with no packet loss, over a 24 hour period. However we had some limitations. With only one Ixia test-set with 100 GE cards at hand to generate and receive packets and not enough time to ship that equipment to every PoP and test each link, we had to create a test scenario that would generate confidence that all the deployed routers and optical hardware, optics, the fiber connections, and the underlying fiber would performing flawlessly in production.

This implied creating a scenario where the 100 Gbps traffic stream being generated by the Ixia would be carried bi-directionally over every router interface deployed in ESnet5, traverse it only once and cover the entire ESnet5 topology before being directed back to the test hardware. A creative traffic loop was created that traversed the entire footprint, and we called it the ‘Snake Test’. Even though the first possible solution was used to create the ‘snake’, I am wondering if this could be framed as a NP-hard theoretical computer science and optimization approach known as the traveling salesman problem for more complex topologies?

The diagram below illustrates the test topology:

So after sending around 1.2 petabytes of data in 24 hours, and accounting for surprise fiber maintenance events that caused the link to flap, the engineering team was happy to see a zero loss situation.

Here’s a sample portion of the data collected:

Automation is key – utility scripts had been built to do things like load/unload the config from the routers, poll the firewall counters (to check for loss ingress/egress at every interface), clear stats, parse the resulting log files and turn them into CSV (a snapshot you see in the picture) for analysis.

Phew! – the transition from ESnet4 to ESnet5 continues without a hitch. Watch out for the completion news, it may come quicker than you think…..

On behalf of the ESnet Engineering team

Science Data Weathers the Storm

Hurricane Sandy took a terrible toll this week, and our thoughts remain with the millions of people whose lives were impacted.

From the nightly news, we learned about critical systems that temporarily failed during the extreme weather, and others that continued to function.  I’d like to share an example in the second category.

Here’s a screenshot from our public web portal, http://my.es.net, showing traffic to and from Brookhaven National Laboratory on Long Island, during the worst of the storm:

Network traffic to and from Long Island’s Brookhaven National Laboratory was not impacted by the devastating storm that struck the New York area this week.

Why does it matter that Brookhaven’s connection to ESnet, and to the broader Internet, remained functional during the disaster? It matters because modern science has entered an age of extreme-scale data. In more and more fields, scientific discovery depends on data mobility, as huge data sets from experiments or simulations move to computational and analysis facilities around the globe. Large-scale science instruments are now designed around the premise that high-speed research networks exist. In this kind of architecture, loss of network connectivity impairs scientific productivity. In the worst case, unique and vital data can be lost.

Much of the network traffic in these graphs can be attributed to ATLAS, one of the extraordinary experiments at CERN’s Large Hadron Collider, outside of Geneva. It’s been a great year at CERN, with announcement of a Higgs-like particle this summer. At ESnet, we were very happy that productivity of the ATLAS experiment was not affected by the massive storm.

Although the research networking community may have benefited from good fortune during storm, it’s important to recognize that a lot of careful planning and sound engineering – conducted over the course of many years – contributed to this outcome. Research networks like ESnet and Internet2, regional networks like NYSERNet, campus networking groups at Brookhaven and elsewhere, and exchange points like MAN LAN all work very hard to harden themselves against individual points of failure. As with all engineering, the devil is in the details: fiber diversity, elimination of shared fate in single conduits and data centers, location and specification of generators, availability of fuel, optical protection schemes, failover for dynamic circuits… the list goes on and on.

We won’t always be so fortunate, but the screenshot above is something our research networking community can take pride in. My thanks to the hundreds of people whose sound decisions made this good outcome possible.

The Risks of Not Deploying IPv6 in the R&E Community

Observations from ESnet’s resident IPv6 Expert Michael Sinatra

When having discussions with CIOs of various colleges, universities, and national laboratories, I often hear about such issues as “risk,” “return on investment,” “up-front-costs,” “CAPEX/OPEX,” and the like. When the topic turns to IPv6, costs are cited as well as potential risks involved with adopting IPv6. However, any good risk assessment should include risks and costs of not doing something as well as doing it. Until recently, much of the risk of not deploying IPv6 was centered around running out of IPv4 addresses and not much more. Organizations that had a lot of IPv4 addresses (or thought they did) presumably didn’t have to consider such risks. In the discussion below, I note several more risks of not deploying IPv6, advantages of IPv6, and reasons to move forward. This discussion can be combined with more traditional risks and costs associated with deploying IPv6, to provide the seeds of a more complete risk assessment.

Adoption, not migration

It’s important to understand that adoption, not migration is of principal concern. It is widely understood that IPv4 will remain active for some time and that a dual-stack environment–where networks, computers, and other devices all run both IPv4 and IPv6 simultaneously–is still the “best” way to achieve an IPv6 transition. My concern is principally about the adoption portion, where we add IPv6 functionality to networks and hosts, in order to achieve a dual-stack environment. Indeed, this assumes a reasonable abundance of IPv4 addresses within an organization.

Risk 1: Security

Conventional wisdom has it that adopting IPv6 brings with it a range of security issues, namely owing to the traditionally poor support for IPv6 in security appliances. Although open-source firewalls have long had near-parity between IPv4 and IPv6, many proprietary firewall and IDS devices have lacked sufficient IPv6 features. While it is true that security equipment has in the past turned a blind eye to IPv6, this is changing rapidly as vendors move to support IPv6.

Nevertheless, there are risks to ignoring IPv6 on the campus or the lab site. Because of the widespread and largely “default-on” support of IPv6 tunneling technologies, such as 6to4 and Teredo, IPv6 tunnels can and do easily exist on IPv4-only networks. Security devices which don’t understand IPv6 are unlikely to understand these tunneling technologies, and they will be unable to peel open the tunnel layers to see what’s really going on inside the tunnels.

Many black-hats and grey-hats understand this and will attempt to use IPv6 to transport illegal peer-to-peer content and malware. If the bad guys are adopting IPv6, the good guys need to adopt it as well so they can see the bad stuff and clean it up. Some IDSes and firewalls do understand IPv6 and they even understand the tunneling protocols. While it’s possible to block wholesale some of the tunneling protocols, it isn’t easy to block them all–especially if there are legitimate users of tunnels on your network. Moreover, blocking protocols at the border or at a router doesn’t block it within your enterprise or within individual LANs.

If you haven’t been including IPv6 support (and, ideally, feature-parity between IPv4 and IPv6) in your purchasing decisions and RFPs for security equipment, you are exposing yourself to this risk. Ignoring IPv6 simply won’t keep it off your network. However, adopting IPv6 as a natively-routed protocol will bring most of the tunneled traffic out in the open as native IPv6 traffic, where it will be easier to detect anomalies. That, combined with making IPv6 a key consideration in purchasing decisions will help mitigate the security risk.

Of course, purchasing life-cycles often span three- or five-year periods. That’s why it’s crucial to start thinking about IPv6 now, so that you can get IPv6 requirements embedded into purchasing processes before you really need IPv6. This is true for both network (routers, switches) and security equipment. Realizing you need IPv6 after a purchasing cycle has completed is not a good position to be in.

Risk 2: Your eyeballs

Of course, I am speaking here of “eyeball networks”–networks with client computers that access content and data. For colleges and universities, these include your wireless nets, your residence hall networks, and lab and research networks that consume and process data. That last category is also prevalent at national lab sites. Many of these organizations feel that they have enough IPv4 address to satisfy network growth for several years. However, that does not mitigate the risks that these organizations face: That eyeball networks–even those with abundant IPv4 address space–will still need access to IPv6 content and data, and that “several years’” worth of IPv4 address space still may not be enough.

As is the case with security, ignoring the need for IPv6 on those eyeball networks also poses risks. While users may be perfectly happy consuming content and data over IPv4, there is no guarantee that that content and data will always be available over IPv4. Indeed, with the recent run-out of IANA’s IPv4 free-pool, and of the Asia-Pacific region’s IPv4 address space, IPv4 address space is becoming scarce in the larger Internet. Secondary markets are beginning to open in IPv4 address space, and the prices this far have been around $10 per IP address–far more expensive than most R&E organizations are used to spending (or for that matter, can afford). Government and foundation grants are unlikely to support shopping for IPv4 addresses on the secondary market, and they will tend to view IPv6 as a more viable (and cheaper) alternative.

New colleges and universities will not have access to an abundance of IPv4 addresses. Moreover, as new scientific sites and special instruments come on-line around the world, it is increasingly likely that those (especially in Europe and Asia) will have access to fewer and fewer IPv4 addresses. These research centers will have two options: Entirely forego IPv4 addresses or get a very small number of IPv4 addresses and run some sort of NAT and/or protocol translation to support IPv4. In this latter scenario, IPv6 can be supported end-to-end, due to address abundance, while the limited IPv4 space requires middleboxes.

ESnet’s work with the Science DMZ concept has revealed that middleboxes have a detrimental effect on network performance for data-intensive science applications. The lack of a clean end-to-end path for IPv4 will mean that IPv4 can only be supported as a legacy “slow-path” protocol. For real performance, IPv6 will be necessary. Even legacy support for IPv4 may not be available in certain regions for much longer.

A reasonable scenario that may be encountered within the IT departments of research institutions–universities and national labs–is as follows: Faculty members and research staff will need access to data from a particular instrument or reactor. They will either be unable to get the data they need over IPv4 or they will have to go through a middlebox or bastion to get to the data, and that will have serious performance implications for the researchers. They will approach the IT department and request IPv6. Knowing that lead times for such requests do not often extend past the order of hours or days begs the question, will you be ready when a researcher comes to you and asks for IPv6 connectivity in her lab “by the end of the week”?

Risk 3: Your Content

As the developing world mobilizes economically, corporations, foundations, entrepreneurs, benefactors, and prospective students will increasingly hail from countries such as China, India, Brazil, and other parts of Asia and Latin America. These prospective donors, collaborators, and scholars will need access to information resources at your university or lab. And increasingly, these people will have better access to IPv6 than to IPv4. The next-generation research network in China, CERNET2, is IPv6-only, for example.

Moreover, this is not solely true in the developing world. As ISPs in the US and Europe become further strapped for IPv4 resources, they will turn to large-scale NAT (LSN, aka “carrier-grade NAT”). LSN promises to reduce performance and increase troubleshooting headaches. Avoiding these pitfalls requires IPv6 support, which is why large ISPs like Comcast are proceeding aggressively with IPv6 adoption. As the effects of IPv4 run-out become more pronounced, more people will be trying to access your information resources via IPv6. Will they be able to reach you easily?

As campus development and public-affairs offices continue to push the outreach envelope, using social media and a variety of Internet-based technologies, they will want to ensure that they are reaching the maximum range of benefactors and prospective students. How will you answer the vice president of development when he asks you if your institution is doing all it can to make information resources available to the maximum range of prospective donors?  How will you respond when a researcher at your lab site asks about how to improve real-time collaboration experiences with partners in India?  How will you ensure the director of admissions that prospective students in China, India, and Brazil will have easy access to admissions materials and information about programs of study?  IPv6 plays an important role in answering these questions. How well you answer them depends on how well positioned you are for IPv6 adoption.

Risk 4: Even if you have a lot of IPv4 addresses, you don’t have enough

There are a lot of applications for which NAT, or the use of private IPv4 addresses without NAT, is “good enough.” Home networks frequently use NAT. Large high-performance compute clusters (HPCCs) frequently use private IPv4 space to number internal nodes. Small labs may use a consumer-grade NAT device at your campus or site. In many cases, these devices work fine. But often, the network could work much better if each device could be individually addressed.

In the case of HPCCs, I frequently encounter cases where two clusters at disparate sites use the same private IPv4 range (usually the lower end of 10.0.0.0/8). When the cluster owners decide to connect the HPCCs together via a private layer-2 or layer-1 (wave) link, they suddenly have address collisions. I have seen several cases where rounds of iterative negotiation are needed to properly renumber hosts into non-colliding ranges. Surprisingly, this is not infrequent in HPCCs, and it certainly has the potential to occur in many other applications. IPv6 solves this problem in two ways: First, because of its massive address space, a chunk of globally-routable address space can be used to number hosts with little impact. In the HPCC example, a single /64 can number all of the internal nodes in both clusters!  Even if a similar “private” address space is desired, IPv6 provides a mechanism called Unique Local Addressing, which allows different sites to “create” their own private address space. The algorithm specified by RFC 4193 allows for a high likelihood of uniqueness, so that if and when clusters are eventually merged, address collisions won’t occur. ULAs aren’t an exact replacement for IPv4 private addresses, but they are useful in certain circumstances, such as this HPCC example.

In this case, the use of IPv6 lowers the risk of escalating OPEX costs in maintaining a private address space. Moreover, the internal nodes could be numbered using EUI-64 based stateless autoconfiguration (SLAAC), further lowering costs. Because of the closed nature of the network, SLAAC may be a good candidate for an easy and maintainable configuration. Using EUI-64, which is based on the hardware addresses of the physical interfaces, makes documentation easy (hardware addresses are often easily known in these clusters, so hosts files and internal DNS can easily be generated from the known MAC addresses) and greatly reduces the likelihood of number collisions.

While large-scale HPCCs can benefit from IPv6, it can also help with small-scale NAT installations. Even in my own home network, I find IPv6 to be valuable over the existing IPv4 NAT system. I often need to manage individual hosts and sometimes, I need end-to-end transparency for such things as video and voice conferencing. Using IPv6 is much easier and more efficient than trying to poke holes or configure special redirects in my NAT box. I can access individual hosts at home directly over IPv6 without having to go through a NAT box. Now, some people may view this as a security risk–to them having my hosts “exposed” on the Internet is a big risk that NAT otherwise “solves.”

I view this security issue not as a risk but as a benefit. Instead of my security policy being dictated by the technology, I am able to develop my own security policy for my home network and use stateful firewalls to enforce that policy technically. Moreover, this produces a much cleaner security policy than having to place redirects and other kludges in my NAT configuration to support video conferencing and other needs.

IT readiness: The overarching risk

How many large-scale IT projects in your organization finish on-schedule, let alone can be completed on short notice? When that PR person or researcher comes to you and needs IPv6 “real soon now,” do you really want to be in the position of having never enabled IPv6 on any of your networks, or–worse yet–having no plan for IPv6 adoption?  You certainly don’t want to wait until a prominent member of your scientific staff or faculty is demanding IPv6 on her network before you even start thinking about IPv6, do you?

IT projects are hard. They have a lot of dependencies. They have a lot of unforeseen obstacles. And, of course, there are risks that arise from deploying IPv6, as there are with any large IT project. The big problem for IT managers will occur when the risks of not deploying IPv6 begin to outweigh the risks of deploying IPv6, and there’s suddenly a lot of pressure to move forward quickly. How do you ensure that things aren’t moving too quickly?  How do you mitigate all of the risks, or at least the majority?

There is a simple answer: Start before you really need to. Ideally, you should have already started and you may even be enabling IPv6 on networks and services right now. But if you haven’t begun yet, now is the time to start. There are a number of things you need to do just to get going, and ESnet has put together a useful checklist to get you started. You’re going to run into problems–all will definitely not be smooth sailing. That’s why you need to start adopting IPv6 before one of the risks that I have identified comes to bear. Even if you can’t deploy IPv6 on your production network just yet, you can get a feel for how it works and what the pitfalls are by creating a special “IPv6 DMZ.”  Better yet, if you plan to build a Science DMZ, make sure that it supports both IPv4 and IPv6 from day one. That will go a long way toward ensuring that you and your colleagues and staff fully understand IPv6, and it will provide improved connectivity options for the Science DMZ itself.

By now it should be clear that none of the risks I have identified are mitigated in any way by how much IPv4 address space you have. Simply put, having lots of IPv4 address space–even your own /8–is not reason enough to delay IPv6 implementation for one second. Your IPv4 address space no longer matters in the IPv6 equation. In this increasingly interconnected world, you need to be able to reach everyone else, and they need to be able to reach you. If you delay adopting IPv6, you make it less likely that your resources will be available to all, and that poses risks to you and your institution.

New MyESnet Interface Launched for SC11

The ESnet tools team is pleased to announce that they are launching a brand new interface to http://my.es.net that will showcase real time statistics on the 100 Gbps demos running at SC11.

This interface is designed to provide the community with access to multiple live visualization tools of 100 Gbps network utilization and topology.  The tools team continues to build out the MyESnet portal to meet the evolving needs of the community—we will keep you posted on new developments.

ESnet is supporting eight 100 Gbps community projects at SC11:
·      Brookhaven National Laboratory with Stony Brook University: Booth 2443
·      Indiana University with collaborators Brocade, Ciena, Data Direct Networks, IBM, Whamcloud, and ZIH: Booth 2239
·      Lawrence Berkeley National Laboratory, Booth 512
·      NASA with collaborators from MAX, International Center for Advanced Research (iCAIR) at Northwestern University, Laboratory for Advanced Computing at University of Chicago, Open Cloud Consortium, Ciena, Alcatel, StarLight, MREN, Fujitsu, Brocade, Force 10, Juniper, Arista: Booths 615, 2615, 635
·      Orange Silicon Valley with the InfiniBand Trade Association and OpenFabrics Alliance:  Booth 6010
·      The California Institute of Technology: Booth 1223
·      Fermi National Accelerator Laboratory and University of California, San Diego: Booth 1203 and 1213.

For a schedule of booth demos, click here.

–Jon Dugan

At 25, ESnet Transfers a Universe of Data

ESnet turns 25 this year. This anniversary marks a major inflection point in our evolution as a network in terms of bandwidth, capability, and control. We are running data through our network a billion—that is 10⁶—times faster than when ESnet was established.   Yet, we are still facing even greater demands for bandwidth as the amount of scientific data explodes and global collaborations expand.

Created in 1986 to meet the Department of Energy’s needs for moving research data and enabling scientists to access remote scientific facilities, ESnet combined two networks serving researchers in fusion research; (MFEnet) and high-energy physics (HEPnet). But ESnet’s roots actually stretch back to the mid-1970s, when staff at the CTR Computer Center at Lawrence Livermore National Laboratory installed four acoustic modems on a borrowed CDC 6600 computer. Our technology morphed over the years from fractional T1s to T1s to DS3, then to ATM, and in the last ten years we have moved to packet over SONET—all driven by the needs of our thousands of scientific users.

Over the last 25 years, ESnet has built an organization of excellence driven by the DOE science mission. We have consistently provided reliable service even as the science areas we support—high energy physics, climate studies, and cosmology, to name a few—have become exponentially more data-intensive. These fields especially rely on ever more powerful supercomputers to do data crunching and modeling. Other fields, such as genomics, are growing at a rapid pace as sequencing technologies become cheaper and more distributed.

Based on the dizzying trajectory of change in scientific computing technology, we urgently need to act now to expand the capacity of scientific networks in order to stay ahead of the demand in years to come.  At this point the ESnet high-speed network carries between 7 and 10 petabytes of data monthly (a single petabyte is equivalent to 13.3 years of HD video). The level of ESnet data traffic is increasing an average of 10 times every 4 years, steeply propelled by the rise in data produced. More powerful supercomputers can create more accurate models, for instance, of drought and rainfall patterns—but greater resolution requires vastly bigger datasets. Scientific collaborations can include thousands of researchers exchanging time-sensitive data around the world. Specialized facilities like the Large Hadron Collider and digital sky surveys produce torrents of data for global distribution. All these factors are poised to overload networks and slow scientific progress.

Tonight we face our biggest milestone yet: We are launching the first phase of our brand new 100 gigabit per second (Gbps) network, currently the fastest scientific network in the world today. Called the Advanced Networking Initiative (ANI), this prototype network forms the foundation of a soon-to-be permanent national network that will vastly expand our future data handling capacity.

The ANI prototype network will initially link researchers to DOE’s supercomputing centers: the National Energy Research Scientific Computing Center (NERSC) at Berkeley Lab, Oak Ridge Leadership Computing Facility (OLCF) , and Argonne Leadership Computing Facility (ALCF), as well as MANLAN, the Manhattan Landing Exchange Point.  ESnet will also deploy 100 Gbps-capable systems throughout the San Francisco Bay Area and Chicago. The ANI prototype network will then transition into a permanent, national 100 Gbps network, paving the way to an eventual terabit-scale DOE network.

To prepare the way, ESnet acquired almost 13,000 miles of dark fiber. This gives the DOE unprecedented control over the network and its assets, enabling upgrades and expansion to capacity and capabilities according to the needs of our scientists. By owning the fiber, we are able to lock in the cost of future upgrades for decades to come.

The third facet of ANI is a nationwide testbed which is being made available to researchers both in the public and private sector as a first of its kind platform to test experimental approaches to new network protocols and architectures in a greater than 10 Gbps network environment.

Researchers are already working on multiple experiments investigating emerging technologies like Remote Direct Memory Access (RDMA)-enabled data transfers at 40 Gigabits per second (Gbps) or new TCP congestion control algorithms that scale to 100Gbps and beyond. By creating a research testbed, ESnet will enable researchers to safely experiment with disruptive technologies that will build the next generation Internet—something impossible to do on networks that also carry daily production traffic.

Bringing You the Universe at 100 Gbps speed

Just this past week our engineers completed the installation of the first phase of network – nearly 6 weeks ahead of schedule. Tonight at the SC11 conference we are showcasing the inauguration of our brand new 100 Gbps network by demonstrating how high-capacity networks can open up the universe – or at least a highly sophisticated computer simulation of the early days of the universe.  The demo will include the transfer of over 5 terabytes of data over the new network from NERSC in Oakland, CA to the Washington State Convention Center in Seattle.

This demonstration is important as astrophysicists are interested in studying high-resolution simulations to better understand the complex structural patterns in the universe. ESnet’s new 100 Gbps network will enable scientists to interactively examine large datasets at full spatial and temporal fidelity without compromising data integrity. This novel capability will also enable remotely located scientists to gain insights from large data volumes located at DOE supercomputing facilities such as NERSC. For comparison purposes, a simulation utilizing a 10 Gpbs network connection will also be displayed on a complementary screen to showcase the vast difference in quality that a magnitude difference of bandwidth can bring to scientific discovery.

To view the 100 Gbps and 10 Gbps simulations, visit: http://www.es.net/RandD/advanced-networking-initiative/

In addition to this great demo, seven other community collaborations will be leveraging the new ESnet 100 Gbps network to support innovative HPC demonstrations at SC11.

As we toast to this great new accomplishment this evening, we recognize that we are building on an amazing 25-year legacy in network innovation. We owe tremendous gratitude to the ESnet staff both past and present for their over two decades of hard work and dedication that has been keenly focused on helping the DOE community solve some of society’s greatest challenges.  Given our team’s accomplishments to date, I cannot wait to see what this next chapter in our history brings.

–Steve Cotter

You are invited: Raise a toast with us at SC11!

 

ECSEL leverages OpenFlow to demonstrate new network directions

ESnet and its collaborators successfully completed three days of demonstrating its End-to-End Circuit Service at Layer 2 (ECSEL) software at the Open Networking Summit held at Stanford a couple of weeks ago. Our goal is to build “zero-configuration circuits” to help science applications seamlessly use networks for optimized end-to-end data transport. ECSEL, developed in collaboration with NEC, Indiana University, and the University of Delaware builds on some exciting new conceptual thinking in networking.

Wrangling Big Data 

To put ECSEL in context, the proliferating tide of scientific data flows – anticipated at 2 petabytes per second as planned large-scale experiments get in motion – is already challenging networks to be exponentially more efficient. Wide area networks have vastly increased bandwidth and enable flexible, distributed, scientific workflows that involve connecting multiple scientific labs to a supercomputing site, a university campus, or even a cloud data center.

Heavy network traffic to come

The increasing adoption of distributed, service-oriented computing means that resource and vendor independence for service delivery is a key priority for users. Users expect seamless end-to-end performance and want the ability to send data flows on demand, no matter how many domains and service providers are involved.  The hitch is that even though the Wide Area Network (WAN) can have turbocharged bandwidth, at these exponentially increasing rates of network traffic even a small blockage in the network can seriously impair the flow of data, trapping users in a situation resembling commute conditions on sluggish California freeways. These scientific data transport challenges that we and other R&E networks face are just a taste of what the commercial world will encounter with the increasing popularity of cloud computing and service-driven cloud storage.

Abstracting a solution

One of the key feedback from application developers, scientists and end-users is that they do not want to deal with the complexity at the infrastructure level while still accomplishing their mission. At ESnet, we are exploring various ways to make networks work better for users. A couple of concepts could be game-changers, according to Open Network Summit conference presenter and Berkeley professor Scott Shenker: 1) using abstraction to manage network complexity, and 2) extracting and exposing simplicity out of the network. Shenker himself cites Barbara Liskov’s Turing Lecture as inspiration.

ECSEL is leveraging OSCARS and OpenFlow within the Software Defined Networking (SDN) paradigm to elegantly prevent end-to-end network traffic jams.  OpenFlow is an open standard to allow application-driven manipulation of network flows. ECSEL is using OSCARS-controlled MPLS virtual circuits with OpenFlow to dynamically stitch together a seamless data plane delivering services over multi-domain constructs.  ECSEL also provides an additional level of simplicity to the application, as it can discover host-network interconnection points as necessary, removing the requirement of applications being “statically configured” with their network end-point connections. It also enables stitching of the paths end-to-end, while allowing each administrative entity to set and enforce its own policies. ECSEL can be easily enhanced to enable users to verify end-to-end performance, and dynamically select application-specific protocol forwarding rules in each domain.

The OpenFlow capabilities, whether it be in an enterprise/campus or within the data center, were demonstrated with the help of NEC’s ProgrammableFlow Switch (PFS) and ProgrammableFlow Controller (PFC). We leveraged a special interface developed by them to program a virtual path from ingress to egress of the OpenFlow domain. ECSEL accessed this special interface programmatically when executing the end-to-end path stitching workflow.

Our anticipated next step is to develop ECSEL as an end-to-end service by making it an integral part of a scientific workflow. The ECSEL software will essentially act as an abstraction layer, where the host (or virtual machine) doesn’t need to know how it is connected to the network–the software layer does all the work for it, mapping out the optimum topologies to direct data flow and make the magic happen. To implement this, ECSEL is leveraging the modular architecture and code of the new release of OSCARS 0.6.  Developing this demonstration yielded sufficient proof that well-architected and modular software with simple APIs, like OSCARS 0.6, can speed up the development of new network services, which in turn validates the value-proposition of SDN. But we are not the only ones who think that ECSEL virtual circuits show promise as a platform for spurring further innovation. Vendors such as Brocade and Juniper, as well as other network providers attending the demo were enthusiastic about the potential of ECSEL.

But we are just getting started. We will reprise the ECSEL demo at SC11 in Seattle, this time with a GridFTP application using Remote Direct Memory Access (RDMA) which has been modified to include the XSP (eXtensible Session Protocol) that acts as a signaling mechanism enabling the application to become “network aware.”  XSP, conceived and developed by Martin Swany and Ezra Kissel of Indiana University and University of Delaware,  can directly interact with advanced network services like OSCARS – making the creation of virtual circuits transparent to the end user. In addition, once the application is network aware, it can then make more efficient use of scalable transport mechanisms like RDMA for very large data transfers over high capacity connections.

We look forward to seeing you there and exchanging ideas. Until Seattle, any questions or proposals on working together on this or other solutions to the “Big Data Problem,” don’t hesitate to contact me.

–Inder Monga

imonga@es.net

ECSEL Collaborators:

Eric Pouyoul, Vertika Singh (summer intern), Brian Tierney: ESnet

Samrat Ganguly, Munehiro Ikeda: NEC

Martin Swany, Ahmed Hassany: Indiana University

Ezra Kissel: University of Delaware

Routers, the Portal, and Transcontinental 100G Links – Oh My!

Things are moving quickly at ESnet, and we’re not talking only about data transfers.  Our Advanced Networking Initiative (ANI) 100G roll-out is gaining even more momentum and we are really gearing up for the SC11 conference next month.

This week, it was announced that we are working with LGS Innovations to deploy Alcatel-Lucent 7750 Service Routers (SR) on the new ANI prototype network. The first routers were delivered about a month ago and we are already well into acceptance testing and deployment. So far we have routers up and running in Sunnyvale, NERSC, StarLight and Argonne.  After we installed this equipment, our engineering team worked with Internet2 to light the first coast-to-coast network path from Washington, D.C. all the way to Sunnyvale in our backyard  –  the first 100G transcontinental link in the world! We were particularly proud of this milestone, which we reached well ahead of schedule. As next steps, we anticipate that 6 routers of our 10 total will be deployed in time for SC11.

To help the community keep tabs on our rapid ANI deployment progress, the ESnet Tools team has been expanding the MyESnet portal to provide a real-time view of the rollout. You can see the result of their efforts here.

This summer we introduced MyESnet to provide DOE Office of Science researchers and IT staff with a wide range of customized tools intended to greatly improve their ability to understand network issues in real time. The initial functionality includes a dashboard that provides a bird’s-eye view of the local area networks of the DOE facilities connected by ESnet, including their traffic patterns and system status.

Using this new ANI visualization tool, users can see which 100G network links are active, as well as information on each of our node and interconnect locations across the country.  The tool will be expanded to show traffic load in the near future.  In the next few weeks, the portal will also provide a real-time view of the eight community-driven projects that will use the new 100G network for large scale demonstrations at SC11. This interface will allow viewers to view the details and traffic load of these demonstrations.

With fewer than 25 days to go until SC11, our team is gearing up for some exciting announcements and ANI demos . . . check back for info on these activities soon and check in with us in Seattle. ESnet is part of Berkeley Lab booth 512.

Nobel Prize Lesson: Distinguish the Improbable from the Impossible

Saul Perlmutter was woken at 3 am yesterday by a reporter asking how he felt about winning the Nobel Prize. Any confusion was cleared up a few minutes later when the Nobel Committee in Sweden called. Perlmutter, an astrophysicist who holds a joint appointment at Berkeley Lab and UC Berkeley, and his colleagues Brian Schmidt and Adam Reiss received the 2011 Nobel Prize in Physics for their work in the 1990s in using supernovae to measure the accelerating expansion of the universe. They found, independently, improbable evidence that the universe was expanding at ever faster speeds.

In the process they uncovered other mysteries. The universe is made up of only about 5 percent visible (or baryonic) matter. The remaining 25 percent of the universe is dark matter, and 75 percent of the universe is made up of dark energy. Perlmutter described dark energy as making space more bouncy and elastic. It appears as “a negative pressure—we see it in the equations.” Nobody is certain of the relationship between dark matter and dark energy; scientists are looking for a theoretical concept that will solve them both at the same time.

In yesterday’s press conference at Berkeley Lab, Perlmutter said that the award “recognizes what it is possible to do when whole communities of science come together.” He went on to say, “when you are driven to learn things about the world, you find yourself inventing new things.”

Of course, that is what ESnet is all about—the ESnet network links scientists so they can efficiently exchange data, collaborate, and discover new things about the world. ESnet’s Bill Johnston recalls the early days of data transfer. “Saul started out doing observations at Chabot and Hamilton. He wrote the images on Tektronix cartridge tape and ferried them around in his car.  When his tape reader at LBL broke, we would help him out. In the graphics lab that Harvard Holmes and I ran, we had several Tek tape readers. That must have been [the] late 1970s.”

And then there’s Daniel Schectman, a hard-core materials scientist at the Technion-Israel’s Institute of Technology, who today was awarded the Nobel Prize in Chemistry for discovering a new state of matter called quasi-crystals, where atoms make a structured pattern that never repeats.  While on sabbatical at NIST in 1982, Schectman was working at the electron microscope, imaging a sample of aluminum and maganese. He made a measurement that seemed to be a mistake– the symmetry of its five-fold crystal structure violated the laws of physics. It is possible to get an identical pattern from structures with 3,4,6 and 8 sides, just like square floor tiles can be repeated in an identical pattern, no matter how you rotate them. But Schectman knew it was theoretically impossible to tile a space in five-fold rotational symmetry.

A former student who took his class in electron beam crystallography at the Technion recalls that Schectman showed the class the research notebook that he kept at NIST. Beside the measurement of the anomalous sample, Schectman scribbled in Hebrew, “there is no creature like that.” Schectman persisted through years of caustic opposition from his peers (Linus Pauling, a two time Nobel-winner, was particularly hostile), to verify his results—he was even asked to leave his research group. The original paper he attempted to publish in the Journal of Applied Physics in 1984 was immediately rejected, but he later published another paper with collaborators that rocked the field of crystallography.

It turns out that to tile a space and achieve five-fold symmetry uses not a single tile, but two—one distorted, a concept called Penrose tiling. But at the time, Schectman was unfamiliar with Roger Penrose’s work; it was an idea that only pure mathematicians played with–it wasn’t even applied mathematics. On an atomic level, quasi-crystals resemble aperiodic mosaics, such as those found in the medieval Islamic mosaics of the Alhambra Palace in Spain and the Darb-i Imam Shrine in Iran.

Schectman’s discovery caused a paradigm shift in chemistry, the Swedish Academy of Sciences said. But exotic quasi-crystals, since they don’t scratch easily, are now appearing in our everyday lives in the non-stick coatings on razor blades, drill-bits, and pots and pans.

Schectman and Perlmutter both stood on the shoulders of people who came before them, and relied on their communities to test and verify their results.  The pace of scope of modern day science is making international collaboration not a luxury, but a necessity. Although individual persistence and courage drive many new discoveries, they still take place in the context of a global community of scientists.  ESnet and other research and education networks make this community possible, by allowing scientists to share datasets and knowledge and to work together towards even greater discoveries.

“Science is a method, not a finished project; we don’t know where it will lead in the future,” Perlmutter summed up. And networks are forging the way.

Idea Power: Two ESnet Projects are Honored With Internet2 IDEA Awards

We are proud to announce that two of ESnet’s projects have received IDEA (Internet2 Driving Exemplary Applications) awards in Internet2’s 2011 annual competition for innovative network applications that have had the most positive impact and potential for adoption within the research and education community. (see: Internet2’s press release).

Internet2 recognized OSCARS (On-Demand Secure Circuits and Advance Reservation System), developed by the ESnet team led by Chin Guok, including Evangelos Chaniotakis, Andrew Lake, Eric Pouyoul and Mary Thompson. Contributing partners also included Internet2, USC ISI and DANTE.

ESnet’s MAVEN (Monitoring and Visualization of Energy consumed by Networks) proof of concept application was also recognized with an IDEA award in the student category. MAVEN was prototyped by Baris Aksanli during his summer internship at ESnet. Baris is a Ph.D student at the University of California, San Diego conducting research at the System Energy Efficiency Lab with his thesis advisor, Dr. Tajana Rosing. Baris worked closely with his summer advisor, Inder Monga, and Jon Dugan to implement MAVEN as part of ESnet’s new Green Networking Initiative.

The idea behind OSCARS

OSCARS enables researchers to automatically schedule and guarantee end-to-end delivery of scientific data across networks and continents. For scientists, being able to count on reliable data delivery is critical as scientific collaborations become more expansive, often global. Meanwhile, in disciplines ranging from high-energy physics to climate, scientists are using powerful, geographically dispersed instruments like the Large Hadron Collider that are producing increasingly massive bursts of data, challenging the capabilities of traditional IP networks.

OSCARS virtual circuits can reliably schedule time-sensitive data flows – like those from the LHC – round the clock across networks, enabling research and education networks to seamlessly meet user needs. OSCARS code is also being deployed by R&E networks worldwide to support an ever-growing user base of researchers with data-intensive collaboration needs. Internet2, U.S. LHCnet, NORDUNet, RNP in Brazil as well as over 10 other regional and national networks have currently implemented OSCARS for virtual circuit services. Moreover, Internet2’s NSF-funded DyGIR and DYNES projects will in 2012 deploy over 60 more instances of OSCARS at university campuses and regional networks to support scientists involved in LHC, Laser Interferometer Gravitational-Wave Observatory (LIGO), Large Synoptic Survey Telescope (LSST) and Electronic Very-Long Baseline Interferometry (eVLBI) programs.

We are proud of the hard work and dedication the OSCARS development team has demonstrated since the start of this project. Just as importantly we are proud to see this work paying off in with new science collaboration and discoveries.

The potential of MAVEN

The Monitoring and Visualization of Energy consumed by Networks (MAVEN) project is a brand new prototype portal that will help network operators and researchers better track live network energy consumption and environmental conditions. MAVEN – implemented by Baris during his summer internship – is a first major step for ESnet in instrumenting our network with the tools to understand these operational dynamics. As networks continue to get bigger and faster, they will require more power and cooling in an era of decreased energy resources. To address this pressing challenge, ESnet is leading a new generation of research aimed at understanding how networks can operate in a more energy-efficient manner. We are grateful for Baris’ significant contributions in leading the development of MAVEN and glad to see that his talent is being recognized by the R&E networking community through this award.

Baris is now back in school at UCSD, completing his Ph.D in computer science. http://cseweb.ucsd.edu/~baksanli/. Congratulations Baris!