Science Data Weathers the Storm

Hurricane Sandy took a terrible toll this week, and our thoughts remain with the millions of people whose lives were impacted.

From the nightly news, we learned about critical systems that temporarily failed during the extreme weather, and others that continued to function.  I’d like to share an example in the second category.

Here’s a screenshot from our public web portal, http://my.es.net, showing traffic to and from Brookhaven National Laboratory on Long Island, during the worst of the storm:

Network traffic to and from Long Island’s Brookhaven National Laboratory was not impacted by the devastating storm that struck the New York area this week.

Why does it matter that Brookhaven’s connection to ESnet, and to the broader Internet, remained functional during the disaster? It matters because modern science has entered an age of extreme-scale data. In more and more fields, scientific discovery depends on data mobility, as huge data sets from experiments or simulations move to computational and analysis facilities around the globe. Large-scale science instruments are now designed around the premise that high-speed research networks exist. In this kind of architecture, loss of network connectivity impairs scientific productivity. In the worst case, unique and vital data can be lost.

Much of the network traffic in these graphs can be attributed to ATLAS, one of the extraordinary experiments at CERN’s Large Hadron Collider, outside of Geneva. It’s been a great year at CERN, with announcement of a Higgs-like particle this summer. At ESnet, we were very happy that productivity of the ATLAS experiment was not affected by the massive storm.

Although the research networking community may have benefited from good fortune during storm, it’s important to recognize that a lot of careful planning and sound engineering – conducted over the course of many years – contributed to this outcome. Research networks like ESnet and Internet2, regional networks like NYSERNet, campus networking groups at Brookhaven and elsewhere, and exchange points like MAN LAN all work very hard to harden themselves against individual points of failure. As with all engineering, the devil is in the details: fiber diversity, elimination of shared fate in single conduits and data centers, location and specification of generators, availability of fuel, optical protection schemes, failover for dynamic circuits… the list goes on and on.

We won’t always be so fortunate, but the screenshot above is something our research networking community can take pride in. My thanks to the hundreds of people whose sound decisions made this good outcome possible.

ECSEL leverages OpenFlow to demonstrate new network directions

ESnet and its collaborators successfully completed three days of demonstrating its End-to-End Circuit Service at Layer 2 (ECSEL) software at the Open Networking Summit held at Stanford a couple of weeks ago. Our goal is to build “zero-configuration circuits” to help science applications seamlessly use networks for optimized end-to-end data transport. ECSEL, developed in collaboration with NEC, Indiana University, and the University of Delaware builds on some exciting new conceptual thinking in networking.

Wrangling Big Data 

To put ECSEL in context, the proliferating tide of scientific data flows – anticipated at 2 petabytes per second as planned large-scale experiments get in motion – is already challenging networks to be exponentially more efficient. Wide area networks have vastly increased bandwidth and enable flexible, distributed, scientific workflows that involve connecting multiple scientific labs to a supercomputing site, a university campus, or even a cloud data center.

Heavy network traffic to come

The increasing adoption of distributed, service-oriented computing means that resource and vendor independence for service delivery is a key priority for users. Users expect seamless end-to-end performance and want the ability to send data flows on demand, no matter how many domains and service providers are involved.  The hitch is that even though the Wide Area Network (WAN) can have turbocharged bandwidth, at these exponentially increasing rates of network traffic even a small blockage in the network can seriously impair the flow of data, trapping users in a situation resembling commute conditions on sluggish California freeways. These scientific data transport challenges that we and other R&E networks face are just a taste of what the commercial world will encounter with the increasing popularity of cloud computing and service-driven cloud storage.

Abstracting a solution

One of the key feedback from application developers, scientists and end-users is that they do not want to deal with the complexity at the infrastructure level while still accomplishing their mission. At ESnet, we are exploring various ways to make networks work better for users. A couple of concepts could be game-changers, according to Open Network Summit conference presenter and Berkeley professor Scott Shenker: 1) using abstraction to manage network complexity, and 2) extracting and exposing simplicity out of the network. Shenker himself cites Barbara Liskov’s Turing Lecture as inspiration.

ECSEL is leveraging OSCARS and OpenFlow within the Software Defined Networking (SDN) paradigm to elegantly prevent end-to-end network traffic jams.  OpenFlow is an open standard to allow application-driven manipulation of network flows. ECSEL is using OSCARS-controlled MPLS virtual circuits with OpenFlow to dynamically stitch together a seamless data plane delivering services over multi-domain constructs.  ECSEL also provides an additional level of simplicity to the application, as it can discover host-network interconnection points as necessary, removing the requirement of applications being “statically configured” with their network end-point connections. It also enables stitching of the paths end-to-end, while allowing each administrative entity to set and enforce its own policies. ECSEL can be easily enhanced to enable users to verify end-to-end performance, and dynamically select application-specific protocol forwarding rules in each domain.

The OpenFlow capabilities, whether it be in an enterprise/campus or within the data center, were demonstrated with the help of NEC’s ProgrammableFlow Switch (PFS) and ProgrammableFlow Controller (PFC). We leveraged a special interface developed by them to program a virtual path from ingress to egress of the OpenFlow domain. ECSEL accessed this special interface programmatically when executing the end-to-end path stitching workflow.

Our anticipated next step is to develop ECSEL as an end-to-end service by making it an integral part of a scientific workflow. The ECSEL software will essentially act as an abstraction layer, where the host (or virtual machine) doesn’t need to know how it is connected to the network–the software layer does all the work for it, mapping out the optimum topologies to direct data flow and make the magic happen. To implement this, ECSEL is leveraging the modular architecture and code of the new release of OSCARS 0.6.  Developing this demonstration yielded sufficient proof that well-architected and modular software with simple APIs, like OSCARS 0.6, can speed up the development of new network services, which in turn validates the value-proposition of SDN. But we are not the only ones who think that ECSEL virtual circuits show promise as a platform for spurring further innovation. Vendors such as Brocade and Juniper, as well as other network providers attending the demo were enthusiastic about the potential of ECSEL.

But we are just getting started. We will reprise the ECSEL demo at SC11 in Seattle, this time with a GridFTP application using Remote Direct Memory Access (RDMA) which has been modified to include the XSP (eXtensible Session Protocol) that acts as a signaling mechanism enabling the application to become “network aware.”  XSP, conceived and developed by Martin Swany and Ezra Kissel of Indiana University and University of Delaware,  can directly interact with advanced network services like OSCARS – making the creation of virtual circuits transparent to the end user. In addition, once the application is network aware, it can then make more efficient use of scalable transport mechanisms like RDMA for very large data transfers over high capacity connections.

We look forward to seeing you there and exchanging ideas. Until Seattle, any questions or proposals on working together on this or other solutions to the “Big Data Problem,” don’t hesitate to contact me.

–Inder Monga

imonga@es.net

ECSEL Collaborators:

Eric Pouyoul, Vertika Singh (summer intern), Brian Tierney: ESnet

Samrat Ganguly, Munehiro Ikeda: NEC

Martin Swany, Ahmed Hassany: Indiana University

Ezra Kissel: University of Delaware

Nobel Prize Lesson: Distinguish the Improbable from the Impossible

Saul Perlmutter was woken at 3 am yesterday by a reporter asking how he felt about winning the Nobel Prize. Any confusion was cleared up a few minutes later when the Nobel Committee in Sweden called. Perlmutter, an astrophysicist who holds a joint appointment at Berkeley Lab and UC Berkeley, and his colleagues Brian Schmidt and Adam Reiss received the 2011 Nobel Prize in Physics for their work in the 1990s in using supernovae to measure the accelerating expansion of the universe. They found, independently, improbable evidence that the universe was expanding at ever faster speeds.

In the process they uncovered other mysteries. The universe is made up of only about 5 percent visible (or baryonic) matter. The remaining 25 percent of the universe is dark matter, and 75 percent of the universe is made up of dark energy. Perlmutter described dark energy as making space more bouncy and elastic. It appears as “a negative pressure—we see it in the equations.” Nobody is certain of the relationship between dark matter and dark energy; scientists are looking for a theoretical concept that will solve them both at the same time.

In yesterday’s press conference at Berkeley Lab, Perlmutter said that the award “recognizes what it is possible to do when whole communities of science come together.” He went on to say, “when you are driven to learn things about the world, you find yourself inventing new things.”

Of course, that is what ESnet is all about—the ESnet network links scientists so they can efficiently exchange data, collaborate, and discover new things about the world. ESnet’s Bill Johnston recalls the early days of data transfer. “Saul started out doing observations at Chabot and Hamilton. He wrote the images on Tektronix cartridge tape and ferried them around in his car.  When his tape reader at LBL broke, we would help him out. In the graphics lab that Harvard Holmes and I ran, we had several Tek tape readers. That must have been [the] late 1970s.”

And then there’s Daniel Schectman, a hard-core materials scientist at the Technion-Israel’s Institute of Technology, who today was awarded the Nobel Prize in Chemistry for discovering a new state of matter called quasi-crystals, where atoms make a structured pattern that never repeats.  While on sabbatical at NIST in 1982, Schectman was working at the electron microscope, imaging a sample of aluminum and maganese. He made a measurement that seemed to be a mistake– the symmetry of its five-fold crystal structure violated the laws of physics. It is possible to get an identical pattern from structures with 3,4,6 and 8 sides, just like square floor tiles can be repeated in an identical pattern, no matter how you rotate them. But Schectman knew it was theoretically impossible to tile a space in five-fold rotational symmetry.

A former student who took his class in electron beam crystallography at the Technion recalls that Schectman showed the class the research notebook that he kept at NIST. Beside the measurement of the anomalous sample, Schectman scribbled in Hebrew, “there is no creature like that.” Schectman persisted through years of caustic opposition from his peers (Linus Pauling, a two time Nobel-winner, was particularly hostile), to verify his results—he was even asked to leave his research group. The original paper he attempted to publish in the Journal of Applied Physics in 1984 was immediately rejected, but he later published another paper with collaborators that rocked the field of crystallography.

It turns out that to tile a space and achieve five-fold symmetry uses not a single tile, but two—one distorted, a concept called Penrose tiling. But at the time, Schectman was unfamiliar with Roger Penrose’s work; it was an idea that only pure mathematicians played with–it wasn’t even applied mathematics. On an atomic level, quasi-crystals resemble aperiodic mosaics, such as those found in the medieval Islamic mosaics of the Alhambra Palace in Spain and the Darb-i Imam Shrine in Iran.

Schectman’s discovery caused a paradigm shift in chemistry, the Swedish Academy of Sciences said. But exotic quasi-crystals, since they don’t scratch easily, are now appearing in our everyday lives in the non-stick coatings on razor blades, drill-bits, and pots and pans.

Schectman and Perlmutter both stood on the shoulders of people who came before them, and relied on their communities to test and verify their results.  The pace of scope of modern day science is making international collaboration not a luxury, but a necessity. Although individual persistence and courage drive many new discoveries, they still take place in the context of a global community of scientists.  ESnet and other research and education networks make this community possible, by allowing scientists to share datasets and knowledge and to work together towards even greater discoveries.

“Science is a method, not a finished project; we don’t know where it will lead in the future,” Perlmutter summed up. And networks are forging the way.

ESnet is going directly to the “light” source

Over the past few years, ESnet staff has worked to engage researchers across a variety of disciplines to ensure our services meet their needs. As a part of this ongoing effort, I will be leading a session titled “The Energy Sciences Network: Moving Data, Advancing Science” on October 4^th at the Advanced Light Source User Meeting being held at Berkeley Lab. I hope to involve ALS users in a discussion around ways to use the ESnet network to more efficiently exchange and transfer their scientific data. The talk will also review some common problems encountered in the transfer of large scientific data sets, as well as highlight solutions showcasing how research groups have worked with ESnet to effectively use the network to expedite their data sharing and analysis.

ALS has a number of different experiments that can benefit from ESnet’s capabilities in different ways. Computing-intensive disciplines such as protein crystallography, X-ray microdiffraction, and other fields that utilize the ALS are experiencing unprecedented growth in data production. This is straining traditional methods of data distribution and analysis. As a matter of fact, within the next five years, we expect that many ALS user groups will experience an increase in data production of up to a factor of 1000, which will far exceed traditional digital media distribution capabilities. Without better ways of distributing data, scientific productivity could be significantly impeded.

In recent months, ESnet has developed new tools and software innovations as part of our goal of making our network and services more accessible to our users.  If you would like to get tips on how to use ESnet focused on your particular scientific discipline, or just have general questions, please contact us at info@es.net.

–Eli Dart

Transfer lots of data, effortlessly. Here’s how. Tune in to webcast Sept 8th.

Got a lot of data to move around? Say, more than 100 Gigabytes? ESnet recommends that you attend this webcast on how to use Globus Online, one of ESnet’s recommended file transfer services.  The webcast (https://www.globusonline.org/esnet-webcast-09-08-11/) will show you how to move data as you need it without having to become an IT expert, learn a new command vocabulary or install software.

ESnet provides the high-bandwidth, reliable connections that link scientists at national laboratories to universities and other research institutions so they can more effectively collaborate. We provide the infrastructure, but after listening to our customers, we are taking the next step and recommending the tools to make our network easier to use. As part of this effort, we are introducing our users to services like Globus Online that will help you to move data faster and more reliably.

Globus Online (https://www.globusonline.org/) is a fast, reliable file transfer service that simplifies the process of secure data movement. This free service automates the activity of managing file transfers, whether between computing facilities or from a facility to your local machine. Users can fire-and-forget their request and Globus Online will manage the entire operation – monitoring performance, retrying failed transfers, recovering from faults automatically whenever possible, and reporting status. Globus Online makes it a trivial thing to move big data around, to whatever location needed, without spending lots of time figuring out the right commands or dealing with complicated systems. Simply sign up, specify your endpoints (where the file is now (source) and where you are moving it (destination) authenticating as necessary on the servers, and then click to transfer. Once the endpoint you need is configured into Globus Online, it really is that simple – try it out for yourself!  (https://www.globusonline.org/SignUp)

In this session, the Globus Online project lead Steve Tuecke will cover all the basics of usage, including command line interface and using Globus Connect to make your own server or laptop an endpoint. I will be there to provide the ESnet perspective. We will answer your questions and also provide any constructive feedback. To get the most out of the session, sign up and try the service beforehand so you know what you are looking for and are ready to ask the right questions.

For more information, see https://www.globusonline.org/esnet-webcast-09-08-11/ or contact info@globusonline.org.

Brian Tierney, ESnet

100 Gbps ANI Prototype Network is Just the Beginning

After much work and planning, we proudly introduce you to our new scientific network-to-be.  Berkeley Lab just announced the signing of an agreement to begin construction of a 100 gigabits per second (Gbps) prototype network, part of the Lab’s Advanced Networking Initiative funded by the American Recovery and Reinvestment Act (ARRA).  Under the terms of the deal, Internet2 and its industry partners will construct the network under ESnet’s direction.

Coming Soon: ANI 100 Gpbs Prototype Network, First Step Towards Nationwide 100 Gbps Network

While our ANI network testbed is already open for business (mark your calendar –the next call for research proposals is October 1^st) in a matter of months researchers can conduct experiments on the 100 Gbps prototype network. This network will link the three supercomputing facilities at national laboratories – the National Energy Research Scientific Computing Center (NERSC), Oak Ridge Leadership Computing Facility (OLCF), and Argonne Leadership Computing Facility (ALCF) to MANLAN – the Manhattan Landing international exchange point.

But more is yet to come; the ANI prototype network will also be framework for ESnet to obtain concrete network energy use data for research into “green networking” – a priority here at ESnet.  Also part of the agreement, Berkeley Lab negotiated access to a nation-wide pair of dark fiber for 20 years which will be immediately available to network researchers and industry to experiment with new protocols and disruptive technologies.  Stay tuned to our blog to follow construction of the network and the exciting happenings at ESnet.

The 100 Gbps prototype network is just the beginning.  Berkeley Lab and ESnet will leverage the experience gained deploying the prototype network to extend these capabilities to the national labs and facilities ESnet currently serves, and connect DOE scientists to university collaborators around the world.  When moved to production status, the new network will increase the information carrying capacity of ESnet’s current 10 Gbps network by several orders of magnitude.  One terabyte of data, which takes approximately 13 minutes to transfer on ESnet’s present 10 Gbps network, will be delivered in under a minute on the 100 Gbps network.

What we hope you will get out of this

While our network currently meets the capacity needs of its users, we see new challenges on the horizon.  On behalf of the DOE Office of Science, we regularly survey scientists about their projected needs in order to better target services for our users. Demand on ESnet’s network has grown by a factor of 10 every 47 months since 1990.  Over the last year, we have seen a rise in ESnet network traffic by 70 percent – most of that data traffic coming from the Large Hadron Collider at CERN but with genomics and climate data also picking up steam.

Just as investments in national infrastructure built the Interstate Highway System that sped the delivery of goods and opened up commerce in the United States, high-speed networks will speed the scientific discovery that will drive this nation’s economy in the future.  New, large-scale instruments are in the offing, and we anticipate a growing tide of data in various disciplines as scientists work to understand our climate, develop clean fuels, and investigate the basic nature of matter.  High-speed networks like this one will have a huge impact on an increasing number of disciplines, as more scientists collaborate in global teams and depend on remote supercomputers to model and solve complex problems.

How to be really thrifty with $62 million.

This is a challenging time when budgets are under severe pressure, both for government and the research and education community.  We believe that by pooling resources and expertise we can help curb the costs of scientific research.  By working closely with Internet2 to build this prototype network, we are leveraging our ARRA award and their funding for synergistic effect and getting more return on investment for the taxpayer.

We strongly believe this new 100Gbps network infrastructure will allow us to better support data-intensive science by providing more capacity at lower cost, lower energy consumption, and lower carbon emissions per bit.  And that makes us feel good about what we do.

I.T. in-depth at DUSEL

This guest blog is contributed by Warren Matthews, Cyber-Infrastructure Chief Engineer at the Deep Underground Science and Engineering Lab (DUSEL).

 

Guest Blogger: Warren Matthews, DUSEL

The Deep Underground Science and Engineering Laboratory (DUSEL) is a research lab being constructed in the former Homestake gold mine in Lead, South Dakota, now resurrected to mine data about the earth, new life forms, and the universe itself. When finished, DUSEL will explore fundamental questions in particle physics, nuclear physics and astrophysics. Biologists will study life in extreme environments. Geologists will study the structure of the earth’s crust. Early science programs have already begun to explore some of these questions. In addition, DUSEL education programs are underway to inspire students to pursue careers in science, technology, engineering, and mathematics. This interdisciplinary collaboration of scientists and engineers is led by the University of California at Berkeley and the South Dakota School of Mines and Technology.

 

I am the cyberinfrastructure chief engineer for DUSEL. As such, my concern is the research environment and advanced services that will be needed to accomplish our scientific goals. To enable future discoveries, scientists will need to capture, analyze, and exchange their data. We will have to deploy and perhaps even develop new technologies to provide the scientists with the technical and logistical support for their research. We expect that the unique research opportunities and instrumentation that will be established at DUSEL will draw scientific teams from all over the world to South Dakota, so high-speed national and International network connectivity will also critical.

National laboratories have made many important contributions in the development of IT and networking technology.  I’m very pleased that DUSEL is the newest member of the ESnet community and I have no doubt that we’ll be leveraging their expertise.  In conversations with numerous colleagues at other labs it has become apparent that although DUSEL is starting with a clean slate and there are no legacy systems to support, we still have common issues and some difficult decisions to consider. All the labs have the challenges of meeting the needs of both large and small scientific collaborations. We all feel the budget crunch and are streamlining our support infrastructure. We are all wondering how we can optimize our use of the Cloud.

Delving into underground research

At DUSEL we have our own particular challenges, starting with an extreme underground environment. On the surface, the Black Hills of South Dakota may be freezing, but the further you go down in the mine, the hotter it gets. Rock temperatures at the 4850′ level, where the mid-level campus is under construction, are around 70F (21〫C) and humidity is around 88%. At the 7400′ level, where the deep-level campus is planned, temperatures hover around 120F (50〫C). The high levels of temperature and humidity have a significant impact on computer equipment.  We’ll figure out our challenges as we go, depending on shared expertise. After all, national labs were created to focus effort and move forward knowledge where no one university could marshal the resources required. Our goal is to provide a platform where science, technology, and innovation are able to flourish.

We anticipate technology partnerships with the many experiments are going underground at DUSEL. Currently we are expanding IPv6 and deploying perfSONAR. We are leveraging HD video conferencing. We are worrying about identity management and cyber security. We are establishing the requirements for dynamic network provisioning.  And at the same time we’re wondering what other technologies will emerge in the next 20 or 30 years and what will be required to dig for new discoveries. You can keep track of our progress here at the Sanford Laboratory Youtube Channel.

–Warren Matthews

Harnessing bugs for new drugs and potential jet fuel

D. radiodurans is impervious to ionizing radiation; can clean up mercury and toluene contamination

The high speed networking that ESnet provides supports an incredibly varied range of scientific projects. The express purpose of DOE national labs is to conduct research in the national interest. Much is basic research, exploring fundamental issue in physics, energy, cosmology, and climate science. However the study of microbial evolution is one intriguing area that is already changing our lives with everything from new medicines to potential fuels.

Microbes are single-cell organisms that live in colonies and can be found in nearly every corner of our planet, in places ranging from insects’ intestines to some of the most toxic chemical environments. The site for the most detailed information on the genetic makeup of these organisms only lives in one place – at the DOE’s Joint Genome Institute – and is accessed via ESnet.  The Integrated Microbial Genomes (IMG) Data Management System provides the genetic makeup of thousands of microbes and tools for analyzing the functional capability of microbial communities based on their metagenome DNA sequence. Understanding these tiny organisms can provide new insights into a wide range of important problems, but in order to study the microbes, scientists need reliable access to the genomic data.

Microbes such as bacteria are responsible for a number of diseases, such as plague, tuberculosis and anthrax, while microbes known as protozoa cause diseases such as malaria, sleeping sickness and toxoplasmosis. On the plus side, microbes also live helpfully in human digestive systems, helping to digest carbohydrates and synthesize certain vitamins.

Where the wild things are

Famously, taq polymerase, an enzyme isolated from the bacterium Thermus aquaticus found in a hot spring at Yellowstone National Park in 1965, became the basis for PCR, the technique for amplifying short strands of DNA that has revolutionized biologic and genetic research. While some scientists examine exotic microbes like Deinococcus radiodurans, the most radiation-resistant organism known (and the darling of exobiologists, found as a contaminant of irradiated canned meat in 1956) for clues to how life began on earth and could evolve on other planets, your laundry detergent probably contains enzymes developed from bacteria that evolved in hot, alkaline conditions.

In the private sector, microbes are genetically modified to develop innovative drugs as well as industrial products. In an example close to home, a few years back, LBNL’s Joint BioEnergy Institute director Jay Keasling used microbial evolution techniques to synthesize artemisinic acid, a precursor to artemisinin, an anti-malarial compound. Artemisinin is derived from Artemisia annua or wormwood, a plant known to Chinese medicine for centuries. Keasling made a steady supply that could be manufactured extremely cheaply and at large volumes, so accessible to people in developing countries.

Keasling’s next project is to use microbes as potential energy sources by turning them into factories to produce sugars. Certain microbes living in termite guts are essential for digesting the wood fibers eaten by the insect. While this can be bad news for homeowners, the chemical capabilities of the “bugs” in these bugs are being studied as for their potential in converting wood and other plant matter into new energy sources. Successfully producing fuel using plant waste instead of food crops like corn, could improve our country’s future energy options.  To be sure, there still are prosaic bars to overcome, such as chemical separation and processing in volumes high enough to meet our insatiable demand for transport fuels. Other scientists at DOE national labs such as NREL and a host of private companies, some right across San Francisco Bay from our headquarters at ESnet, are investigating how to make gasoline and jet fuel from microorganisms. Genetic analysis is the essential first step in growing hardworking bacteria with the desired qualities.

Another promising microbial application is environmental cleanup. One form of bacteria found almost 2 miles underground in a South African gold mine lives in total darkness, 140 degrees Fahrenheit – and no oxygen. The organism gets its energy not from the sun but from hydrogen and sulfate produced by the radioactive decay of uranium. By understanding how life can thrive in such an apparently toxic setting, scientists may get new insight into using microbes to clean up environmental contamination.

Currently, the IMG database contains complete genomes for 4,879 microbes, with another 1,569 in draft form. Of the total, 1,107 are bacteria and 2,536 are viruses. The information, containing data for more than 20 million genes, is provided freely to interested researchers.

ESnet gives CISCO Nerd Lunch talk, learns televangelism is harder than it seems

As science transitions from lab-oriented to a distributed computational and data-intensive activity, the research and education (R&E) networking community is tracking the growing data needs of scientists. Huge instruments like the Large Hadron Collider are being planned and built. These projects require global-scale collaborations and contributions from thousands of scientists, and as the data deluge from the instruments grows, even more scientists are interested in analyzing it for the next breakthrough discovery. Suffice it to say that even though worldwide video consumption on the Internet is driving a similar increase in commercial bandwidth, the scale, characteristics, and requirements of scientific data traffic is quite different.

And this is why ESnet got invited to Cisco Systems’ headquarters last week to talk about how we how we handle data as part of their regular Nerd Lunch talk series. What I found interesting although not surprising, was that with Cisco being a big evangelist of telepresence, more employees attended the talk from their desks than in person.  This was a first for me and I came away with a new appreciation for the challenges of collaborating across distances.

From a speaker’s perspective, the lesson learnt by me was to brush up my acting skills. My usual preparations are to rehearse the difficult transitions and  focus on remembering the few important points to make on every slide. When presenting, that slide presentation portion of my brain goes on auto-pilot, while my focus turns towards evaluating the impact on the audience. When speaking at a podium one can observe when someone in the audience opens a notebook to jot down a thought, when their attention drifts to email on the laptop in front of them, or when a puzzled look appears on the face of someone as they try to figure out the impact of the point I’m trying to make. But these visual cues go missing with a largely webcast audience, making it harder to know when to stop driving home a point or when to explain the point further to the audience.  In the future, I’ll have to be better at keeping the talk interesting without the usual clues from my audience.

Maybe the next innovation in virtual-reality telepresence is just waiting to happen?

Notwithstanding the challenges of presenting to a remote audience, enabling remote collaboration is extremely important to ESnet. Audio, video and web collaboration is a key service offered by us to the DOE labs. ESnet employees use video extensively in our day-to-day operations. The “ESnet watercooler”, a 24×7 open video bridge, is used internally by our distributed workforce to discuss technical issues, as well as, to have ad-hoc meetings on topics of interest. As science goes increasingly global, scientists are also using this important ESnet service for their collaborations.

With my brief stint in front of a stage now over, it is back to ESnet and then on to the 100G invited panel/talk at IEEE ANTS conference in Mumbai. Wishing all of you a very Happy New Year!

Inder Monga

Why this spiking network traffic?

ESnet November 2010 Traffic

Last month was the first in which the ESnet network crossed a major threshold – over 10 petabytes of traffic! Traffic volume was 40% higher than the prior month and 10 times higher than just a little over 4 years ago. But what’s behind this dramatic increase in network utilization?  Could it be the extreme loads ESnet circuits carried for SC10, we wondered?

Breaking down the ESnet traffic highlighted a few things.  Turns out it wasn’t all that demonstration traffic sent across thousands of miles to the Supercomputing Conference in New Orleans (151.99 TB delivered), since that accounted for only slightly more than 1% of November’s ESnet-borne traffic.  We observed for the first time significant volumes of genomics data traversing the network as the Joint Genome Institute sent over 1 petabyte of data to NERSC. JGI alone accounted for about 10% of last month’s traffic volume. And as we’ve seen since it went live in March, the Large Hadron Collider continues to churn out massive datasets as it increases its luminosity, which ESnet delivers to researchers across the US.

Summary of Total ESnet Traffic, Nov. 2010

Total Bytes Delivered: 10.748 PB
Total Bytes OSCARS Delivered: 5.870 PB
Pecentage of OSCARS Delivered: 54.72%

What is is really going on is quite prosaic, but to us, exciting. We can follow the progress of distributed scientific projects such as the LHC  by tracking the proliferation of our network traffic, as the month-to-month traffic volume on ESnet correlates to the day-to-day conduct of science. Currently, Fermi and Brookhaven LHC data continue to dominate the volume of network traffic, but as we see, production and sharing of large data sets by the genomics community is picking up steam. What the stats are predicting: as science continues to become more data-intensive, the role of the network will become ever more important.