New article in @Patterns_CP covers:
A) patterns for linking big data instruments & computers;
B) capturing such “flows” in reusable forms;
C) Globus automation services to run flows;
D) examples from synchrotron light sources;
E) implications for scientists & facilities, 1/8

A) Many instrument data capture and analysis applications engage a modest number of stereotypical patterns or "flows": e.g., acquire & quality control  data; reconstruct data, train ML model on data, catalog & store data, deploy trained ML model, 2/8


B) We show how such flows can be represented in reusable forms, as a sequence of actions but with details like analyses and target instruments, computers, and repositories unspecified. We introduce the Gladier Python toolkit for that purpose, 3/8

C) We describe how flow orchestration, and mapping to specific computers, repositories, etc., can be handled by Globus Automation services, which extend the popular @globus platform to encompass flow creation, orchestration, and related tasks, https://www.globus.org/instruments 4/8

D) We present five synchrotron light source examples: x-ray photon correlation spectroscopy, serial synchrotron crystallography, 2x high-energy diffraction microscopy, ptychography. Each includes simplified code that you can run yourself, 5/8

E) We use those same five examples to illustrate experiences at  experimental facilities (@advancedphoton, #LCLS) & compute facility (@argonne_lcf), examining trends in usage, compute volume, data volumes, & performance, & discuss implications for scientists and facilities, 6/8

Article is "Linking Scientific Instruments and Computation: Patterns, Technologies, Experiences", Patterns 3(10):100606, 2022, https://doi.org/10.1016/j.patter.2022.100606. Co-authors on Twitter: @BenBlaiszik @chard_kyle @lzhengchun @mpapka. Give it a try and let us know how you do! 7/8

Also see Supplementary Information with pointers to code at https://www.cell.com/cms/10.1016/j.patter.2022.100606/attachment/138fdc7e-ae62-4734-a802-4b6763337386/mmc1.pdf. Thanks to @NSF CISE OAC, @doescience ASCR, @doescience BES, @NIST, @argonne_lcf, @argonne, @UChicago, @advancedphoton 8/8

Tagging @BenBlaiszik @Sergei_Imaging
@helsoeste @parasharmanish @A_Aspuru_Guzik @leecronin @aicooper @RafaelSilva_com @MaximZiatdinov @arvindr_
@doescience @margmartonosi @Cyclogenesis_au @NIHDataScience @rabernat @ilkayaltintas @indermo @ESnet
@KPatBerkeley

You can read the unrolled version of this thread here: https://typefully.com/ianfoster/Mo4VbGh

Linking scientific instruments and computation: Patterns, technologies, and experiences

The word "exciting" can be overused, but the new UChicago "AI+Science" postdoc program is truly exciting. 20 positions, well paid, with great freedom to work with whom, and on what, you want. First cohort of 10 starts Jan 2023, second Sept 2023.

The mission: Train scholars who seek to advance and accelerate the adoption of AI in the natural sciences. Work with scientists @UChicago @argonne @Fermilab @MBLScience. Work with @LabsGlobus, in particular! Great science, great colleagues, great data. https://aiscience.uchicago.edu

New @TheHDSR paper on methods & experiences for realizing "continuous and ubiquitous FAIRness" by using simple tools to track all data all the time: Lightweight IDs to name things; BDBag/BagIt/Research Object to create aggregations; flow automation https://hdsr.mitpress.mit.edu/pub/qjpg8oik/release/2 

We refer to fascinating #Zebrafish memory study in which tools are used to avoid #FAIR becoming "data sharing theater." We also relate to historical practices and future research training requirements. 

The @NSF CC-* program connects institutions to the world at high speeds via the @globus-based @ESnet #ScienceDMZ architecture. To ease adoption, the @UChicago Globus team offers free subscriptions to new CC* awardees. Contact @globus for details
https://www.globus.org/blog/globus-support-2022-nsf-cc-program 1/3

As background: The Modern Research Data Portal (#MRDP) site illustrates how the not-for-profit, subscriber-supported @globus allows for effective use of CC-* supported science infrastructure, providing sample code and application examples.
https://docs.globus.org/modern-research-data-portal/ 2/3

Under the covers: The @ESnet #ScienceDMZ architecture with its Data Transfer Nodes https://fasterdata.es.net/DTN/ allows  transfers to proceed directly (under @globus control) to high-speed storage, without traversing firewalls. The result: multi-GB/s transfers and faster science 3/3

1/4 On February 12, 2022 we started replicating 7.5 PB #CMIP climate data from @Livermore_Lab to @argonne_lcf and @OLCFGOV. We used @globus to automate this transfer of 28,907,532 files: mostly LLNL->ALCF->OLCF, but also LLNL->OLCF->ALCF during ALCF maintenance. @doescience @NSF

2/4 The transfer completed on May 3, just in time for #GlobusWorld where @climate_dude presented on #ESGF2 futures. The transfer log shows how @globus automation navigated maintenance periods and GPFS failures.   

3/4 Already by March 29, my colleague @Cyclogenesis_au was using @ProjectJupyter notebooks to look at @NCAR_Science
#CESM ScenarioMIPS simulation results over #Chicago https://twitter.com/Cyclogenesis_au/status/1508862593913393154 

4/4 This is just a first step towards realizing the power of @DOEscience #LCF supercomputing coupled with massive #CMIP simulation data. New questions, new methods (e.g., AI methods applied at scale), linking of simulation and observational data. So many possibilities! 

I posted "The History of the Grid" on Arxiv. This 2011 article reviews the work that, from the 1990s onwards, pioneered on-demand and federated computing and data over then-new high-speed networks. https://arxiv.org/abs/2204.04312 

2/ The concept of computing utilities dates to the 1960s, but little could be done with dial-up lines. Yet as Gilder noted, "when the network is as fast as the computer’s internal links, the machine disintegrates across the net into a set of special-purpose appliances. 

3/ Fast networks spurred innovative "Grid" technologies, applications, and infrastructures. Access protocols, security mechanisms, virtual machines, distributed schedulers. Systems linking 100,000s nodes. Global data federations. Virtual reality over networks. (Bad map: no NZ!)

4/4 This article reviewed the origins, development, and impact of Grid work. Today, cloud has made utility computing a commonplace. New applications and new concerns (e.g., data sovereignty) drive new interest in federation. Perhaps grid protocols are needed again. 

The History of the Grid on Arxiv

Inaugural AI+Science Summer School this August in Chicago,
co-hosted by @DSI_UChicago and @IMSI_Institute. Apply by 4/25. A week of summer in Chicago, great lecturers, smart  colleagues--what's not to like? https://datascience.uchicago.edu/events/2022-ai-science-summer-school/ 

2/ Topics include:
• Physics-informed and -constrained neural networks
• AI for geosciences and climate modeling
• AI-guided search in chemistry and molecular biology
• Tensor networks for classical and quantum systems
• Active learning and AI-guided experimental design

3/ Lecturers include @risi_kondor, @solislemuslab, Roman Garnett, @stochastician, Romit Maulik, @MStoudenmire, and @SJRiesenfeld.  

Congratulations to Jack Dongarra for well-deserved Turing Award. His use of mathematics, in particular linear algebra, as a language for programming increasingly complex computer architectures has lasted 30+ years and enabled scaling by billions.   https://fortune.com/2022/03/30/turing-award-dongarra-supercomputer-expert-winner-1-million-nobel-prize-of-computing/

2/ while his commitment to communication and his work on building community, from NetLib and Linpack to the MPI Forum and Top500 list, have spurred collaboration and co-opetition, often on grand scales. Today linear algebra underpins the deep learning revolution. Viva Dongarra!  

"A National Discovery Cloud" is a white paper that @dplopresti, Bill Gropp, Mark D. Hill, Katie Schuman, and I wrote for  CCC. I presented these slides at a CRA "Cloud for Computing Research" workshop https://cra.org/industry/events/cloudworkshop/ this week https://www.slideshare.net/ianfoster/foster-cra-march-2022pptx 

2) The white paper is at https://cra.org/ccc/wp-content/uploads/sites/2/2021/04/CCC-Whitepaper-National-Discovery-Cloud-2021.pdf. Our message is: "Think big. We have a historic opportunity to rethink how computing is organized and applied to empower the research community." In my talk, I led off with Doug Engelbart, who imagined powerful tools in 1962.

3) 60 years later, the opportunity is to create new intellect-augmenting tools that harness the power of 3 new technologies: a) public cloud, b) new sensors/big data/ML, and c) numerical (exasal) simulation as a powerful complement to experiment. What might these new tools be?

4) For example: vast datasets configured for interactive and collaborative analysis; a global knowledge graph linking all pubs, data, models; constantly updating digital twins of complex systems like the human body, climate, economy, manufacturing plants; and much more

5) We can create science services to democratize access to powerful analysis capabilities. E.g.: MG-RAST allows any scientist to upload a metagenomic dataset to obtain sophisticated analysis results--something otherwise impossible for many scientists www.mg-rast.org   

6) We can also leverage public cloud to create powerful services to power data- and compute-rich distributed science.   @globus is an example. Hosted on @awscloud, linking storage and data at 10000s of locations. What other services can we imagine? 

7) We point out in the white paper that realizing a fully integrated "National Discovery Cloud"  will require development in many areas, and addressing important issues of stability vs. innovation, incentives for creation and operation of new services, and much more.

8) I noted in summary: when using cloud for science, look beyond "public cloud as elastic computer"--that short changes it. It's a powerful platform for delivering new digital services. Our task is to work out what those services should be. 

9) These are questions that I have been pondering for a while, at least since this 1995 article on "Service-Oriented Science" --but substitute "cloud" for "grid" :-) https://people.cs.clemson.edu/~johnmc/courses/cpsc950/814.pdf 

A National Discovery Cloud

What a difference a decade makes for connecting scientists worldwide: The @globus counter in 2012 (7 PB total transferred, 100s MB/s aggregate) ...

2/ ... and in 2022 (1500 petabytes, 20s GB/s aggregate) -- and 200,000 registered users with 37,000 active endpoints at 1600 institutions in 80+ countries. @globus

3/ Plus so many other new capabilities: sharing, search, interfaces to dozens of storage systems, scheduled transfers, automation, computation via #funcx ...   Come to GlobusWorld in Chicago (and online) May 10-12, 2022 to learn more. https://globusworld.org 

You can read the unrolled version of this thread here: https://typefully.com/ianfoster/EIV4TOS