New article in @Patterns_CP covers:
A) patterns for linking big data instruments & computers;
B) capturing such “flows” in reusable forms;
C) Globus automation services to run flows;
D) examples from synchrotron light sources;
E) implications for scientists & facilities, 1/8
A) Many instrument data capture and analysis applications engage a modest number of stereotypical patterns or "flows": e.g., acquire & quality control data; reconstruct data, train ML model on data, catalog & store data, deploy trained ML model, 2/8
B) We show how such flows can be represented in reusable forms, as a sequence of actions but with details like analyses and target instruments, computers, and repositories unspecified. We introduce the Gladier Python toolkit for that purpose, 3/8
C) We describe how flow orchestration, and mapping to specific computers, repositories, etc., can be handled by Globus Automation services, which extend the popular @globus platform to encompass flow creation, orchestration, and related tasks, globus.org/instruments 4/8
D) We present five synchrotron light source examples: x-ray photon correlation spectroscopy, serial synchrotron crystallography, 2x high-energy diffraction microscopy, ptychography. Each includes simplified code that you can run yourself, 5/8
E) We use those same five examples to illustrate experiences at experimental facilities (@advancedphoton, #LCLS) & compute facility (@argonne_lcf), examining trends in usage, compute volume, data volumes, & performance, & discuss implications for scientists and facilities, 6/8
Article is "Linking Scientific Instruments and Computation: Patterns, Technologies, Experiences", Patterns 3(10):100606, 2022, doi.org/10.1016/j.patter.2022.100606. Co-authors on Twitter: @BenBlaiszik @chard_kyle @lzhengchun @mpapka. Give it a try and let us know how you do! 7/8
Also see Supplementary Information with pointers to code at cell.com/cms/10.1016/j.patter.2022.100606/attachment/138fdc7e-ae62-4734-a802-4b6763337386/mmc1.pdf. Thanks to @NSF CISE OAC, @doescience ASCR, @doescience BES, @NIST, @argonne_lcf, @argonne, @UChicago, @advancedphoton 8/8