Blog

A space for users to share ideas, research, insight, and experiences and learn about the exciting things happening around the community. Interested in authoring a guest blog? Email us!


Passing Complex Data Structures in Computational Workflows – A Big Problem with a Simple Solution

Ben Clifford (Parsl) and Andrew S. Rosen (University of California, Berkeley)
30 November 2023

Quantum Accelerator (quacc) streamlines computational materials science workflows for clean energy solutions. Using Parsl for orchestration, quacc simplifies the process, allowing users to concentrate on science and leverage advanced HPC resources. Despite the growth, quacc revealed limitations in data passing between tasks. This post discusses this challenge and how our community swiftly addressed this challenge, enhancing the effectiveness of materials discovery campaigns.

Debugging Memory Leaks in LightningBug's Virtual Label Reconstruction Pipeline

Mark Hereld, Senior Experimental Systems Engineer and Senior Scientist, Argonne National Laboratory
31 August 2023

Hundreds of millions of pinned insects must be digitized to enable large-scale data-driven discovery in biodiversity science. We are using Parsl to facilitate high-throughput computational reconstruction of label information on each specimen. Here we will describe how a memory problem became a barrier to scaling and how we identified and fixed it!

Parsl at OLCF

Ketan Maheshwari, Sean Wilkinson, Tyler Skluzacek, and Rafael Ferriera da Silva (NCCS and OLCF, ORNL)
30 September 2022

In this blog post, we discuss three developments surrounding Parsl at OLCF.

An overlay architecture based on in-memory content delivery for funcX in edge-fog-cloud

Dante D. Sánchez-Gallegos, Alfredo Barron, J. L. Gonzalez-Compean (Cinvestav Tamaulipas)
22 September 2022

With the constant production of data, it is required tools for the efficient management of those data through different infrastructures (e.g., the edge, the fog, or the cloud). To this end, we have designed an overlay architecture based on in-memory content delivery for funcX in edge-fog-cloud. Figure 1 presents the conceptual design of this overlay architecture, which includes a layer for managing the functions created by funcX. In this layer, the functions can be organized...

An Efficiency Evaluator for Parallel Scripting (EEPS) for Parsl

Ved Kommalapati and Ananth Hariharan
16 August 2021

Hello! Our names are Ved Kommalapati and Ananth Hariharan and we are seniors at Normal Community High School in Normal, Illinois. We are both passionate about computer science and its applications in research. We both also plan on pursuing careers in the computer science field and we believe that this research opportunity helped us widen our horizons and gain experience in applying our computer science knowledge in research.

FluxExecutor efficiently integrates Parsl and Flux

James Corbett (LLNL) and Dong H. Ahn (LLNL)
30 June 2021

To provide ExaWorks’s Software Development Kit (SDK) with Level-1 interoperability between Parsl and Flux, the ECP ExaWorks team recently integrated Parsl with Flux through a new Executor class, the FluxExecutor, for use wherever the Flux resource manager is installed. Groundwork for Parsl’s FluxExecutor was laid in Flux itself with the addition of a lower-level executor for use by all workflow systems. To use the new FluxExecutor, the only prerequisite is that there be an installation...

The ExaWorks SDK: Technologies for Composable and Scalable HPC Workflows

Dan Laney (LLNL), Dong Ahn (LLNL), Kyle Chard (ANL), Shantenu Jha (BNL), Tom Uram (ANL), Justin Wozniak (ANL)
22 April 2021

ExaWorks is a new Exascale Computing Project (ECP) effort to curate a community-designed Software Development Kit (SDK) of composable and reusable workflow technologies that can be incorporated into existing workflow systems and bespoke ECP application workflows. We are working with the workflow community to break down barriers between workflow management systems, initiating a move towards increased sharing of low-level functionality, and increasing the robustness and scalability of the crucial workflow components relied upon by ECP...

Using FuncX to execute Artificial Neural Networks on Remote Industrial Edge Resources: An application for fish processing industries

Ioan Petri, Ioan Chirila, Ciprian Chirila, Yacine Rezgui, Omer Rana, School of Engineering, Cardiff University, UK
12 March 2021

Within the climate change agenda, research studies report that 15% of global energy is consumed by operations related to refrigeration and air conditioning in the fish industry, which highlights the need for smart energy management solutions. While fish processing industries have high energy costs with continuous refrigeration, air conditioning, and ice making processes, there is a real need to analyse and model energy use. One solution that can be applied to optimize energy use in...

Diving for Treasure in a Sea of Scientific Literature: Extracting Scientific Information from Free Text Articles

Aarthi Koripelly, University of Chicago
25 January 2021

It has become impossible for researchers to keep up with the more than 2.5 million publications published every year. We explore scalable approaches for automatically extracting relations from scientific papers (e.g., melting point of a polymer). We implement a dependency parser-based relation extraction model to understand relationships without the need for a Named Entity tagger, integrate several word embeddings models and custom tokenization to boost learning performance for scientific text.

SciFlow project using Parsl to execute Scientific Workflows on HPC resources

Amanda Wijewickrama and Rajini Wijayawardana, University of Colombo School of Computing, Sri Lanka
19 January 2021

The majority of tasks that we, as researchers and analysts, perform, are conveniently expressible as scientific workflows. These workflows provide abstraction, integration and reusability, thereby easing the scientific knowledge discovery process. However, as the complexity of the scientific problem increases, the complexity of the workflow too increases proportionately. To facilitate such interactions, workflows utilize complex connectors.

A Look at Parsl and Funcx: Two Excellent Parallel Scripting Tools for Clouds and Supercomputers

Dennis Gannon, School of Informatics, Computing and Engineering, Indiana University
11 January 2021

In 2019, Yadu N Babuji, Anna Woodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin Michael Joseph Wozniak, Michael Wilde and Kyle Chard published a paper in HPDC ’19 entitled Parsl: Pervasive Parallel Programming in Python. I have been looking forward to finding the time to dig into it and give it a try. The time did arrive and, as I started to dig, I discovered some of this...