Parsl at SC24

Attending SC24 in Atlanta? Drop by these events to see what the Parsl community is up to.

Removing Channels

In November 2024, Parsl's little used Channel abstraction is going away. Here are some details.

Outreachy Intern Spotlight – Meet Kelechi Nwankwo

Kelechi Nwankwo, Parsl's Outreachy intern this summer, is working on improving our user guide documentation to make it more accessible and easier to navigate for all levels of users. Check out this interview blog post to learn more about her!

Parsl and Globus Compute

We can conceptualize Globus Compute as an extension of Parsl, transforming a distributed network of Parsl executors into a Function as a Service (FaaS) platform.

Student Spotlight – Meet Jamison Kerney

Jamison Kerney (he/him/his), a recent computer science graduate from the Illinois Institute of Technology, is working on improving Parsl for fine-grained parallelism. Check out this interview blog post to learn more about him and his work with our community.

Passing Complex Data Structures in Computational Workflows – A Big Problem with a Simple Solution

Quantum Accelerator (quacc) streamlines computational materials science workflows for clean energy solutions. Using Parsl for orchestration, quacc simplifies the process, allowing users to concentrate on science and leverage advanced HPC resources. Despite the growth, quacc revealed limitations in data passing between tasks. This post discusses this challenge and how our community swiftly addressed this challenge, enhancing the effectiveness of materials discovery campaigns.

Debugging Memory Leaks in LightningBug's Virtual Label Reconstruction Pipeline

Hundreds of millions of pinned insects must be digitized to enable large-scale data-driven discovery in biodiversity science. We are using Parsl to facilitate high-throughput computational reconstruction of label information on each specimen. Here we will describe how a memory problem became a barrier to scaling and how we identified and fixed it!

Parsl at OLCF

In this blog post, we discuss three developments surrounding Parsl at OLCF.

An overlay architecture based on in-memory content delivery for funcX in edge-fog-cloud

With the constant production of data, it is required tools for the efficient management of those data through different infrastructures (e.g., the edge, the fog, or the cloud). To this end, we have designed an overlay architecture based on in-memory content delivery for funcX in edge-fog-cloud. Figure 1 presents the conceptual design of this overlay architecture, which includes a layer for managing the functions created by funcX. In this layer, the functions can be organized...

An Efficiency Evaluator for Parallel Scripting (EEPS) for Parsl

Hello! Our names are Ved Kommalapati and Ananth Hariharan and we are seniors at Normal Community High School in Normal, Illinois. We are both passionate about computer science and its applications in research. We both also plan on pursuing careers in the computer science field and we believe that this research opportunity helped us widen our horizons and gain experience in applying our computer science knowledge in research.

FluxExecutor efficiently integrates Parsl and Flux

To provide ExaWorks’s Software Development Kit (SDK) with Level-1 interoperability between Parsl and Flux, the ECP ExaWorks team recently integrated Parsl with Flux through a new Executor class, the FluxExecutor, for use wherever the Flux resource manager is installed. Groundwork for Parsl’s FluxExecutor was laid in Flux itself with the addition of a lower-level executor for use by all workflow systems. To use the new FluxExecutor, the only prerequisite is that there be an installation...

The ExaWorks SDK: Technologies for Composable and Scalable HPC Workflows

ExaWorks is a new Exascale Computing Project (ECP) effort to curate a community-designed Software Development Kit (SDK) of composable and reusable workflow technologies that can be incorporated into existing workflow systems and bespoke ECP application workflows. We are working with the workflow community to break down barriers between workflow management systems, initiating a move towards increased sharing of low-level functionality, and increasing the robustness and scalability of the crucial workflow components relied upon by ECP...

Using FuncX to execute Artificial Neural Networks on Remote Industrial Edge Resources: An application for fish processing industries

Within the climate change agenda, research studies report that 15% of global energy is consumed by operations related to refrigeration and air conditioning in the fish industry, which highlights the need for smart energy management solutions. While fish processing industries have high energy costs with continuous refrigeration, air conditioning, and ice making processes, there is a real need to analyse and model energy use. One solution that can be applied to optimize energy use in...

Diving for Treasure in a Sea of Scientific Literature: Extracting Scientific Information from Free Text Articles

It has become impossible for researchers to keep up with the more than 2.5 million publications published every year. We explore scalable approaches for automatically extracting relations from scientific papers (e.g., melting point of a polymer). We implement a dependency parser-based relation extraction model to understand relationships without the need for a Named Entity tagger, integrate several word embeddings models and custom tokenization to boost learning performance for scientific text.

SciFlow project using Parsl to execute Scientific Workflows on HPC resources

The majority of tasks that we, as researchers and analysts, perform, are conveniently expressible as scientific workflows. These workflows provide abstraction, integration and reusability, thereby easing the scientific knowledge discovery process. However, as the complexity of the scientific problem increases, the complexity of the workflow too increases proportionately. To facilitate such interactions, workflows utilize complex connectors.

A Look at Parsl and Funcx: Two Excellent Parallel Scripting Tools for Clouds and Supercomputers

In 2019, Yadu N Babuji, Anna Woodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin Michael Joseph Wozniak, Michael Wilde and Kyle Chard published a paper in HPDC ’19 entitled Parsl: Pervasive Parallel Programming in Python. I have been looking forward to finding the time to dig into it and give it a try. The time did arrive and, as I started to dig, I discovered some of this...