Blog

FluxExecutor efficiently integrates Parsl and Flux

James Corbett (LLNL) and Dong H. Ahn (LLNL)
30 June 2021
To provide ExaWorks’s Software Development Kit (SDK) with Level-1 interoperability between Parsl and Flux, the ECP ExaWorks team recently integrated Parsl with Flux through a new Executor class, the FluxExecutor, for use wherever the Flux resource manager is installed. Groundwork for Parsl’s FluxExecutor was laid in Flux itself with the addition of a lower-level executor for use by all workflow systems. To use the new FluxExecutor, the only prerequisite is that there be an installation...

The ExaWorks SDK: Technologies for Composable and Scalable HPC Workflows

Dan Laney (LLNL), Dong Ahn (LLNL), Kyle Chard (ANL), Shantenu Jha (BNL), Tom Uram (ANL), Justin Wozniak (ANL)
22 April 2021
ExaWorks is a new Exascale Computing Project (ECP) effort to curate a community-designed Software Development Kit (SDK) of composable and reusable workflow technologies that can be incorporated into existing workflow systems and bespoke ECP application workflows. We are working with the workflow community to break down barriers between workflow management systems, initiating a move towards increased sharing of low-level functionality, and increasing the robustness and scalability of the crucial workflow components relied upon by ECP...

Using FuncX to execute Artificial Neural Networks on Remote Industrial Edge Resources: An application for fish processing industries

Ioan Petri, Ioan Chirila, Ciprian Chirila, Yacine Rezgui, Omer Rana, School of Engineering, Cardiff University, UK
12 March 2021
Within the climate change agenda, research studies report that 15% of global energy is consumed by operations related to refrigeration and air conditioning in the fish industry, which highlights the need for smart energy management solutions. While fish processing industries have high energy costs with continuous refrigeration, air conditioning, and ice making processes, there is a real need to analyse and model energy use. One solution that can be applied to optimize energy use in...

Diving for Treasure in a Sea of Scientific Literature: Extracting Scientific Information from Free Text Articles

Aarthi Koripelly, University of Chicago
25 January 2021
It has become impossible for researchers to keep up with the more than 2.5 million publications published every year. We explore scalable approaches for automatically extracting relations from scientific papers (e.g., melting point of a polymer). We implement a dependency parser-based relation extraction model to understand relationships without the need for a Named Entity tagger, integrate several word embeddings models and custom tokenization to boost learning performance for scientific text. The exponential growth of scientific...

SciFlow project using Parsl to execute Scientific Workflows on HPC resources

Amanda Wijewickrama and Rajini Wijayawardana, University of Colombo School of Computing, Sri Lanka
19 January 2021
The majority of tasks that we, as researchers and analysts, perform, are conveniently expressible as scientific workflows. These workflows provide abstraction, integration and reusability, thereby easing the scientific knowledge discovery process. However, as the complexity of the scientific problem increases, the complexity of the workflow too increases proportionately. To facilitate such interactions, workflows utilize complex connectors. The SciFlow framework provides a set of compositional channel connectors in a control thread, which can be used to...

A Look at Parsl and Funcx: Two Excellent Parallel Scripting Tools for Clouds and Supercomputers

Dennis Gannon, School of Informatics, Computing and Engineering, Indiana University
11 January 2021
In 2019, Yadu N Babuji, Anna Woodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin Michael Joseph Wozniak, Michael Wilde and Kyle Chard published a paper in HPDC ’19 entitled Parsl: Pervasive Parallel Programming in Python. I have been looking forward to finding the time to dig into it and give it a try. The time did arrive and, as I started to dig, I discovered some of this...