Implement PSI/J support in Pydra

[adi611]

About me: I am a Google Summer of Code 2023 contributor at INCF. Currently, my focus lies on enhancing the Pydra dataflow engine through the implementation of PSI/J support, which aims to provide a unified API for various schedulers in Pydra. Pydra: Pydra is a lightweight Python library for creating workflows that seamlessly scale to clusters. It provides a simple and consistent interface for defining tasks and workflows, as well as mechanisms like parallel execution.

My proposal involves the design and implementation of a Pydra worker class that utilizes PSI/J for the creation and submission of jobs (Pydra workflows) across different schedulers. Working alongside the PSI/J team to contribute to the development of psij-python and pursuing this objective would present a remarkable opportunity for me. I appreciate your consideration. Thank you.

[hategan]

Hi @adi611 and welcome.

We are here to help, so please don't hesitate to ask.

[adi611]

Hi, during my GSoC journey, I worked on creating a new worker class for Pydra called PsijWorker, which can be found here. The tests pass perfectly for this, but the only issue is that running the tests with slurm using psij is much slower than running with the default SlurmWorker class. Am I missing something in the implementation that could be causing this? Thanks, and I look forward to your reply.