BP – Do you still have spare compute? We’d like to do some testing. I could send you manager configs.
DD + JW – Yes
DD – Yeah, happy to spin these up. We have a few clusters - A kubernetes cluster, an lsf, something at UCI and Max Planck. We usually submit managers as jobs and run pool
BP – Cool, you could do that, or I could help you set up Parsl. In the new version you can run parsl locally. Not sure if we’ll still have processpool support, maybe we’ll have threadpool.
BP – In the new version I should be able to have a single manager take from several queues.
DD – Using parsl may be helpful for alchemiscale… I’ve been taking a lot of the best parts of QCF for tihs. Would parsl manager need to be a single long-running process?
BP: Yeah, it does. Parsl is a bit more like a manager, DGA took inspiration from it to build QCF. You need to have something running on a head node. And, it doesn’t clean up that well, there would be some leftover processes that have to be killed.
BP – May reach out to parsl folks and try to fix up some of the rough bits. Project still seems active.
DD – Sounds good, I’m excited to learn about this
BP – Bit feature of the new managers is that you won’t need everything in the same environment - Using subprocess will allow us to use different conda envs, and things like docker and singularity. Would also allow using different versions of deps like psi4
DD – That’ll be handy - This was a big desire in the spice dataset work. Required lots of manual intervention at the time. Would this come with a system for task constraints, like “for these certian tasks, use these pinned deps”
BP – Looking forward to doing something like that but it’s not there now. Results already return which versions of deps were used to compute them.
DD – Happy to update managers any time, especially if we can keep you from needing to support multiple manager pathways
BP - Parsl is as difficult as dask, or pool, so figuring out one is good enough.
DD - JW, any updated on omsf servers?
JW - We have two NIH grants in line, one the renewal and the other is equipment supplement, no updates on both yet.