2022-10-04 Protein-ligand benchmarks meeting notes

DD : requesting 1 month extension on deployed system delivery; targeting 11/1 - decision required.

• RG – Approve

• JW – Anything that we can monitor and keep on track/can you identify the biggest bottlenecks or threats that we can keep pressure on?

• DD – Most of the remaining stuff is me. The biggest risk is that I get hung up on something with nobody to support me/bounce ideas. I’m not sure if there’s anyone in this group that would fit this role.

  • JC – We could look for additional people

  • JW – Hiring probably wouldn’t fix this on the timeline that we need.

  • LN – I know a little about dask, but probably not at the level we need.

  • DS – I’m not sure if I have time to contribute here.

  • RG – Would there be an in-kind exchange that we could do with DS’s time?

  • JW – I’d be happy to put in time toward the PDB reader in turn.

  • RG – Deal. DS should spend up to 3 hours per week on this, and record the total at the end. JW will make up for this in the PDB reading efforts.

  • DD – Also the noneq cycling protocols and protocolsettings taxonomy in GUFE. But I think that those are less at-risk than my components.

• (RG + JC + JW approve extending this to Nov 1)

• DD – Any other risks?

  • JC – F@H is a bit shaky, I’ve been assuming that this will need a few iterations and I’ve built this into my timeline. And me and Sukrit can provide support here.

  • DD – Understood.

MB, IA : protein-ligand-benchmark update (protein-ligand-benchmark#52):

IA – Meeting with openFE board recently - Board is concerned that we’re moving away from historical benchmarks. They’re concerned that we’re losing the ability to compare to how things were before. I told them that many of the structures before were unuable. They said that some of the new structures didn’t have minimizations that would finish in either gromacs or openmm.

JC – Some of the ligands were dropped (especially from the janssen sets) because they had large divergences in the common scaffold.

• MB – This happened for several targets - We’d drop several scaffolds because of this.

• JC – All of our codes require that the scaffolds be aligned. I’m not sure how these came in initialy.

IA – By “minimzation problems”, I’m referring to PLB issue #24. In that, there were clashes in the initial dataset, and the discussion indicated that they’re resolved by minimization. BNut now they’re concerned that the new sets won’t be ….

JC – I’d point them to the FE best practices paper and indicate that we’re applying those recommendations. These are often cases where the crystallographic conditions conflict with the assay considtion, or where the ligand can’t be docked in a reasonably confident pose.

• MB – That’s an accurate and consistent view of the issues we’re having. I think this is a reasonable thing to drop from the systems, because htese issues aren’t related to the free energy method, they’re related tot eh preparation of the systems. The other things that are causing the ligands to drop are different definitions of the core.

DD – JC, are we talking about how the benchmarking paper defined ebnchmrking vs. validation? Or the setup paper?

• JC – The….

• DD – It sounds like the board needs to distinguish between validationa nd benchmarking.

• JC – We may need to clarify this difference, and point out which sets are which and what goals we have. But I think the reponse to the board begins with the papers.

IA – That covers one part of it. The other part is that there seems to be a problem with the openmm minimizer. The board is also very concerned with reverse compatibility/comparability to the pmx-based benchmarks.

• JC – Could we run this on orion?

• MB – …

• JC – I think the board wants to see a few things run using the old systems. OpenEye has internally curated systems that they run. They may want to see…

• DD – I worked with David Hahn and Lorenzo ~one year ago to reproduce the pmx benchmarks, and I think it will be very difficult to reproduce it. I was unable to get it working. I think a lot of the stuff was manually handled and the code was on a fork of PMX.

• JC – Agree that we shouldn’t try to reproduce the old calcs.

• IA – I think we want to show that, if we’re trying to convince industry people who are using pmx to switch to something else, we should show that we get comparable results. So they want evidence to show that they can switch.

• DD – How are they using pmx at the moment? We’ve had a lot of trouble with it.

• IA – I don’t know. They want to be convinced that the results aren’t getting worse.

• DD – Would the orion path even satisfy them? IT’s not clear that it will.

• IA – I think it would satisfy them. It’s largely pmx with fixes.

• JC – Yeah, it’s basically an automated pmx with improvements and an openmm minimization.

• DD – IA, does this satisfy your needs?

• IA – I’d love to pursue the orion idea.

• JC will connect IA and MB with CBayly and GCalabro, and we can run some systems that we share using both types of preparation and both FE workflows (4 sets of calcs)

JW – OpenFF had previously talked about the risk of losing the “science” content of the set. So if there’s important “quality” lost then openFF will be with you in needing to prepare it again with more “science” content from the original.

DD – What’s remaining for PLB #52?

• IA – I think we should squash-merge it immediately and continue work in a new PR

• DD – I may have found some new clashes. I can work with IA and optionally MB

• MB – Re: large files. Because I ahd to manually prepare systems with different conditions, the docking grid files are pretty large (~130 MB a piece).

• JC – Had we agreed that the docking grid generation input scripts are sufficient? We could still upload the grids, but keep them out of the git tree.

• MB – Will do. I’ll upload the grid gen scripts to the git tree, and will keep the docking grids separate and upload them as a zip. I have the commit to remove the docking grids staged so I’ll push that momentarily.

• DD – Great, and IA and I will take it from there.

DD : fah-alchemy - current board status

• fah-alchemy : Phase 1 - MVP

• David Dotson development effort now focused on FahAlchemyAPIServer, FahAlchemyClient, and FahAlchemyComputeServer

  • Able to both submit and request AlchemicalNetworks via HTTP against FahAlchemyAPIServer pushing and pulling from neo4j. Can also reconstitute any other GufeTokenizable, in particular Transformations (edges) and ChemicalSystems (nodes).

  • networks with ProteinComponents also now work, but working through performance issues (gufe#76)

• DD – DS, I know you’re working on some fixes to the protein serialization. I’m using that in my testing, is that close to final?

  • DS – Yes.

• DD – I fixed some stuff re: deterministic sortings of alchemical systems.

• DS – I also made some significant performance improvements, coming soon.

• DD – PR #42 - Should we move result storage out of GUFE? I think OpenFE has some needs for results storage, fah-alchemy has similar but different ones.

  • DS – That will be a big discussion, so let’s do that later, and stay with the current trajectory for now.

  • DD – Agree

IP : Nonequilibrium Cycling Protocol (perses#1066) update:

• (IP is sick this week)

• DD – RG, would you have time to meet on this? Just want to do some high-level planning.

JC – IP had worked to set up and run the system from protein-ligand benchmark and open issues for problems. Did this happen?

• (General) – Yes, this happened and the discussion is ongoing.

MH : ProtocolSettings taxonomy (gufe#37) update:

• MH – Working on some stuff regarding encoding/serialization in openff-models. There’s a little problem with floatquantity and I’ve opened a PR to resolve.

• JW – Thanks for being proactive. MT is working half days this week but I can make sure he prioritizes action on this with his time if needed.

IA – Some OMSF folks are meeting with Oracle in a few minutes. I’m going to ask for about 50k GPU hours on the OpenFE side. Is there anything else we need?

JC – I’m also talking to google cloud. And AWS spot and AWS disaster recovery.

DD – I think the fah-alchemy doesn’t need this - We’re mostly using fah, and the object stores are on the chodera lab AWS.

DS – I think oracle’s motivation is to show that we’re running on their cloud. I’ll come to the meeting and represent our stance.

JW – I don’t think OpenFF has big GPU compute needs. And I think oracle’s goals are to get more paying customers, and we’re not really paying.

…

JC – I think Oracle wants to show pharma saving money by replacing wet chemistry with compute. I also think that some cloud providers like google want to show useful work coming from TPUs, where they have an edge in availability.

LN – I’ll also be on that call – we’ll be requesting nonspecific things for OpenFE and OpenFF, and also latest-hardware availability for CI.