2021-06-30 Industry benchmarks meeting notes

Xavier issue with analysis

Jeff

• DD: I’m going to instruct Xavi to inject a try-except around the rdkit failure, report conformer id, remove from the set so analysis can proceed

Postmortem/retrospective

Jeff

• JW: would like us to have a retrospective on what went well, what challenges we encountered, and how we could better address those challenges if we did this again (such as another campaign)

• DH: Experience with cloud-based deployments, still challenges there, especially basic questions from users with e.g. Jupyter notebooks, cloning repos, etc.

• JW: think these platforms improve folks' technical ability by giving them a variety of platforms to do productive work on, identifying commonalities, differences

  • I do like the Docker model

• DH: I don’t think we could have made a much smoother experience in deployment than we had, given our audience

• JW: At Pfizer, had a common compute platform that had plenty of its own quirks; imagine that platforms across companies so fragmented that deploying across all of them is a general challenge

• Scheduled retrospective for July 21

Thomas Fox’s analysis

Lorenzo

• LD: after meeting with DM, DH, GT, have some ideas for using Thomas Fox’s analysis

  • JW – Bonds and angles make sense to look at directly, but torsions may be more complex, since there are many torsion terms about a single central bond. So if a particular torsion is bad, flag ALL the torsion parameters that got assigned to the central bond.

    • LD + DH – Agree. Though torsion violations are even more complicated because they could also be due to nonbonded interactions

    • SB – Since you’re looking at parameters that are OVERrepresented in bad molecules. So we can handle some amount of noise as long as we’re aggregating things over a large dataset.

    • JW – I think that torsions are relatively “noisy”, but bonds

  • JW: you could start using the openff-interchange object for a more interactive view of the parameterization; not clear if it will offer a distinct advantage over your current approach, but would help MT to stress-test

  • LD: regarding central atoms, and torsions being a summation of all the torsions across the same central bond, need to carefully handle deduplication based on atom indices

  • JW: importantly, you’ll want to skip charge assignment if you choose to use the interchange object; will need to ask Matt on how to do this, as not sure how to do it

    • SB: can remove the charge handler; that may be the only real way to do it at the moment with the interchange object

  • LD: working on advancing this analysis in the following days

• SB – Sometimes this information will be hard to interpret without having the full molecule. So this analysis will be most useful to run in-house.

  • DH – Agree, we should start with the public molecules.

  • DD – This fits well with our organizational goals and the scope of openff-benchmark, so please open a PR when you think it’s ready.

New toolkit release

Jeff

• JW: had long standing issue where the sum of partial charges from ambertools would be 10^-3 for some atoms across a molecule

  • when we switch to using the new toolkit going forward (unpin old version), we may see differences in charges, but I doubt that these will affect the conformer benchmarks.

Refactor/rewrite of openff-benchmark

• SB – More solid plans for rewrite of benchmark?

  • in particular, running custom FFs through it

  • JW – Not yet, I’m scheduled to start scoping in July. I’ll have a meeting on scoping (probably hijacking this meeting slot)

• Scheduled retrospective for July 21