2021-06-23 Industry benchmarks meeting notes

Topics and presentation content for partner call

David D.

• DD: What would you like to see presented at the partner call?

• DH: Think it should be a wrap up of Season 1.

  • next steps on publication, timeline

  • Schrodinger protocol as a supplement

  • Sage release inclusion as a supplement

  • perhaps discussion of Season 2

• LD: Xavier’s vs. Bill’s analysis

  • we haven’t heard anything from them on trying out the new analyses

  • DD: definitely will include as a slide to acknowledge Lorenzo’s work, want feedback

  • DH: want to avoid lengthy scientific discussion; not enough time

    • JW: on that topic, a follow up workshop on benchmarking is in demand

      • first hour could be about the benchmarking and results, followed by 20-30 mins discussion

• DD: presentation of results / figures aggregated across partners should be included

  • DH: certainly start with aggregated result plots

  • JW: would coverage analysis be of interest?

    • DD: perhaps produce

    • JW: might be too high dimensional, may have to obfuscate the data with principal components

      • would be interesting to know if we did well on small molecules vs. large molecules. So we could split the aggregate dataset by “molecules with 5 or less conformers” and “molecules with 6 or more conformers” and see if there’s a significant difference in metrics.

  • DH: I think seeing variance in parameter usage would be of interest; some partners may

  • JH: would like to see which parameters are very rarely exercised, which heavily used

  • DD: Agree. It would be good to put up some representation of parameter usage (the coverage report), perhaps normalized by number of molecules

  • JW - Review of how often match-minima selects a different minimum than the starting one

  • DH – Aggregated plots → Total n_mols, n_conformers,

    • DD – Agree, it’ll be cool to celebrate the collaboration+scale

    • JW – Analysis of gaps in molecule indexing to look at %success of the workflow.

  • DH: one could do a ranking of force fields, using Lim approach

    • derived from the ridge plots; but could also extract dE and RMSD values and rank against lowest overall

  • DD: Are there questions we can put at the end to gather inputs for decision making?

  • JH: could ask them to try running an analysis that Thomas Fox did, this would be useful for getting feedback on problematic chemistry

    • basically, would this feature be of interest more broadly

    • JW: is there an advantage of running this on their molecules vs. running on the public set?

    • DH – True, this would potentially be more work for us. So it may be good for us to focus on running this on the public set.

    • DD – It’s probably good to ask them for input. We have our own goals, but if those goals don’t align with what they’re interested in, then we may need to re-evaluate our goals, not the other way around. So we could ask for their input without committing to necessarily rolling the new analysis out.

  • DD – Probably not worth asking about season 2, unless something has changed since we last talked.

    • JW – If they ask about season 2, what is the current status?

    • DH – We want to wrap up season 1, then do refactoring and additional features, and season 2 would only start once we’re done with it.

    • DD – Agree. So we could say “late fall”

    • JW – Is season 2 a “maybe” or a “definitely”?

    • DD – I think there’s strong demand for it – They want to include torsiondrives, like the structure+schedule.

    • DH – So we could say “end of fall, IF we’re ready”. One strong argument for pushing it back is that future work won’t include more molecules, so unless we add new functionality (like torsiondrives) then there’s nothing that would be gained.

    • JW – Agree

    • Decision – We’ll say “Late Fall, IF infrastructure is ready”

    • DD – Additionally, this doesn’t mean “begin running in October”, it could mean “begin speccing/gathering feedback in october, deploy+run starting in January”

      • would be nice to set up an annual cadence for this kind of thing, and would be in advance of annual workshop

Follow-up workshop

Jeff

• JW: for benchmarks, both protein-ligand and optimizations, DH may be most knowledgeable

  • DH do you have the bandwidth to be the owner for this?

  • DH: I do not, unfortunately

  • JW: LD do you feel comfortable owning the workshop and organizing the talk and discussion for benchmarking?

    • LD: would feel more comfortable with optimization benchmarking; when is it?

• JW: late July is likely; probably an advisory board timeslot

  • DD and DH could be leaned on for protein-ligand component

    • DH: Lorenzo won’t need to talk on protein-ligand benchmarking

• JW: would be good to get Lorenzo more exposure as scientific role, credit for work in season 1 (and future season 2)

Schrodinger pathway

David H.

• Christina tried it out; hit same issue we did where schrodinger’s optimization fails to converge;

  • DH: may be a consequence of optimizer criteria choices aiming to mimic what we do for QC* optimizations

    • next step would be to just use schrodinger defaults; would be at some level a fairer comparison, and we still measure dE and RMSD relative to QM