2022-05-11 Industry Benchmarking meeting notoes

Benchmarking manuscript

• LD – Just met with DH and GT on benchmarking paper. Doing last round of revisions before sending to coauthors.

• LD – Working on final to-do list

  • Make figure colors consistent

  • JHorton has mentioned that we should check charged vs. neutral molecules to check performance.

    Open

    

  • JW – I don’t like that this is dependent on the order that the points are plotted

  • JW – Also it looks like we’re doing better on charged molecules than uncharged, since the blue cloud is bigger than the red

  • JW – Also, I’m not sure that it’s helpful to show the signed error, since it may not indicate what we want

    • DD – Should show standard deviation

  • LD – It looks like the blue distribution is more spread than the red distribution

  • LD – Also, we need to communicate that the uncharged error is smaller than the charged error. It looks different because there are a different number of points.

  • DD – We should present the standard deviation, NOT the standard deviation of the mean.

  • JW+LD – Could do scatter with histograms on the axes, but put it in the supp info. Because the scatter by itself doesn’t tell an interesting story, it’s just good for readers to see that there is no trend.

  • LD – JHorton and DMobley looked at these and noted that, while the charged error is more negative, OpenFF didn’t train on charged mols. Also that OPLS had a more negative mean error on charged mols and that custom parameters didn’t help much. Also that OPLS did well on RMSD but just about as poorly as others on ddE.

• LD – Analyzing outliers

  • DD – I’m confused by the slide showing the conventional/lucas/swope analyses

Open

• DD – This doesn’t make sense to me.

• JW – I need to leave, but I think the outlier analysis is a great opportunity to add to the impact of the paper. Maybe instead of looking at this just in terms of RMSD and ddE, a simple fingerprint/tanimoto/cheminformatics analysis could show the chemical features that correlate with high RMSD or ddE, and whether it’s different chemical groups for different FF.

•