Benchmark dashboard

Problem Statement

We need a way to communicate how our force fields compare to other public force fields.

We have different needs for reaching different communities:

Consortium Partners:

• Our Consortium Partners need a clear, visually appealing way to persuade management to continue to invest in the Consortium. If we can clearly show both (1) good performance compared to public force fields like GAFF, and (2) continued improvement on benchmarks, we provide them with the tools they need to do this.

• Consortium Partner scientists would also like a way to examine how well our force fields perform in different scenarios compared to (1) various quantum chemical methods, (2) various physical properties of interest, and (3) in predicting free energies for various target classes. If we can avoid these scientists from having to engineer their own tools to do these comparisons, our Partner scientists will go to bat for us.

Force field communities:

• We need to persuade other force field communities to work with us to get their force fields into our automated benchmarks. If we have a critical mass of force fields included in this benchmark, other force field developers will be incentivized to work with us to get their force fields into our ecosystem.

Open Force Field Scientists:

• An automated benchmark framework that allows OFF scientists to drill down and inspect failures more interactively will---until we have fully automated means of refining typing schemes---accelerate the refinement of typing schemes.

Scope

Must have:

• Interactive online dashboard illustrating quantum chemical accuracy benchmarks using data pulled live from QCArchive by QCPortal

• Interactive online dashboard illustrating physical property comparisons, presumably pulled from GitHub repo in common format

• Interactive online dashboard illustrating free energy benchmarks, presumably pulled from GitHub repo in common format

• Hover-over SMILES strings or calculation IDs

Nice to have:

• Hover-over molecule structures (and possibly protein structures and transformation mappings for protein-ligand complexes)

• Data available for multiple free energy methods

• Typing coverage statistics for large molecule sets of interest to drug discovery (e.g. Enamine REAL, DrugBank, etc.)

Not in scope:

• Dashboard will be static, so no new calculations can be triggered this way

• Automated free energy and physical property computation infrastructure will be worked out in a different project; the dashboard will just visualize results

Timeline

Milestones and deadlines

Reference materials

• See example from Daniel Smith for inspiration:
  https://qcarchive-webapps.herokuapp.com/reaction_viewer/

  (code is available to us and only ~100 lines of Python)

• Will make use of dash to quickly construct a dashboard with Python
  See the dash app gallery for examples

  • Here’s an example that combines a plot with a molecule structure display

• Individual plots will use plotly to plot data interactively