Benchmark dashboard

Driver

Approver

Contributors

Stakeholder

Driver

Approver

Contributors

Stakeholder

@John Chodera

 

@John Chodera @Jaime Rodríguez-Guerra @Jeffrey Wagner @David Hahn @Simon Boothroyd @Daniel Smith (Deactivated)

@Karmen Condic-Jurkic @David Mobley

Objective

Develop an online dashboard prototype displaying comparison benchmarks between public force fields and Open Force Field Initiative force fields (multiple generations)

Due date

May 1, 2019 

Key outcomes

A basic dashboard displaying quantum chemical, physical property, and free energy benchmarks should be available by the May OpenFF meeting

Status

PLANNING

Problem Statement

We need a way to communicate how our force fields compare to other public force fields.

We have different needs for reaching different communities:

Consortium Partners:

  • Our Consortium Partners need a clear, visually appealing way to persuade management to continue to invest in the Consortium. If we can clearly show both (1) good performance compared to public force fields like GAFF, and (2) continued improvement on benchmarks, we provide them with the tools they need to do this.

  • Consortium Partner scientists would also like a way to examine how well our force fields perform in different scenarios compared to (1) various quantum chemical methods, (2) various physical properties of interest, and (3) in predicting free energies for various target classes. If we can avoid these scientists from having to engineer their own tools to do these comparisons, our Partner scientists will go to bat for us.

Force field communities:

  • We need to persuade other force field communities to work with us to get their force fields into our automated benchmarks. If we have a critical mass of force fields included in this benchmark, other force field developers will be incentivized to work with us to get their force fields into our ecosystem.

Open Force Field Scientists:

  • An automated benchmark framework that allows OFF scientists to drill down and inspect failures more interactively will---until we have fully automated means of refining typing schemes---accelerate the refinement of typing schemes.

Scope

Must have:

  • Interactive online dashboard illustrating quantum chemical accuracy benchmarks using data pulled live from QCArchive by QCPortal

  • Interactive online dashboard illustrating physical property comparisons, presumably pulled from GitHub repo in common format

  • Interactive online dashboard illustrating free energy benchmarks, presumably pulled from GitHub repo in common format

  • Hover-over SMILES strings or calculation IDs

Nice to have:

  • Hover-over molecule structures (and possibly protein structures and transformation mappings for protein-ligand complexes)

  • Data available for multiple free energy methods

  • Typing coverage statistics for large molecule sets of interest to drug discovery (e.g. Enamine REAL, DrugBank, etc.)

Not in scope:

  • Dashboard will be static, so no new calculations can be triggered this way

  • Automated free energy and physical property computation infrastructure will be worked out in a different project; the dashboard will just visualize results

Timeline

Milestones and deadlines

Milestone

Owner

Deadline

Status

Milestone

Owner

Deadline

Status

Graphical mockup webpage

@John Chodera @Jaime Rodríguez-Guerra

7 Mar 2020

IN PROGRESS

Ensure MM QC data for MM force fields is available in QCArchive

@John Chodera @Daniel Smith (Deactivated)

15 Mar 2020

IN PROGRESS

Prototype dashboard for quantum chemistry data

@John Chodera

15 Mar 2020

NOT STARTED

Integration of free energy benchmarks

@David Hahn @John Chodera

30 Mar 2020

NOT STARTED

Integration of physical property data plotting

@Simon Boothroyd @John Chodera

15 Apr 2020

NOT STARTED

Usability refinements

@John Chodera @Karmen Condic-Jurkic @Jaime Rodríguez-Guerra

30 Apr 2020

NOT STARTED

Public launch of prototype

@John Chodera @Karmen Condic-Jurkic

4 May 2020

NOT STARTED

Reference materials