Internal benchmarking project plan

Owned by Lily Wang, created
Last updated: Apr 19, 2023
2 min read

 

Driver

Approver

Notify list

Driver

Approver

Notify list

@Matt Thompson

@Lily Wang

@Jeffrey Wagner

 

Objective

Develop a package to support flexible, modular components of benchmarking for internal efforts to benchmark force fields

Due date

end of 2023

Status

IN progress

Problem Statement

Up until now benchmarking force fields has required a fair amount of work from users. Benchmark metrics used consistently to benchmark force fields (e.g. ddE, TFD) have largely been done using scripts passed around and iteratively modified to fit the situation. The industry benchmarking effort [LINK] was a huge project and bunch of scripts that again required ad-hoc modifications when new targets, force fields or systems were introduced. Developing a benchmarking package that makes individual steps of benchmarking modular and general would allow internal users to more easily do benchmarking and trial new targets.

Phase 1 - Milestones

Milestone

Status

Milestone

Status

Can rerun the quick QM targets (TFD, ddE, RMSD) and compare them to another OpenFF force field, e.g. an early version of Rosemary or Parsley

NOT STARTED

Can rerun all Sage benchmarks (QM and physical property data) and compare them to Rosemary

NOT STARTED

Support custom SMIRNOFF force fields (i.e. support using whatever SMIRNOFF/interchange plugin system exists at the time)

NOT STARTED

(Stretch) Support more general benchmarks (e.g. NMR shifts for Rosemary)

NOT STARTED

 

Phase 1 - In scope

  • Dispatch distributed calculations to QCFractal and Evaluator; support caching etc

Phase 1 - Out of scope

  • Force fields not in OFFXML format

    • GAFF via openmmforcefields – proceed as if it is in scope

  • Using this infrastructure for FF optimization (eg, calling benchmarking routines repeatedly with slight numerical differences in the OFFXML and expecting derivatives or fast answers)

  • Guaranteed API stability

Not considered for upcoming phases

 

 

 

 

Molecules that can’t be loaded by OFF Toolkit (maybe)

 

OPLS/schrodinger suite stuff

 

References

Previous meeting notes (move if there should be a better home)