Interface
Command-line interface executable from any shell preferable.
Discovery
Workflow components
...
Each workflow component from the diagram above is numbered below.
Options for software components indicated for each.
Separability of required workflow components will allow for parallelism in development activity. The
| Status | ||
|---|---|---|
|
will require the most development, and so should be prioritized.Status colour Red title dev
may require some development.Status colour Yellow title dev
has a well-known and heavily-used software pathway.Status colour Green title dev
Identifier assignment
Status colour Yellow title dev new, include in benchmarking library
Conformer generation (~10 conformers per molecule)
Status colour Green title dev openff-toolkit{rdkit}
Parameterization of molecules
Status colour Green title dev openff-toolkit{rdkit}
FF coverage report
Status colour Yellow title dev Reach out to Trevor, Jessica, Pavan for existing implementations
QCSubmit can give a list of all parameters used; doesn’t do counts currently, but could be made to
we’ll want counts, as this is richer information and allows us to prioritize coverage gaps
Energy minimization with Psi4 (QM), OpenMM (MM)
Status colour Red title dev multiple options
QCSubmit->QCFractal(->QCEngine->GeomeTRIC->QCEngine->Psi4/OpenMM)
this path allows easier extension to torsiondrives; not directly possible without significant development work with other paths
orchestration mostly solved in this path compared to the others
QCEngine->GeomeTRIC->QCEngine->Psi4/OpenMM
GeomeTRIC->QCEngine->Psi4/OpenMM
each option requires different considerations for deployment on queueing systems
simpler in terms of components may require additional development for deployment
Analysis and report generation
Status colour Red title dev can use components from
benchmarkff; need to extract and fold into benchmarking libraryno matter the approach chosen for optimizations in (5), we will need extraction tooling for flat file output, reports
...
encoder of OpenFF's preferences for dataset submissions to QCArchive
no compute on its own; requires use of QCFractal if part of workflow
important to ensure CMILES metadata in place to allow seamless MM calculations
client+worker+server for executing and storing procedures, such as optimizations
perhaps not strictly necessary, but may still be easiest path
complex solution may present failure modes that we have a hard time pinning down
QCEngine {vital}
features wrapper procedure to GeomeTRIC taking as input
QCElemental.OptimizationInputno need for QCFractal
not certain of value-add vs. GeomeTRIC directly, unless simplifies input
GeomeTRIC {vital}
optimization protocol
can use QCEngine internally to optimize using gradients from a variety of programs (engines)
evaluation analyses high value
not currently installable as a package; only scripts/notebooks
dependent on OpenEye Toolkit
will likely pull functionality out and create infrastructure home in
openff-benchmark
openff-toolkit {vital}
required for parameterization of molecules for OpenFF forcefields
required for GAFF, but also usable as abstraction layer for OpenFF forcefields, others
used in QCEngine for OpenMM execution
nouveau functionality for working with QCArchive data; utility functions in service to Trevor Gokey's research and work
possible to pull some prototype functionality we don't have in an infrastructure package
...
OpenEye Toolkit
cannot use for this purpose; must not be necessary for any part of the workflow
Packaging Options
openff-benchmark
Library components and entry points can be placed in openff.benchmark.geometry_optimization.
openff-cli
Could introduce an entrypoint in this package for distribution. (optional, and for later)
Proposal
Interface
Command-line interface executable from any shell preferable.
Identifier assignment
Implemented in openff
...
Currently abandoned; may be the perfect place for this right now.
Doesn't feature anything at this time.
Bit of a clean slate.
...
.benchmark.utils. Can be as simple as a function that takes as input group/company code (3-letter), all molecules (with predefined conformers, if present). Will then produce a mapping of identifiers to molecule objects as: COM-XXXXX-YY
three-letter company code (COM)
molecule-index (XXXXX)
numerical conformer-index (YY); 01, 02, 03,…
Note that with this approach, each molecule submitted in the dataset will have exactly one conformer. We would not be stacking multiple conformers into each Molecule.
Conformer generation
For molecules with fewer than 10 conformers predefined, additional conformers will be generated to give a total of 10. This can already be done with openforcefield.topology.Molecule.generate_conformers.
We will need the mapping from Identifier assignment after this, so it likely makes sense to switch the order of these workflow components.
Remaining questions
Do we care about easily distinguishing which conformers were provided vs. generated after the fact?
Parameterization of molecules
Parameterization of molecules will be performed with e.g.:
| Code Block |
|---|
from openforcefield.typing.engines.smirnoff import ForceField
# Load the OpenFF "Parsley" force field
forcefield = ForceField('openff-1.0.0.offxml')
# Parametrize the topology and return parameters used
off_topology = molecule.to_topology()
molecule_labels = forcefield.label_molecules(off_topology) |
The labels can then be fed directly to the Forcefield coverage report generator. An entry-point wrapping this and the coverage report can be placed in openff.benchmark.parameterization.
This step should be performed with each forcefield we are benchmarking.
Molecules that fail this step should be noted and left out of the energy minimization submission. We still want these in the coverage report that consumes the output of this step.
Forcefield coverage report
A function taking multiple sets of molecule labels from Parameterization of molecules to generate coverage reports should go into openff.benchmark.parameterization. This will the produce a report giving the counts for each parameter in the forcefield, aggregated over the molecules provided.
Although possible to provide a report for each molecule, to mitigate privacy concerns on the molecules used, it is recommended to generate a single report for the whole dataset.
Remaining questions
Should we enforce reports be aggregated? Can we show how possible it is to back-calculate a molecular structure based on the parameters used to parameterize it?
Energy minimization with Psi4 (QM), OpenMM (MM)
Proposing a three-pronged approach.
High-throughput (primary)
QCSubmit->QCFractal(->QCEngine->GeomeTRIC->QCEngine->Psi4/OpenMM)output extraction executable at any time for pulling available data
need error cycling process
High-throughput debug approach (secondary)
Trevor's local optimization executor
add this to QCSubmit; generally usable for OpenFF QCArchive users in debugging
components shared with (3)
GeomeTRIC->QCEngine->Psi4/OpenMMoutput still usable for reporting
Fully-local execution (alternative)
Like Horton's local TorsionDrive script, minus QCFractal execution if possible
components shared with (2)
GeomeTRIC->QCEngine->Psi4/OpenMMoutput still usable for reporting
In principle, (2) and (3) could be served via the same entrypoint.
(1) would make use of QCFractal with a persistent server to handle most of the compute orchestration.
These approaches should be given entry-points in openff.benchmark.geometry_optimization.
Once errors fail to clear in (1) and cannot be cleared in (2) or (3), these should be noted as failures in a way consumable by Analysis and report generation.
Analysis and report generation
Outputs produced in Energy minimization with Psi4 (QM), OpenMM (MM) should be directly consumable via an entry-point in openff.benchmark.geometry_optimization. We need the following included for each ID from Identifier assignment:
Relative energies (
E_MM - E_QM)Geometry comparison (RMSD or TFD, MM vs. QM)
Existing implementations should be drawn from benchmarkff. Where implementations are dependent on OpenEye Toolkit, alternatives must be put in place.
Deployment
A document describing compute stack installation, server stand up, and worker submission to queueing systems in use will need to be written and shared. This should include an upgrade pathway for the compute stack. This will likely draw on existing approaches for public QCArchive production compute.