2020-11-11 Benchmarking for Industry Partners - Development Meeting notes

How to handle direct requests from industry partners

David H

• @David Dotson and @David Hahn will handle cases where partners send non-3D-SDF files, e.g. SMILES; usual approach would be:

  • ask them to generate 3D SDF files

  • if they push back:

    • offer to proceed with what they’ve provided, but point out possible problems and that they may not be getting exactly what they intended

• For stereocenters, randomly generate one representation. ditto for protomers

• For charges, if not explicitly stated, do neutral molecule

Coverage report backcalculation

Jeff

• Want to make sure there is no risk of people producing a single-molecule coverage report

  • the entrypoint could throw an exception if fewer than 10 molecules fed to it; could be overriden with e.g. --force.

Specification for workflow steps

• Should we have a common structure for steps between components? Files? Python objects?

  • DH – For benchmarking metric calculation, SDF would be best

  • JH – How to store energies?

    • SDF tags – Each value should be printed as a string like “number * unit”

• Should we have a common hierarchy format?

  • JW – In favor of folders like 1-molecule_naming, 2-conformer_generation, etc

• Should we have a common structure for workflow steps? CLI tools? Other?

  • CLI entrypoints, SDF files as molecule intermediates in directories

  • guardrails : if an output directory already exists, CLI tool should fail; this helps ensure consistency in the workflow

  • for long-running steps, could be a continue flag

  • Could have a provenance file in each step, containing openff-benchmark version, and maybe also deps version

Revision pathway

Jeff

• How do we want to do revisions?

  • we need to come up with scenarios, and possible ways it could work.

  • We’ll make an issue for this, solicit ideas.