Geometry Optimization Benchmark - Season 1 : Announcement Stream

Owned by David Dotson
Last updated: Sept 13, 2022 by Lorenzo D'Amore
23 min read

2022.07.06-0

Draft, not yet distributed

Greetings!

We have now received approval to submit our benchmark manuscript from all partners!

  • Arjun Narayanan of Vertex 09/06

  • Ian Craig of BASF got clearance on 07/07

  • Chris Bayly from OpenEye got clearance on 07/06

  • Gary Tresadern of Janssen got clearance on 07/06

  • Thomas Fox of Boehringer-Ingelheim got clearance on 06/23

  • Xavier Lucas of Roche got clearance on 06/22

  • Katharina Meier of Bayer got clearance on 06/21

  • Alberto Gobbi of Genentech got clearance on 06/16

  • Christina Schindler of Merck got clearance on 06/13

  • Bai Xue of XtalPi got clearance on 06/08

  • Kaushik Lakkaraju of BMS got clearance on 03/17

We are extremely grateful to each of you for the commitment to request approval.

Whether possible, we look to hear back from the other partners as soon as possible.

We aim to submit the manuscript to Journal of Computer-Aided Molecular Design once all partners will have completed their internal approval process. Please let us know when this is done.

Cheers! Thanks again to everyone for their contribution to the OpenFF Geometry Optimization Benchmark Manuscript!

The OpenFF Benchmark Team

2022.07.01-0

Distributed to all stakeholders on 2022.07.01 14:30 UTC

Happy Friday!

We have now received approval to submit our benchmark manuscript from 7/10 partners!

  • Thomas Fox of Boehringer-Ingelheim got clearance on 06/23

  • Xavier Lucas of Roche got clearance on 06/22

  • Katharina Meier of Bayer got clearance on 06/21

  • Alberto Gobbi of Genentech got clearance on 06/16

  • Christina Schindler of Merck got clearance on 06/13

  • Bai Xue of XtalPi got clearance on 06/08

  • Kaushik Lakkaraju of BMS got clearance on 03/17

We are extremely grateful to each of you for the commitment to request approval.

Whether possible, we look to hear back from the other partners as soon as possible.

We aim to submit the manuscript to Journal of Computer-Aided Molecular Design once all partners will have completed their internal approval process. Please let us know when this is done.

Cheers! Thanks again to everyone for their contribution to the OpenFF Geometry Optimization Benchmark Manuscript!

The OpenFF Benchmark Team

2022.06.13-0

Distributed to all stakeholders on 2022.06.21 16:20 UTC

Greetings!

We have now received approval to submit our benchmark manuscript from 3/10 partners!

  • Alberto Gobbi of Genentech got clearance on 06/21

  • Christina Schindler of Merck got clearance on 06/13

  • Bai Xue of XtalPi got clearance on 06/08

We are extremely grateful to each of you for the commitment to request approval.

Whether possible, we look to hear back from the other partners as soon as possible.

We aim to submit the manuscript once all partners will have completed their internal approval process. Please let us know when this is done.

Cheers! Thanks again to everyone for their contribution to the OpenFF Geometry Optimization Benchmark Manuscript!

The OpenFF Benchmark Team

2022.06.02-0

Distributed to all stakeholders on 2022.06.02 21:35 UTC

Greetings!

A final draft of the Geometry Optimization Benchmark Manuscript is finally available on Overleaf.
We thank all partners for their valuable feedback and comments. The suggestions made have substantially improved the manuscript.

If you believe that the final draft is ready for submission, please do ask your company for clearance as soon as possible. If you think something is missing or still needs to be addressed, please let us know.
At the moment, the Overleaf link works in read-only mode. We are reaching out Support to fix the issue with edit mode. Meanwhile, if you want to add comments or make changes to the final draft, please reach out directly to us until it gets fixed.

Whether possible, we look to hear back from partners within a couple of weeks. We aim to submit the manuscript once all partners will have completed their internal approval process. Please let us know when this is done.

Cheers! Thanks again to everyone for their contribution to the OpenFF Geometry Optimization Benchmark Manuscript!

The OpenFF Benchmark Team

2022.03.17-0

Distributed to all stakeholders on 2022.03.17 12:45 UTC

Greetings!

We thank all the partners who already reviewed the manuscript.

For all those partners who have not yet reviewed it and wish to contribute,
we kindly recall the deadline of Monday 21th March.

You can access the manuscript on Overleaf.

Cheers! We value your feedback and comments to further improve the manuscript!

The OpenFF Benchmark Team

2022.02.26-0

Distributed to all stakeholders on 2022.02.26 10:50 UTC

Greetings!

A first draft of the Optimization Benchmark Manuscript is now available on Overleaf.

We thank all partners for their effort to produce the aggregated results that are discussed in the draft.

Our intention is to add further analysis based on problematic torsions, concerning cases and outliers with significance on the dataset.

We warmly welcome suggestions of further areas for analysis, ideas and inputs to the discussion from the rest of the industry group.

The draft is open to contribution from any partner, we aim to have this first iteration of discussion completed by 21th March.

Cheers! We value your feedback and comments to further improve the manuscript!

The OpenFF Benchmark Team

2021.11.22-0

Distributed to all stakeholders on 2021.11.22 18:43 UTC

Greetings!

Thanks to everyone that has given us valuable feedback via our retrospective survey!

If you haven’t yet, there is still time to fill out the survey: OpenFF Optimization Benchmark - Season 1 : Retrospective Survey

Please answer as many questions as you can. We are looking for your candid feedback, and this will inform our decision-making for the next Season.

We look forward to working with you in the next campaign to evaluate and accelerate the work of OpenFF!

The OpenFF Benchmark Team

2021.11.19-0

Distributed to all stakeholders on 2021.11.19 19:45 UTC

Happy Friday!

We have now received all (10/10) Sage RC2 results!

  • Thomas Fox of Boehringer-Ingelheim submitted his results on 11/15

  • Arjun Narayanan of Vertex submitted his results on 11/9

  • Kaushik Lakkaraju of BMS submitted his results on 11/5

  • Bill Swope of Genentech submitted his results on 11/4

  • Christina Schindler of Merck submitted her results on 10/15

  • Katharina Meier of Bayer submitted her results on 10/8

  • Bai Xue of Xtal Pi submitted his results on 9/28

  • Lorenzo D’Amore of Janssen/OpenFF submitted his results on 9/21

  • Ian Craig of BASF submitted his results on 9/20

  • Xavier Lucas of Roche submitted his results on 8/17

We are extremely grateful to each of you for leading this extra benchmark effort.

All the results are now being aggregated and will be presented in the OpenFF manuscript coordinated by Gary Tresadern of Janssen. We are working on a first draft of the manuscript.

Cheers! Thanks again to everyone for all the efforts done and the valuable contribution over the course of this campaign!

The OpenFF Benchmark Team

2021.11.09-0

Distributed to all stakeholders on 2021.11.10 17:50 UTC

Greetings!

Thank you to everyone that worked to make the Season 1 benchmark a successful, coordinated effort!

In order to build on our experience and lessons-learned from the Season 1 campaign, we have prepared this short survey: OpenFF Optimization Benchmark - Season 1 : Retrospective Survey

Please answer as many questions as you can. We are looking for your candid feedback, and this will inform our decision-making for the next Season.

We look forward to working with you in the next campaign to evaluate and accelerate the work of OpenFF!

The OpenFF Benchmark Team

2021.10.29-0

Distributed to all stakeholders on 2021.10.29 16:00 UTC

Greetings!

We have now received Sage RC2 results from 6/10 partners!

  • Christina Schindler of Merck submitted her results on 10/15

  • Katharina Meier of Bayer submitted her results on 10/8

  • Bai Xue of Xtal Pi submitted his results on 9/28

  • Lorenzo D’Amore of Janssen/OpenFF submitted his results on 9/21

  • Ian Craig of BASF submitted his results on 9/20

  • Xavier Lucas of Roche submitted his results on 8/17

Thanks to each of you for your submission!

We ask the remaining four partners to submit Sage RC2 results as soon as possible, as a key component for the manuscript being coordinated by Gary Tresadern of Janssen

You can run the analysis workflow and export your shareable data only for Sage RC2 results to save time:

openff-benchmark report compare-forcefields \ --input-path 4-compute-qm-filtered \ --input-path 4-compute-mm-filtered/openff-2.0.0-rc.2 \ --ref-method b3lyp-d3bj \ --output-directory 5-compare-forcefields-openff-2.0.0-rc.2 openff-benchmark report match-minima \ --input-path 4-compute-qm-filtered \ --input-path 4-compute-mm-filtered/openff-2.0.0-rc.2 \ --ref-method b3lyp-d3bj \ --output-directory 5-match-minima-openff-2.0.0-rc.2

To export:

  1. Archive and compress only the new analysis report CSVs. Everything in this archive will become public domain, so please double check that the resulting archive does not include anything that should not be shared.

  2. Be sure to substitute your <group-id> and the date <YYYY.MM.DD> in the filename.

tar -czvf openff-benchmark-<group-id>-openff-2.0.0-rc.2-<YYYY.MM.DD>.tar.gz \ 5-compare-forcefields-openff-2.0.0-rc.2/*.csv \ 5-match-minima-openff-2.0.0-rc.2/*.csv

  1. Submit your compressed archive to the shared Google Drive folder: OpenFF Public Drop Zone


We aim to receive Sage RC2 results from the remaining partners as soon as possible.

Please let us know if you encounter any issues; we are grateful and happy to assist!

Cheers! Thank you to everyone for this extra effort over the course of this campaign!

The OpenFF Benchmark Team

2021.10.14-0

Distributed to all stakeholders on 2021.10.14 04:00 UTC

Greetings!

We have now received Sage RC2 results from 5/10 partners!

  • Xavier Lucas of Roche submitted his results on 8/17

  • Ian Craig of BASF submitted his results on 9/20

  • Lorenzo D’Amore of Janssen/OpenFF submitted his results on 9/21

  • Bai Xue of Xtal Pi submitted his results on 9/28

  • Katharina Meier of Bayer submitted her results on 10/8

Thanks to each of you for your submission! All results can be downloaded from the OpenFF Public Drop Zone.

As a key component for the manuscript being coordinated by Gary Tresadern of Janssen, we ask that you submit Sage RC2 results as soon as possible!

You can run the analysis workflow and export your shareable data only for Sage RC2 results to save time:

openff-benchmark report compare-forcefields \ --input-path 4-compute-qm-filtered \ --input-path 4-compute-mm-filtered/openff-2.0.0-rc.2 \ --ref-method b3lyp-d3bj \ --output-directory 5-compare-forcefields-openff-2.0.0-rc.2 openff-benchmark report match-minima \ --input-path 4-compute-qm-filtered \ --input-path 4-compute-mm-filtered/openff-2.0.0-rc.2 \ --ref-method b3lyp-d3bj \ --output-directory 5-match-minima-openff-2.0.0-rc.2

To export:

  1. Archive and compress only the new analysis report CSVs. Everything in this archive will become public domain, so please double check that the resulting archive does not include anything that should not be shared.

  2. Be sure to substitute your <group-id> and the date <YYYY.MM.DD> in the filename.

  1. Submit your compressed archive to the shared Google Drive folder:

OpenFF Public Drop Zone


We aim to receive Sage RC2 results as soon as possible. Please let us know if you encounter any issues; we are grateful and happy to assist!

Cheers! Thank you to everyone for this extra effort over the course of this campaign!

The OpenFF Benchmark Team2021.10.15-0

Distributed to all stakeholders on 2021.10.15 09:00 UTC

Greetings!

We have now received Sage RC2 results from 5/10 partners!

  • Xavier Lucas of Roche submitted his results on 8/17

  • Ian Craig of BASF submitted his results on 9/20

  • Lorenzo D’Amore of Janssen/OpenFF submitted his results on 9/21

  • Bai Xue of Xtal Pi submitted his results on 9/28

  • Katharina Meier of Bayer submitted her results on 10/8

Thanks to each of you for your submission! All results can be downloaded from the OpenFF Public Drop Zone.

As a key component for the manuscript being coordinated by Gary Tresadern of Janssen, we ask that you submit Sage RC2 results as soon as possible!

You can run the analysis workflow and export your shareable data only for Sage RC2 results to save time:

To export:

  1. Archive and compress only the new analysis report CSVs. Everything in this archive will become public domain, so please double check that the resulting archive does not include anything that should not be shared.

  2. Be sure to substitute your <group-id> and the date <YYYY.MM.DD> in the filename.

3. Submit your compressed archive to the shared Google Drive folder: OpenFF Public Drop Zone


We aim to receive Sage RC2 results as soon as possible. Please let us know if you encounter any issues; we are grateful and happy to assist!

Cheers! Thank you to everyone for this extra effort over the course of this campaign!

The OpenFF Benchmark Team

2021.09.16-0

Distributed to all stakeholders on 2021.09.20 14:00 UTC

Greetings!

Thanks again to each of you for submitting Season 1 results!

To continue pursuing the manuscript coordinated by Gary Tresadern of Janssen, we ask that you submit Sage RC2 results as soon as possible. We have received Sage RC2 from 2/10 partners, and more would help add weight to the manuscript substantially.

You can run the analysis workflow and export your shareable data only for Sage RC2 results to save time:

 

  1. Archive and compress only the new analysis report CSVs.
    Everything in this archive will become public domain, so please double check that the resulting archive does not include anything that should not be shared.

  2. Be sure to substitute your <group-id> and the date <YYYY.MM.DD> in the filename.

 

3. Submit your compressed archive to the shared Google Drive folder: OpenFF Public Drop Zone


We aim to receive Sage RC2 results within the next 2-3 weeks. Please let us know if you encounter any issues; we are grateful and happy to assist!

Cheers! Thank you to everyone for this extra effort over the course of this campaign!

The OpenFF Benchmark Team

2021.08.17-0

Distributed to all stakeholders on 2021.08.18 00:48 UTC

Greetings!

We have now received completed Season 1 results from all 10/10 partners!

  • Xavier Lucas of Roche submitted his results on 8/10.

Thanks to each of you for your submissions! All results can be downloaded from the OpenFF Public Drop Zone (Google Drive, requires a Google account (any one will do)).

Although we have collected sharable results from all partners, we encourage you to submit new versions of your results including Sage RC2 and/or OPLS (with permission from Schrodinger, if required). These results are of great interest for the manuscript being coordinated by Gary Tresadern of Janssen.

Should you decide to share results of Sage RC2 or OPLS execution, run the analysis workflow and export your sharable data as before:

  1. Archive and compress only the coverage report, analysis report CSVs, and your README. Everything in this archive will become public domain, so please double check that the resulting archive does not include anything that should not be shared.

  2. Be sure to substitute your <group-id> and the date <YYYY.MM.DD> in the filename.

3. Submit your compressed archive to the shared Google Drive folder:  OpenFF Public Drop Zone

Cheers! Thank you to everyone for all your efforts over the course of this campaign!

The OpenFF Benchmark Team

2021.08.04-0

Distributed to all stakeholders on 2021.08.04 16:10 Los Angeles

Greetings!

We have released version 2021.08.04.0 of openff-benchmark!

Partners that wish to perform geometry optimizations using Sage RC2 (openff-2.0.0-rc.2) can do so after deploying this release into their environment. To execute geometry optimizations with Sage RC2, follow this protocol: Sage RC2 Benchmarking Protocol - Season 1.

Thank you for your patience with the Sage release candidate process. The force field team is reviewing other benchmarks and doing pre-release work, and we intend to have the full Sage release out soon.

This openff-benchmark release also includes OPLS execution. See our protocol for running OPLS evaluations if you are interested in including these in your benchmarking. We have tested this at a few sites, but please let us know if you experience issues. You are encouraged to run this on your dataset, however note that any information you share with OpenFF becomes open under the BSD-3 license, so you may need to get permission from Schrödinger before sharing your results publicly.

Should you decide to share results of Sage RC2 or OPLS execution, run the analysis workflow and export your sharable data as before:

  1. Archive and compress only the coverage report, analysis report CSVs, and your README. Everything in this archive will become public domain, so please double check that the resulting archive does not include anything that should not be shared.

  2. Be sure to substitute your <group-id> and the date <YYYY.MM.DD> in the filename.

3. Submit your compressed archive to the shared Google Drive folder:  OpenFF Public Drop Zone

We have also now received completed Season 1 results from 9/10 partners!

  • Arjun Narayanan of Vertex submitted his results on 7/14.

Apologies for the delayed announcement, and thanks to each of you for your submissions! All results can be downloaded from the OpenFF Public Drop Zone (Google Drive, requires a Google account (any one will do)).

Thank you for all your efforts over the past several months! We look forward to your feedback!

The OpenFF Benchmark Team

2021.07.23-0

Distributed to all stakeholders on 2021.07.23 12:48 Los Angeles

Greetings!

At our last call, we announced plans to release a new version of the openff-benchmark package, which adds the ability to assess our first Sage release candidate. However, during our pre-release testing, we observed a higher error rate for MM optimizations compared to previous releases. We’ve tracked it down to a numerical instability caused by a particular angle parameter in Sage RC1 which causes MM optimizations to fail to converge [1]. We’ve met with the force field team, and they will incorporate our proposed fix into Sage RC2.

To avoid wasted effort from our partners, we have delayed the openff-benchmark release, as this bug will cause a specific chemistry to error and get dropped from the dataset during optimizations. The force field team is reviewing other benchmarks of Sage RC1 that may lead to other possible changes to RC2. As soon as RC2 is available, we will make the new openff-benchmark release and contact you with instructions to update.

The OpenFF Benchmark Team

 

[1] In Sage RC1, valence angle parameter a16, which is assigned to some linear carbons, was given an equilibrium value of 183.8 degrees. This is geometrically impossible, and causes the minimizations to fail to converge. Like a21 in the same file, this angle should have been pegged to 180.0 degrees.

2021.06.18-0

Distributed to all stakeholders on 2021.06.18 23:20 UTC

Happy Friday!

We have now received completed Season 1 results from 8/10 partners!

  • Bai Xue of XtalPi submitted his results on 6/9.

  • Katharina Meier of Bayer submitted her results on 6/15.

Thanks to each of you for your submissions! All results can be downloaded from the OpenFF Public Drop Zone (Google Drive, requires a Google account (any one will do)).

The next partner call is scheduled for 7/8! We look forward to seeing all of you there.

The OpenFF Benchmark Team

2021.06.03-0

Distributed to all stakeholders on 2021.06.04 00:04 UTC

Greetings!

We have now received completed Season 1 results from 6/10 partners!

  • David Hahn, Lorenzo D’Amore, and Gary Tresadern of Janssen submitted their results on 5/26.

Thanks to each of you for this submission! All results can be downloaded from the OpenFF Public Drop Zone (Google Drive, requires a Google account (any one will do)).

If your results have completed, please do follow the export and submission instructions sent previously. If you are still computing results, please do let us know! We are happy to assist with any compute or export issues you are encountering, and are eager to complete the Season 1 benchmark with you!

We are scheduling the next partner call, so please try to get your submissions in soon!

The OpenFF Benchmark Team

2021.05.19-0

Distributed to all stakeholders on 2021.05.20 01:54 UTC

Greetings!

We have now received completed Season 1 results from 5/10 partners!

  • Thomas Fox of Boehringer-Ingelheim submitted his results on 5/11.

  • Kaushik Lakkaraju of BMS submitted his results on 5/19.

Thanks to each of you for your submissions! All results can be downloaded from the OpenFF Public Drop Zone (Google Drive, requires a Google account (any one will do)).

If your results have completed, please do follow the export and submission instructions sent previously. If you are still computing results, please do let us know! We are happy to assist with any compute or export issues you are encountering, and are eager to complete the Season 1 benchmark with you!

Once more, thanks for the time and energy you have put into this collaborative effort!

The OpenFF Benchmark Team

2021.05.07-0

Distributed to all stakeholders on 2021.05.07 23:18 UTC

Happy Friday!

We have now received completed Season 1 results from 3/10 partners!

  • Ian Craig and Pieter in’t Veld of BASF submitted their results on 5/6.

Thank you for your submission! All results can be downloaded from the OpenFF Public Drop Zone (Google Drive, requires a Google account (any one will do)).

We also would like to welcome our 11th partner joining the Season 1 benchmark: Charles Hutchins and Nour Saleh of AbbVie have begun the process of applying the benchmark to their own internal molecules. Due to a later start we expect their results may not make it in time for inclusion in the publication for the Season 1 benchmark, but we are pleased to have them aboard and evaluating our forcefields on their infrastructure.

If your results have completed, please do follow the export and submission instructions sent previously. If you are still computing results, please do let us know! We are happy to assist with any compute or export issues you are encountering, and are eager to complete the Season 1 benchmark with you!

Once more, thank you for all your efforts!

The OpenFF Benchmark Team

2021.04.29-0

Distributed to all stakeholders on 2021.04.30 03:37 UTC

Happy Friday!

We have so far received completed Season 1 results from 2/10 partners.

  • Bill Swope and Alberto Gobbi of Genentech submitted their results on 4/20.

  • Christina Schindler of Merck submitted her results on 4/22.

Thank you to each of your for your submissions!

All results can be downloaded from the OpenFF Public Drop Zone (Google Drive).

If your results have completed, please do follow the export and submission instructions sent previously. If you are still computing results, please do let us know! We are happy to assist with any compute or export issues you are encountering, and are eager to complete the Season 1 benchmark with you!

Thank you all for your efforts!

The OpenFF Benchmark Team

2021.04.09-0

Distributed to all stakeholders on 2021.04.09 21:35 UTC

Happy Friday!

Partners that have completed their production runs can now export and submit their publicly-shareable results!

Before you follow the export instructions below, please upgrade your deployed software environment following the instructions in the Deployment Procedure (see the Software Updates section for the installation approach you have chosen).

  1. Delete any existing result directories: rm -r 5-compare_forcefields 5-match_minima

  2. Filter problematic molecules from your compute results using protocol step “4-2” (newly added between steps 4 and 5). Then rerun the remainder of the protocol (steps 5+) to generate new analysis results.

  3. Provide any notes, such as your selection criteria, in a README.md at your own discretion. These notes will become public domain and care should be taken that they are not sufficient to deduce the identity of individual molecules in your set.

  4. Archive and compress only the coverage report, analysis report CSVs, and your README. Everything in this archive will become public domain, so please double check that the resulting archive does not include anything that should not be shared.

    • Be sure to substitute your <group-id> and the date <YYYY.MM.DD> in the filename.

  5. Submit your compressed archive to the shared Google Drive folder:  OpenFF Public Drop Zone

In case you are unable to upload your tarball to the above folder, we can upload it for you. Please either:

  1. Share your results tarball with us on #benchmarks-partners.

  2. Email us the tarball. Send to david.dotson@openforcefield.org, CC: jeffrey.wagner@openforcefield.org.

Thank you for all your efforts in making Season 1 a remarkable and productive collaborative effort!

The OpenFF Benchmark Team

2021.03.25-0

Distributed to all stakeholders on 2021.03.25 22:50 UTC

Greetings!

Thank you to everyone who could attend yesterday’s partner call, and thank you to Gary for organizing.

The slides from the call can be viewed here: Geometry Optimization Benchmarking for Industry Partners - 2021.03.24

The discussion notes can be viewed on Confluence here: 2021-03-24 Industry Evaluation Meeting notes

Major points of interest for those who could not attend:

  1. 6/10 partners have completed their production sets. If you are still working on yours, there is still time, so do keep going and let us know when you’ve finished!

  2. The slides include instructions on export and submission of publicly sharable data. However, in discussion during the call it was determined that we need a way for e.g. iodine-containing molecules to be filtered out of the report results, since for the QM calculations they are known to be erroneous. We will issue instruction in the coming weeks when we have adjusted the tail end of the protocol to accommodate this.

  3. David Hahn is working on a command tree for running Schrodinger optimizations with OPLS3: openff-benchmark#63. David is hosting a demo/testing call on Friday, 3/26, at 8am PT / 11am ET / 4pm CET: LINK

  4. David Dotson is working on a command tree for running torsiondrives: openff-benchmark#69

  5. We collected additional feature requests from partners based on their experience with the openff-benchmark tooling so far. See the notes for details.

  6. We collected desired characteristics for a future Season 2. See the notes for details.

Thanks again to all of you for the lively and quality discussion. We look forward to finishing out Season 1 with you, and continuing to advance the larger OpenFF effort with you.

The OpenFF Benchmark Team

2021.03.17-1

Draft, not distributed.

Greetings!

We have cut a new release of openff-benchmark: 2021.03.17.0

This features the following improvements to the codebase:

  • QC molecule --> MM molecule exporting now detects when molecule connectivity has changed during QM geometry optimization (e.g. proton transfer). Instances of connectivity rearrangement now are filtered to the error_mols subdirectory of that step's output.

  • Export failures of several varieties are now handled gracefully and routed to the error_mols directory.

  • openff-benchmark optimize execute now routes errored cases into an error_mols subdirectory.

  • Updated to use the 0.8.4 openforcefield toolkit package.

For most of you these changes will have no impact on your use of the protocol; these improvements are targeted to exceptional, somewhat rare cases.

At your convenience, please update the software using either conda or the single-file installer. If you are currently running optimizations, you may allow these to complete before updating, since removal and re-installation of the software would be disruptive to those calculations.

Please let us know if you experience any issues!

The OpenFF Benchmark Team

2021.03.17-0

Distributed to all stakeholders on 2021.03.17 18:00 UTC

Greetings!

Since kicking off production runs on Friday 2/5, two partners have indicated completion of their chosen production set. We would like to hear from each of you on how your set is coming along:

  1. Have you completed your production set?

  2. If not complete, can you estimate when it will likely be complete based on performance so far?

Please do let us know how it’s going in #benchmarks-support, as well as of any issues you’ve encountered in the process. We have also made a bugfix release of openff-benchmark (2021.03.17.0) that may address some rare issues in QM export; if you are encountering difficulties, consider updating your deployed software to this release.

We look forward to meeting with you all on 3/24!

The OpenFF Benchmark Team

2021.02.12

Distributed to all stakeholders on 2021.02.12 18:14 UTC

Hello everyone!

Since kicking off production runs on Friday 2/5, we would like to know how it’s going for each of you. Please let us know:

  1. Have you completed the burn-in? We know many of you have, but please confirm.

  2. Have you started production?

    1. If so, how many molecules, and how many total conformers, are you executing?

    2. If so, based on completion rates so far, do you have an estimated completion date of the QM calculations?

  3. Has your production run already completed?

Thank you all for your participation! We look forward to hearing from each of you!

The OpenFF Benchmark Team

2021.02.05-1

Distributed to all stakeholders on 2021.02.05 20:43 UTC

Greetings!

See the table below for the three-letter company/group ID to use for your production benchmark run. You will enter this three-letter ID at the validation step of the Optimization Benchmarking Protocol - Season 1.

This code will be part of the ID for all molecule conformers that pass through the workflow, giving each a unique, non-proprietary ID that can be shared and combined with result data from other partners.

Partner

Company/Group ID

BASF

BSF

Merck

MRK

Genentech

GNT

Boehringer Ingelheim

BRI

Vertex

VTX

Roche

RCH

Bayer

BYR

Janssen

JNS

XTALPI

XTP

Bristol Myers Squibb

BMS

Thank you!

The OpenFF Benchmark Team

2021.02.05-0

Distributed to all stakeholders on 2021.02.05 20:43 UTC

Greetings!

First, we want to thank all partners for their patience. We have been working behind the scenes to improve the software underlying the protocol based on your feedback from the burn-in. Thank you for your responses!

Our latest release incorporating this feedback has just been made, and we ask all partners to upgrade their deployed software environment following the instructions in the Deployment Procedure (see the Software Updates section for the installation approach you have chosen).

Items of note included in this update:

  1. The heavy-atom RMS cutoff for input conformer validation deduplication has changed from 0.1 A to 0.2 A; this will reduce the chance of nearly identical conformers being accepted by the beginning of the workflow, but still allow small molecules with multiple conformers of interest to make it through.

  2. The heavy-atom RMS cutoff for conformer generation has changed from 1.0 A to 0.5 A. This should ensure that sufficient numbers of conformers are generated for more robust downstream analysis, improving the value of the benchmark results.

  3. The issue in which the connectivity table was mangled in exported SDF files from compute options C and E has been fixed.

  4. The coverage reporting step has been added to the protocol. This is now used as the input for the optimization step.

For partners that have completed the burn-in, please execute the full Optimization Benchmarking Protocol - Season 1 on your production datasets. Please review the points below carefully before beginning:

  • You will use your assigned company/group ID instead of the TST ID used in the burn-in; these will be given in a separate announcement.

  • Please remember to limit your dataset to no more than 1,000 unique molecules.

  • You are encouraged to generate conformers of interest (up to 10) for each molecule with your tool of choice, and these will be preserved by the workflow.

  • The conformer-generation stage of the workflow will attempt to generate conformers beyond these choices, up to 10 conformers total for a given molecule.

  • All input molecules must have hydrogens/protonation explicitly specified.

  • The average molecule in the burn-in set had roughly 10 heavy atoms. Based on experience, we expect the time-cost of QM calculations to scale as n_heavy_atoms ^ 4. So if your set generates about the same number of conformers as the burn-in, and the average molecule in your set has 20 heavy atoms, expect it to take (20/10) ^ 4, or 16x the CPU-hours of the burn-in.

  • We are aiming to compile initial results from the benchmark in ~60 days. Please consider this and the calculation above for your chosen molecules before proceeding with your production run.

For partners that have yet to complete the burn-in, please let us know how we can help! As a reminder, you should follow the instructions below and let us know how it goes, as well as when you are finished. As part of this process you will choose an optimization approach based on your preferences and the constraints of your infrastructure.

  1. Deploy the software stack: Deployment Procedure

  2. Download the "burn-in" dataset (attached); a big thanks to Bill Swope for assembling these molecules for us.

  3. Follow the benchmarking protocol for season 1, using the burn-in dataset as the input: Optimization Benchmarking Protocol - Season 1

  4. Report any issues you encounter with deployment or the protocol on the #benchmarks-support Slack channel.

Please let us know if you have any questions in #benchmarks-support; we are happy to assist in any way we can to make this a smooth experience for everyone involved.

Thank you!

The OpenFF Benchmark Team

2021.01.30

Distributed to all stakeholders on 2021.01.30 17:36 UTC

Hello!

We hope all of you have had a chance to try out the Deployment Procedure and get started with the Optimization Protocol using the burn-in set. If not, please let do give it a go as soon as you can!

So we can help you most effectively, please send us a response to each of the following:

  1. Were you able to deploy the software using the Deployment Procedure?

  2. Were you able to validate the molecules and generate conformers from them using the Optimization Protocol (steps 1 and 2)?

  3. Were you able to begin (or finish) QM optimizations (step 4)? If so, which optimization execution approach did you ultimately use (A,B,C,D, or E)?

  4. Are you completely finished with the burn-in? Have you completed all QM and MM calculations, with report generation (step 5)?

We are happy to help with any problems you encounter, so please do let us know how it’s going in #benchmarks-support.

Currently, there is discussion about whether to segment datasets by molecule size/flexibility in the analysis. This could improve the analysis of the results, however it may be possible to identify the molecules depending on the details of the approach. We would like to resolve this discussion before starting up production runs for all partners.

Please await instruction before beginning your production sets, as we will also be pushing a new release of openff-benchmark and ask you to upgrade your deployed environment before you begin.

Thank you!

The OpenFF Benchmark Team

2021.01.26

Distributed to all stakeholders on 2021.01.26 2:00 UTC

Greetings!

As detailed in the call on  1/22, we are officially kicking off OpenFF Optimization Benchmark - Season 1!

Here are the instructions for how to begin the "burn-in" testing phase; you will need to deploy and execute on Linux host(s):

  1. Review our slides from 1/22, detailing the approach. Please feel free to ask clarification questions on the #benchmarks-partners Slack channel.

  2. Deploy the software stack: Deployment Procedure

  3. Download the "burn-in" dataset (attached); a big thanks to Bill Swope for assembling these molecules for us.

  4. Follow the benchmarking protocol for season 1, using the burn-in dataset as the input: Optimization Benchmarking Protocol - Season 1

  5. Report any issues you encounter with deployment or the protocol on the #benchmarks-support Slack channel.

We would like for all partners to begin the "burn-in" this week (starting 1/26) if possible. This exercise will allow us to help you select the most appropriate compute approach for your cluster environment. We expect that configuring distributed compute will take some time (step 4 above), as you will need to choose an approach best-suited to your compute resources.

This week, the only goal is to ensure the protocol is working as we expect on your infrastructure. On 2/1, we will make a new software release that you can use to begin processing your internal molecules. We anticipate making changes to the software based on issues we encounter during the burn-in, which may affect the outputs of various workflow steps.

To allow us to help you most effectively, as well as to advance the software to meet the needs of the benchmark, we ask that you hold off on executing the workflow on your internal molecules until you have completed the burn-in and we have made the 2/1 software release.

Thank you all; we're looking forward to getting started with you!

The OpenFF Benchmark Team