Sage differences between FB 1.9.3, FB 1.9.5, and 1.9.6

Background

As our software stack updates, our goal is to update from the previous software stack for fitting (ForceBalance 1.9.3 + OpenFF Toolkit 0.10.x) to versions that support Interchange (ForceBalance 1.9.5 + OpenFF 0.13.x, to begin with). This page documents differences between fits using these two stack and diagnoses the causes. Please see Reproducibility for more information.

All files were uploaded to . As this is not intended to be a public work, however, documentation is minimal.

Summary

I attempted to figure out the reasons for all differences between FB 1.9.3, FB 1.9.5, and FB 1.9.6. While I refer to these environments under those names throughout this investigation, they are shorthand for the environments described in the section below, and most differences originate outside ForceBalance. These differences can be summed up below:

• The biggest and most influential change was Difference 1: a bug where Interchange’s charge caching was mis-applying charges to subsequent conformers of the same molecule when the atom order changed.

• In Difference 2, the geometries optimize to the same or very similar structures, but the objective functions are very different. This suggests that we could improve the ForceBalance target by using the best RMSD measure instead of just RMSD.

• In Difference 3, re-ordering the forces in OpenMM systems can result in surprisingly large changes in geometry. From the OpenFF toolkit 0.10.x to Interchange we changed the order in which forces are output.

• In Difference 4, differences in precision explain the remaining differences between the FB 1.9.3 and FB 1.9.5 stacks.

• In Difference 5, all remaining differences between 1.9.5 and 1.9.6 can be explained by a minor change in tolerance in the ForceBalance geometry optimization.

Investigation

This next section below is not comprehensive in detail, and is partly a working document I used to keep track of what other differences to look at. This is meant to be largely a one-off investigation, although it suggests that having a suite of fitting comparisons for each new software stack may be a good idea. Likely this will be most use in the future in copying Python scripts and looking at individual targets for an idea of what their objective terms used to be.

Details

The inputs used are those available at .

The environments used are specified in the attached input files. For each zipped experiment, the full environment is included.

Environments:

• fb-193: fb-193-tk-010-oe-2022 environment

  • ForceBalance 1.9.3

  • OpenFF Toolkit 0.10.4

• fb-193-reordered: fb-193-tk-010-oe-2022-reordered environment

  • The Toolkit code was edited to output forces in the same order as Interchange

  • create_openmm_systems was edited here:

  • return_topology = kwargs.pop("return_topology", False) # 2023-11-22 edit original = self._parameter_handlers order = [ 'vdW', 'Electrostatics', 'LibraryCharges', 'ToolkitAM1BCC', 'ProperTorsions', 'ImproperTorsions', 'Angles', 'Bonds', 'Constraints' ] self._parameter_handlers = {} for key in order: self._parameter_handlers[key] = original[key] for key in original: if key not in order: self._parameter_handlers[key] = original[key] # end 2023-11-22 # Make a deep copy of the topology so we don't accidentally modify it topology = copy.deepcopy(topology)

• fb-195: fb-195-tk-013-oe-2022-interchange-replace-cache environment

  • Interchange 0.3.10, edited:

    • to modify the charge caching

    • to turn off the switching function, as this was a systematic difference between environments ( )

• fb-195-switching: basically the fb-195-tk-013-oe-2022-interchange-replace-cache environment

  • Interchange 0.3.10, edited:

    • to modify the charge caching

    • to turn switching back on

• fb-196: fb-196-ic-0318-oe-2022-crit environment

  • ForceBalance 1.9.6

  • Interchange 0.3.18

• fb-196-crit: fb-196-ic-0318-oe-2022-crit environment

  • ForceBalance 1.9.6, with crit modified back to earlier 1e-4 version.

  • Everything else the same as fb-196

ForceBalance 1.9.3 vs 1.9.5

Difference 1: Switching to Interchange with cached atom charges

The vast majority of differences originated from the opt-geo-target fits and were because of this bug:

Running a single-point calculation of the entire target set brought the objective function differences down substantially:

Remaining differences after Difference 1

Investigating individual targets highlighted these differences, still all geometry optimization targets:

(also attached: , where single_point_all compares all targets, and single_point_differences highlights all differences over 1. These CSVs also include the associated SMILES which were too long to paste here).

Pasted below is the RMSD comparisons of all targets with different objective functions. The first three values columns show atom-to-atom RMSD (via MDAnalysis’s rmsd); the last three atoms show symmetry-corrected RMSDs (via rdkit's AllChem.GetBestRMS).

Difference 2: Different interpretations of internal coordinate RMSDs

The yellow rows highlighted apparently optimize to structures with symmetry-corrected RMSDs of 0, but with atom-wise RMSD values that are somewhat high. I did not look into this further as the objectives weren’t systematically better or worse. However, this suggests that the FB target should do its best to look for the best symmetry-corrected RMSD and do its best.

Difference 3: Section ordering

In several cases, re-ordering the output of the XML changed the optimization and objective function substantially. These are highlighted in blue above.

Case study: opt-geo-batch-113: 19095588-8

Open

• Wrote intermediate OpenMM XMLs out of the OpenMM system (by modifying ForceBalance to do so)

• Running through the ForceBalance target, the OpenMM XML written out by FB 1.9.3 minimizes to the 1.9.3 structure, in the 1.9.5 environment; the XML written by FB 1.9.5/Interchange minimizes to the 1.9.5 structure, in the 1.9.3 environment

• In Interchange, the forces are ordered: NBForce, Torsions, Angles, Bonds

• In 1.9.3, the forces are ordered: Torsions, NBForce, Bonds, Angles

• I can get the 1.9.3 environment to reproduce the 1.9.5 objective by reordering the OpenMM system creation using the Interchange order, i.e. using the fb-193-reordered environment to write forces out like Interchange.

Comparing remaining differences with reordering

I have highlighted in gray the differences between these four systems where the more modern stacks (FB 1.9.5, 1.9.6) align with either fb-193 or fb-193-reordered. As the order of the forces is the only difference between the 193 stacks, to me this implies that these systems are prone to different optimization pathways given the slightest difference, so they weren’t investigated further.

Determining thresholds for investigation

It is common for structures to optimize to very slightly different geometries, especially due to differences in precision. I found the following trends comparing the 1.9.3 environment with 1.9.5:

ForceBalance 1.9.5 remaining differences

Difference 4: differences in precision

This difference, as detailed in the case study below, originates from minor errors in precision between the systems. Every difference in the above remaining table is explained by this difference. An example of the differences is on GitHub at . This file compares the case study system below (opt-geo-batch-20, 18434815-6) generated from fb-193-reordered and compared to fb-196. (fb-195 and fb-196 generate identical systems; in the 1.9.6 diff, however, I had modified the switching function manually to be turned off for another experiment).

Case study: opt-geo-batch-20, 18434815-6

Open

• The main differences between the 1.9.3 and 1.9.5 systems here are again at the level of precision

• Minimizing the 1.9.5 system in the 1.9.3 environments, minimized to the 1.9.5 geometry

• Minimizing the 1.9.3 systems in the 1.9.5 environment, minimized to the 1.9.3 geometry

Broad diagnosis:

• I copied the ForceBalance opt-geo target to run-through one-by-one, and minimized each system (as output by an edited version of ForceBalance) in each environment.

Scripts:

• Python:

• Submission scripts:

• Outputs:

ForceBalance 1.9.6 remaining differences

Difference 5: optimization energy tolerance

In ForceBalance 1.9.6, the energy tolerance in geometry optimization was tweaked from a default of 1e-4 to 1e-2 :

Reverting this change (in fb-196-crit) gave back all the 1.9.5 results.

Case study: opt-geo-batch-58, 18438954-4

• Optimizing the 1.9.5 system in 1.9.6 (with 1.9.5 energy tolerance) re-optimized to the 1.9.5 geometry

• Optimizing the 1.9.5 system in 1.9.6 (with 1.9.6 energy tolerance) re-optimized to the 1.9.6 geometry

• Optimizing the 1.9.6 system in 1.9.5 (with 1.9.5 energy tolerance) re-optimized to the 1.9.5 geometry

• Optimizing the 1.9.6 system in 1.9.5 (with 1.9.6 energy tolerance) re-optimized to the 1.9.6 geometry

• This change covered all cases in the 1.9.6 table above.