Here we compare the benchmarks of Sage 2.1.0 (unconstrained) between YAMMBS and the original benchmarking script used during the development of Sage 2.1.0.
The original benchmarks of Sage 2.1.0 were taken from here. These benchmarks use a geometry optimization convergence threshold of 5e-9 kJ/mol/nm^2. ddE is defined with respect to the lowest energy QM conformer, and ddE’s between the lowest energy conformers are excluded as they are definitionally zero.
The YAMMBS benchmarks were calculated as follows. The MoleculeStore object was generated with this commit of the YAMMBS package, which optimizes the molecules using a convergence threshold of 5e-9 kJ/mol/nm^2. The ddE’s, TFD’s, and ICRMSD’s were calculated using this commit of the YAMMBS package, which has revised the ddE definition to be consistent with the old benchmarking script (defined with respect to the lowest energy QM conformer, and ddE’s between the lowest energy conformers are excluded) and uses mapped SMILES to create molecules for TFD analysis. The RMSD’s have been calculated using the same commit as the ddE and TFD, but with a modified RMSD function that uses all atoms to calculate the RMSD, in order to be consistent with the RMSD definition in the old benchmarks. YAMMBS typically uses a heavy-atom-only RMSD, which is much faster to compute.
DDE
The overall DDE’s are nearly identical between YAMMBS and the old benchmark (“og” in the figures).
However, examining individual records reveals a number of cases where the ddE’s are very different. There are 564 records with the ddE difference > 5 kcal/mol
RMSD
When comparing the all-atom RMSD’s between the two codes, the overall RMSD benchmarks are nearly identical.
However, there are still large deviations when examining individual records. There are 222 records with the difference in RMSD > 0.4 A.
TFD
The overall TFD metric is nearly the same between the two sets of scripts.
However, there are a few records with large deviations in TFD between scripts. There are 56 records with a TFD difference > 0.1.
ICRMSD
Bonds
Actually, the bond RMSE is pretty consistent on an individual-record basis, with the exception of the one outlier molecule identified above.
Angles
The record-by-record comparison for angle RMSE is also relatively good, though 82 records have an angle RMSE difference larger than 1 degree.
Dihedrals
The dihedral angles show large differences when comparing individual records. There are 239 records with a difference in dihedral angle RMSE of > 10 deg
Impropers
The overall behavior of the impropers is concerning, with yammbs showing significantly worse results. I couldn’t find a reason in the code for this.
Same situation for impropers; 332 records have a difference of > 10 deg.
Comparing individual records with large deviations
As seen above, many individual records have large deviations between the two codes for each metric. One might expect if a given record has bad agreement in RMSD, it would also have bad agreement in other metrics, and vice versa. However, there isn’t a lot of overlap between the records that have large deviations in different metrics. Below are the overlaps between different sets. For example, “overlap DDE/RMSD” is reporting the number of records that both have a large discrepency in DDE and a large discrepency in RMSD between the two codes.
Overlap DDE/RMSD 36 Overlap DDE/TFD 10 Overlap RMSD/TFD 29 Overlap DDE/bond 0 Overlap DDE/angle 0 Overlap DDE/proper 44 Overlap DDE/improper 4 Overlap RMSD/bond 0 Overlap RMSD/angle 1 Overlap RMSD/proper 106 Overlap RMSD/improper 5 Overlap TFD/bond 0 Overlap TFD/angle 0 Overlap TFD/proper 22 Overlap TFD/improper 2