Here we compare the benchmarks of Sage 2.1.0 (unconstrained) between YAMMBS and the original benchmarking script used during the development of Sage 2.1.0.
The original benchmarks of Sage 2.1.0 were taken from here. These benchmarks use a geometry optimization convergence threshold of 5e-9 kJ/mol/nm^2. ddE is defined with respect to the lowest energy QM conformer, and ddE’s between the lowest energy conformers are excluded as they are definitionally zero.
The YAMMBS benchmarks were calculated as follows. The MoleculeStore object was generated with this commit of the YAMMBS package, which optimizes the molecules using a convergence threshold of 5e-9 kJ/mol/nm^2. The ddE’s, TFD’s, and ICRMSD’s were calculated using this commit of the YAMMBS package, which has revised the ddE definition to be consistent with the old benchmarking script (defined with respect to the lowest energy QM conformer, and ddE’s between the lowest energy conformers are excluded) and uses mapped SMILES to create molecules for TFD analysis. The RMSD’s have been calculated using the same commit as the ddE and TFD, but with a modified RMSD function that uses all atoms to calculate the RMSD, in order to be consistent with the RMSD definition in the old benchmarks. YAMMBS typically uses a heavy-atom-only RMSD, which is much faster to compute.
Molecules identified as “problematic” were removed--inverted 7-membered rings with bad QM structures, and molecules containing S with 4 single bonds that are not covered correctly.
The code and files used in this are available here
DDE
The overall DDE’s are nearly identical between YAMMBS and the old benchmark (“og” in the figures).
However, examining individual records reveals a number of cases where the ddE’s are very different. There are 563 records with the ddE difference > 5 kcal/mol
RMSD
When comparing the all-atom RMSD’s between the two codes, the overall RMSD benchmarks are nearly identical.
However, there are still large deviations when examining individual records. There are 219 records with the difference in RMSD > 0.4 A.
TFD
The overall TFD metric is nearly the same between the two sets of scripts.
However, there are a few records with large deviations in TFD between scripts. There are 55 records with a TFD difference > 0.1.
ICRMSD
Bonds
Actually, the bond RMSE is pretty consistent on an individual-record basis, with the exception of the one outlier molecule identified above.
Angles
The record-by-record comparison for angle RMSE is also relatively good, though 18 records have an angle RMSE difference larger than 1 degree.
Dihedrals
The dihedral angles show large differences when comparing individual records. There are 229 records with a difference in dihedral angle RMSE of > 10 deg
Impropers
The overall behavior of the impropers is concerning, with yammbs showing significantly worse results. I couldn’t find a reason in the code for this. Lily suggested it could be a difference in the number of times the impropers are applied during evaluation?
Same situation for impropers; 301 records have a difference of > 10 deg.
Comparing individual records with large deviations
As seen above, many individual records have large deviations between the two codes for each metric. One might expect if a given record has bad agreement in RMSD, it would also have bad agreement in other metrics, and vice versa. However, there isn’t a lot of overlap between the records that have large deviations in different metrics. Below are the overlaps between different sets. For example, “overlap DDE/RMSD” is reporting the number of records that both have a large discrepency in DDE and a large discrepency in RMSD between the two codes.
DDE big dif 563 RMSD big dif 219 TFD big dif 55 Bond big dif 5 Angle big dif 18 Dihedral big dif 229 Improper big dif 301 Overlap DDE/RMSD 36 Overlap DDE/TFD 10 Overlap RMSD/TFD 28 Overlap DDE/bond 0 Overlap DDE/angle 0 Overlap DDE/proper 43 Overlap DDE/improper 3 Overlap RMSD/bond 0 Overlap RMSD/angle 0 Overlap RMSD/proper 106 Overlap RMSD/improper 2 Overlap TFD/bond 0 Overlap TFD/angle 0 Overlap TFD/proper 22 Overlap TFD/improper 1