Page Comparison

• Trello review

  • https://trello.com/c/iUi4V7P0/630-look-into-tfd-discrepancies

    • MT – Didn’t dig deeply into this specifically, results indicated that nothing bad is happening with conformer positions or torsion fingerprints. So I’m considering this complete.

  • https://trello.com/c/FyJIW9di/629-construct-and-copy-intermediate-databases-for-debugging

    • MT – Kinda messy the way I’m doing it now, but this can always be changed.

  • https://trello.com/c/UJhwf0mM/614-investigate-bad-minimized-energies

    • ditto point 1

  • https://trello.com/c/3xKW7QI8/631-update-development-environment

    • MT – Done, not fully up to date but using newer stuff. Will test again once a new qcsubmit is out but the old version works fine.

  • https://trello.com/c/Y8ernXOm/616-compare-results-to-existing-pavan-simon-scripts

    • MT – No concrete progress here, but it does seem close. Since I didn’t do a quantitative comparison I’ve put it inot the backlog.

    • LW – I have some runs in progress comparing to SB+PB’s scripts. And BW reported the stereo issues.

    • BW – Looks like everything I raised in Slack has been fixed. I ran on the training optimization data for torsion multiplicies (for sage 2.1.0) (“filtered sage 2.1 optimization training set” + additional molecules that PB picked out and filtered down). This was a test of running through the new process using a custom FF and new dataset.

      • BW (chat) – In case you want more links, here's the torsion multiplicity page: Torsion multiplicity

      • BW (chat) – And this is the data set I ran it on: https://github.com/ntBre/valence-fitting/blob/main/02_curate-data/datasets/filtered-opt.json

    • MT – LW, what would completion of this ticket entail?

    • LW – Ideally a quantitative dataset result comparison, so we can see the difference and judge the accuracy. If they don’t quantitatively match up, we can qualitatively judge as well.

    • MT – LW, you’re running the comparison between the two pipelines now on a dataset?

    • LW – Yes

    • MT – I’ll assume you’ll follow through on that and we can see if there are major issues in the results.

    • LW – Ok, can I pass on raw results to you for comparison?

    • MT – Yup, that’d be fine.

    • JW – This sounds great - During the industry benchmarking project we kept finding that we’d do like heavy atom RMSD in one part and all-atom RMSD in another, so this should help us catch where things are set differently.

• Results from larger (~1600 conformers) dataset

  • https://openforcefieldgroup.slack.com/archives/C03T3LLVC1J/p1691756344187349

• MT – Future tickets/what should I work on next? Could defer until after we’ve checked accuracy of current stuff. My thinking right now is mostly “tidying up”/”making a CLI”

• LW – Looking at project page, it looks like next stage would be phys prop calcs?

  • MT – Big remaining step for conformer analysis is MM minimizations.

  • LW – I think BW and I are in a good place to test the accuracy here. Is there parallelization?

  • MT – That makes sense. Then the next thing I should work on is roadmap item 2, the evaluator interface. Roadmap item 3 (support for custom parameterhandlers) is either easy or already done. I don’t actually know how Simon used evaluator to run these benchmarks.

  • LW – I’ll see if I have any SB scripts around. He had a codebase with scripts from the benchmarking.

• JW – Should we focus on making the workflow more resilient/making more intermediate checkpoints?

  • LW – That’d be nice, but it might be a lot of work.

  • MT – That could be tricky. A lot of looking at the database and judging whether there are conformers/results where they’re expected. If a run dies midway though, the confs that were already minimized will be in the database. So you should be able to hook back into the databsae at that point and resume - So there’s a decent sense in which the database is a natural checkpoint system.

  • LW – Is there any way to tell when an error has occurred? Like, if we’re going through and there’s a molecule with 3 confs but the computation ended because a worker crashed, how could we handle that?

  • MT – There are a few options, I’d need to think about it. Right now there’s no elegant system to handle that. Right now there’s a bare minimum of logging - so info about what crashed and why isn’t handled super well.

  • LW – Ok, sounds like this would be helpful, but we can come back to it if ther’es not bandwidth now

  • JW – The QC stack’s quite-successful error policy is “if it fails 3 times, stop trying”. This can be overridden by user intervention via python.

• JW –

  • Nagl charges? Could be faster than AM1BCC.

    • LW – As an update - I’ve made the 0.3.0 release that’s broken a few things.

    • MT – Gotcha, I handled it.

    • MT – One thing about nagl - When Nagl 0.X comes out with nagl-models 0.Y, and we’re ready to benchmark it, how will this be expected to fit into the pipeline?

    • LW – We’ve discussed that Nagl will live as another backend that provides AM1BCC charges

    • JW – This can be handled in the forcefield - When we want to run a nagl model that isn’t production-ready, we’ll call it through the ChargeIncrementModelHandler. Then once it’d production-ready we’ll have it in the chain of things called for the ToolkitAm1BCCHandler

    • MT – Ok, so this will be handled at a software level

    • JW – Yup

New tickets:

• https://trello.com/c/okbqsXhz/643-look-into-hooking-up-evaluator-to-benchmarking-code

• https://trello.com/c/wLNTkVRf/644-test-benchmarking-code-with-a-force-field-that-uses-plugins