NAGL public roll-out plans | LW | Phases of opt-in/opt-out for NAGL use System for triaging bug reports and complaints Advertising the switch to NAGL Write preprint so there’s a place to explain performance, etc. MS – What do you mean by “use the toolkit registry programmatically”? LW – Defining a new ToolkitRegistry MS – I think people expect to have a simple kwarg to use JW – With environment variables, we have precedent with the INTERCHANGE_EXPERIMENTAL flag with the custom toolkit registry, that’s the standard in our docs
LW – speaking as a user, the ToolkitRegistry code is the standard way, but it’s hard to thread through to downstream packages like Evaluator. So for evaluator I use eg unsetting of the OE license as a flag to disable it. CBy – Hopefully users that use our toolkits could be such nonexperts that they don’t need to know what AM1BCC is. So maybe a more future-looking approach would be to change the flag to say “partial_chargeS_ instead of “am1bcc” specifically. This could make it more future proof. CBy – Maybe env var could do more than just AM1BCC CBy – As the RESP guy and AM1BCC guy, I view these models as different charge models. My original intent was for AM1BCC and rESP to be so functionally similar that they could be sued in the same FF. JW: NAGL is meant to be an AM1-BCC charges. CBy – Can’t a FF request non-AM1BCC charges? Like ESP? LW – Our current FFs are trained with AM1BCC so they request AM1BCC. But if we train a FF with more advanced charges, then we can request those from any tool instead. CB: NAGL is not AM1-BCC charges, and they do differ, but they are a different charge model. Is this different from when AMBER was trained to RESP and now AM1-BCC is substituted for RESP charges? MS – I think NAGL IS an AM1BCC provider right now, but later it’ll provide other models. LW – CBy – Right now AM1BCC could be used in place of RESP, and now maybe NAGL could be used in place of AM1BCC? But NAGL2 will be different and/or better. LW – Somewhat philospohiccal discussion about what “is an AM1BCC provider” given that NAGL is already closer to OE AM1BCCELF10 than AmberTools. So plan now is to provide NAGL1 as an AM1BCC provider. JW – Support bringing this to lead team to see if we want to make changes on this.
LW – With 3 month timeframe, goal is for people to see warnings that AM1BCC charge assignment will go through NAGL. CC – I don’t think we should do this base on timeline, and instead based on FF releases. So I think we shoulod make a point release of Sage trained with NAGL charges, and have that one’s release date be the day we switch over to NAGL. CBy – What I likeabou this suggestion is that it is a bigger public declaration, and it lets old FFs request old charge models and new ones request new charges. LW – CC, were you thinking that previous FFs wouldn’t try to use NAGL charges? CC – I think we should keep the pre-NAGL FFs from using nagl charges.So FF could specify which versions of JW: hesitant as this requires us to keep supporting AmberTools. If NAGL strictly adds dependencies to our stack but doesn’t remove them, I’m much less enthusiastic. CB: we can try to maintain backwards compatibility for a time, but deprecate them after a while (with lots of warnings etc). JW: that’s possible under initial plan. With differentiating charge models this means we deprecate our old FFs altogether. CB: What CC and I are thinking is that Sage 2.2 + NAGL is not really Sage 2.2. You can expect numerical differences. JW: we already have these issues. With RDKit + AmberTools as alternatives, we already see differences. Also versioning – new releases of OE, RDK, AT, we always get differences (+machine differences etc.) We can’t continue maintaining a stack that gives identical results for old AM1-BCC routines. CB: That makes me like Chapin’s idea even more. NAGL charges are deterministic, yes? LW – RIght CBy – JW: user experiences issues I can identify are: CB: … Charge model can be changed by user JW – It would be bad if a lot of our users were hand-modifying our FFs CB – I don’t think that would be too bad, there would be a record of what people were using… Broadly in agreement with CC that NAGL-AM1BCC should be treated as different from “real” AM1BCC.
AMI: I like CC’s idea because it forces users to update their envs to use the new force field LW – If we treated them as separate charge models, would you be ok with us someday not providing AM1BCC software CBy – I think it’s OK to deprecate old things. CC – I think that deprecating AM1BCC in the toolkit DOES implicitly deprecate the parameter sets. Maybe we can fix that by providng docker containers to generate charges using the old methods. PB – People are already using AmberTools AM1BCC in lieu of OE ELF10, so I think it would be fine to use NAGL with previous FFs. CC - I’d consider that different from the old FFs JW – We already allow AT and OE to provide charges which we know are different cand call it same FF. What’s special about this, or are we holding outsflves to hgiher standard? CC – I think we should hold ourselves to a higher standard. LW – … LW – I think we need to bring this back to lead team/ad board/gov board. But this discussion seems to have identified that maybe NAGL should be considered as different from OE AM1BCCELF10, and maybe AT should be differnent from OE as well. CBy – I’d heartily advocate that an AmberTools AM1BCC-trained FF should be treated as a different from one trained with OE. LW – Interesting philospohical discussion here - Should even open source provided charges in current FFs be treated differently depending on AmberTools+RDKit version? CBy – This is similar to why I did ELF10 - mitigating these factors.
LW – System for triaging bug reports and complaints? JW and I discussed handling scientific vs. user experience issues. Also discussed threshold charge difference for when soemthing is a “real” issue to keep from getting flooded with small differences. CBy – In Tuesday meeting, AMI showed some results that look like a case where we could identifyt a scientific issue, where a parameter had, in addition to the spread of parameters around 0, there were errors not centered around zero. This seems like something where errors could be found in an automated way. LW – Kinda thinking of something different - that was measurements of a valence angle around phosphate - but that’s not something I’d fix with a charge model. CBy – For NAGL, I’m behind on looking at the results - were there populations that seemed to have issues wrt ESPs, solvation free energies, etc. LW – Yes, some groups without sufficient training data.
LW – Thoughts on preprint? MS – Nothing too specific, AMI had offered to help out. Mostly a matter of laying out tasks now, what would be the best way for you to start writing? LW – I ususally write on GitHub and overleaf MS – I’ll start an overleaf connected to GitHub LW – Great, once that’s started I’ll begin transferring from various places on GH to the repo.
LW – How to advertise switch to nagl? Having warnings from the code is one option, though that requires people to update. Also looking at social media and slack. JW – Also OMSF newsletter. CBy – Would be good to have a whitepaper that shows method and validation. That’d help us adopt at OpenEye, then we could mention it at our presentations and such. MS – Preprint could serve this purpose PB – Could put recent presentation on Zenodo for use as a kind of whitepaper. LW – Could do that, but would probably collect the best from those ofr the paper PB – Right, but a recent talk could be a good immediate thing to put out. MS – Right. LW – Would a recent talk be sufficient for you, CB, or would you prefer a preprint? CB – Once you figure out whether NAGL will be released with a FF… We’ll need something (like zenodo artifacts) with solvation free energies, numerical charge comparisons, etc MS – Hopefully preprint can come out fast. CB – Being able to cite a preprint would be best. LW – If we retrain a FF and have to rerun benchmarks, the preprint wouldn’t be the limiting factor. So if we go in that direction it’s not a bad idea. If we release NAGL by itself… … Let’s work on the preprint and see how the timing works out.
CC – There are some downstream packages that use OpenFF forcefields - eg CHARMM-GUI - Could we coordinate with them on communicating the change? CB – I’m seeing this as a really nice alternative to running AM1BCC a lot, and another place where this would be helpful would be in high-throughput chemical database operations. The minimal architecture needed for this would be the openff toolkit? If we wanted to put NAGL into quacpac, would that be easy? LW – I think this would be “hard” at the moment. It’d be easy if you could take the model weights and recreate the operations, but something with pytorch doesn’t work with DGL and that makes it hard. CB – Yeah, this could be a good standalone tool for our community.
|
