Day 3 Notes
- Jeffrey Wagner
- James Eastwood
JW notes re: Bespokefit
Timeline for deprecating BespokeFit 1 if we proceed with 2? Paper came out in 2022, what is a good timeline for support?
It will be very expensive to support BespokeFit 1 and 2 at the same time - exposes us to a lot of upstreams
BespokeFit 1 unique deps: xtb-python (unsupported), ForceBalance, celery, uvicorn, starlette
Bespokefit 2 unique deps: At least MACE engine (though licensing issues, so maybe AIMNET2), smee
The number of unique upstreams means that we don’t have pre-existing knowledge of many of the inner workings, so a lot of debugging starts by relearning how the components/architecture works.
It’d be ideal if a potential BespokeFit 2 was “simple”, like by using only local compute. Fancy work distribution is dangerous and can cause user disappointment:
Complex support burden for folks on different HPC setups (e.g., networking issues)
Large scale compute makes things hard to reproduce (e.g., “I encountered this problem after the job ran for 50 hours”)
Confusing things can happen with caching/batch submissions, like when reuse of congeneric molecules does/doesn’t happen. Have there been real cases where the benefit is worth the costs?
MG: The benefit there is to avoid adding noise to a congeneric series
DC: It helps if parameters aren’t changing too much
MG: Sure, but if you’re changing the chemistry maybe it should change, and we shouldn’t worry about it. It’s hard to know when it is a series or if it’s not, it’s not well defined.
JW: Yes, there isn’t a strong metric and as it is now, a change in presenting the molecules will change whether they are grouped or not.
DC: If the science team is involved there could be human input, but you probably don’t want that.
JW: One solution could be that you can’t run with more than one molecule. Especially since we expect that the next step would be to run the free energy calculation and that is the expensive part.
JH: This all sounds reasonable. Right now it seems like we are the only ones that can use Bespoke fit 1 right now and that’s not ideal. Move away from the torsion drives in general
DM: OpenEye has built support for this in Orion, Cresset and others are using it, so we do have a larger base. As to the question of whether we want to deprecate it so soon, we should talk to these stakeholders before making such decisions because they are currently investing into incorporating it.
JH: If we can adapt it to run without caching and a single molecule that can solve those problems in a more simple way.
JW: This is a cool idea and we want people to do this, but there are product lifecycle issues/questions. Our goal is to enable continuing development, publicize metrics of Bespoke 2 roll out and keeping our stakeholders apprised, but related to that, we need a plan for the lifecycle of bespoke fit 1. If bespoke fit 1 lived for 5 years, we can budget it, but the opportunity cost with our implementation of SMEE would exist.
LW: Do we have any obligations for bespoke fit 1?
JW: I don’t think so. Danny / Josh do you have opinions on whether we can give bespoke fit 1 a 5 year expiration date?
DC: I don’t think I have a problem with it, contingent on there being no issues with the scientific validity of bespoke fit 2. Continuing the development of bespoke fit 2 is on Finlay’s priority list, but not at the top.
JE: So until Finlay picks it up, no one is working on Bespoke fit 2?
DC: Correct.
JW: I’m not saying people can’t work on/be in meetings about it, but if the OpenFF name goes on it, that’s a larger conversation.
DM: We should bring this to the Advisory board, some of our stakeholders might push back on that.
LW: Are we going to deprecate the distributed compute.
DC: Josh has brought up similar thoughts, but no one has been assigned
JE: Is that a reasonable thing to do?
FC: I don’t have a lot of time, but I’m here to help.
JW: I would be cautious about deprecating and disappointing users. We should avoid that.
JE: The methods used in bespoke fit changes the compute requirements, do our stakeholders rely on the distributed compute?
JW: If people have it implemented we don’t want to take it out.
JE: The docs have been updated to strongly recommend AIMNET, and that we don’t support calculations that are beyond a local run. Maybe the docs can be updated to more strongly stress that we don’t support issues from distributed compute
JW: I think we do that, but getting back to the topic of expiration date, we need to set one now. Once that is done, we can talk about what bespoke fit 2 looks like and work on it.
JE: If we are working toward implementing SMEE instead of FB, we can’t keep supporting something that relies on FB, so it sounds like the timeline is dependent on what our end date is for FB… but we don’t have that expiration date yet.
JW: Great point, I still think we should start warning them early not when we are ready.
DM – Running high temp MD reminds me of TG’s NEB work. TG, please share that with DC when possible.
Future of Bespoke Fit: Danny Cole
Tom Pope: Working in short term appointment with Danny
MG: How different are the new torsional parameters are those from Bespoke fit to the original Sage?
DC: Josh said that in some cases there were significant changes, but we are considering recomputing those.
JW: (RMSE slides) What conformations are we seeing these distributions for?
DC: we are seeing the RMSE of the FF from each iteration of the fit measured against the QM data from cumulative ensemble from all the iterations
TG: With the explicit dependence with the linearized bonds and angles, how do the bond and angle frequencies agree with those that come out of the modified seminario method?
DC – It’s all trained and tested on energies and forces, so in theory the frequencies will come naturally, but it would be good to check that.
JW – Will high temp MD step inhibit reproducibility/deterministicness of outputs? Will this change the reproducibility?
MS – Also unclear which temperature to use
JE – Metadynamics may help
MS – Metadynamics needs to happen per-torsion, you’d need a parallel tempering run to be sure.
MG: Torsions depends on run parameters (like the grid and such), so a consistent starting point would be deterministic.
DC – These points are right, but the method should be somewhat robust if the runs visit the important conformations.
JW – Thinking about issues yesterday with viscous compounds - What are the risks of high temp MD wrt detecting convergence/knowing when it’s sampled enough of conformational space? How to constrain runtime/make it more predicable for users?
DC – Knowing when to stop is something that TPope has been struggling with - often uses torsion scans to assist/check whether space has been searched. So some hyperparameters that will need to be fit, and documentation for users on which things might need to be tweaked for tricky cases.
TG: There is the case that if we use a stochastic optimizer, that’s hard to reproduce.
DC – We can check that out too. Not sure that we have data on that, I’d hope that this would be a less influential factor than the contribution from the metadynamics.
MS: As long as that’s clearly communicated, it’s fine
MG: I agree it’s unavoidable in a way
DM: I’m not bothered by imperfect reproducibility, since we arguably don’t have that anyway. I agree we can just be more clear.
LW: I agree that there is likely a hardware dependence and run dependence
JW: I think it’s a slightly more significant deal. With bespoke fit 1 we could put up a fight to make a case that it’s deterministic. Bespoke fit 2 would have a software issue with reproducibility, but it is weird for our brand and hypocritical if it can change significantly between runs.
DM: I agree, but maybe we should run the experiment to see if our current process is actually reproducible, but instead we should make the variation known. Then we release a set of representative forcefields instead of just one. Deterministic is impossible.
JW: Do you agree that bespoke fit 2 is less reproducible than bespoke fit 1?
DC: Without complete sampling I think that’s true
MG: What is the advantage of the MD based sampling?
DC: The uncertainty of how long bespoke fit 1 will take with the torsions leave an unknown timeframe on convergence. The MD is a set number of configurations randomly sampled and we get bonds and angles for free also.
JW: So torsion drive doesn’t know when it will finish, but it seems that high-T MD has the same issues, is there an alternative? Is this a problem inherent in exploring arbitrary conformational space of an arbitrary molecule? or do we just set a time and take what we get.
DM: would NEB solve this?
TG: It interpolates and doesn’t look for the absolute min, but a min path. You need the conformations to start.
MG – Could do a search where you do conformaitonal space exploration using hessians/low mode searches.
DC: It must be in OpenMM or we need another dependency
MS: I expect that we can figure out the temperature we need to get over most barriers with replicate exchange, and then high-T will work
MG – When you do high temperature MD, is it on whole mol or just relevant fragment?
DC – Whole molecule
MG – Is that necessary/could the problem space be constrained by fragmenting?
JE: Doesn’t that mean that a user doesn’t have control over the MW? We have assumptions for small molecules, then we are leaving issues for the users to figure out.
DC: Yes
JW: So there is not a way of exploring conformational state that is BOTH time-efficient AND throrough AND deterministic. So we can say that, since fast runtime and user-friendliness are prinicpal goals in this project, we can say that bespokefit 2 will be less reproducible/determinsitic than bespokefit 1, and we will own that.
JE: From a product development standpoint we need to discuss how it fits in the roadmap. Our stakeholders def want this tool, but there is a misperception that because it exists people think they need to use it… but it is a waste of CPU time in some cases.
DC: People are used to FEP+ right?
JE: Yes because they pay to be told that they need new parameters for every molecule.
DC – I thought it would only be used if the schrodinger software determined that the parameters needed imrpovement?
DM – In pracitcal industry application that means they use it for al ost all molecule
JE: That ties back to the difference between OPLS4 and Sage with the magnitude in the number of parameters.
JAC – JE, you asked whether “if we make bespokefit, then people will want to use it unnecessarily”. Isn’t more usage good for us?
JE – Since we’re not selling it, we don’t gain much directly by increased usage. I think strategically, we need to limit the amount of team effort that goes toward supporting bespokefit, and we need to continue this, to ensure that supporting bespokefit is a fairly small amount of our time, since it occupies a small amount of our strategic space. The discussion about determinsitic-ness is valid, and this occupies a good strategic niche in our strategy, but I want to ensure that our level of effort commitment is proportional to the value this adds to our strategic goals.
DM – If an industry partner wants to create a project that does lots of custom parameter assignment, that should be a seaprate project with a separate pot of money.
DC: Maybe down the line, if a industry partner wants to sponsor we can hire Tom to continue this work.
JW: Today we had a great conversion of bespoke fit 1 vs. 2, we can’t make final decisions today, but we should decide the lifespan of bespoke fit 1. Yes it’s coupled to switching from FB to SMEE, but what are people thinking?
DC: We need a new name for Bespoke Fit 2.
JH: FitBespoke
DC: BADFit
??: Bespokerfit
LW: Respokefit!
DM: Right now we have an implicit 1 year support timeline on everything.
LW: Does FB dying make the decision for us?
JW: We can’t let it die until we can, that might mean we fork and maintain it, which would be unfortunate but necessary.
JE: We can leave this decision until we have a decision on the FB to SMEE transition.
DM: If we told the Advisory board and told them that Bespoke fit 1 and FB are killing us, we can have a good discussion with them about this. We don’t need to let it derail priorities without that conversation.
LW: Can we settle on a minimum notice period, like a year?
JW: Great idea, we tell them that we are thinking about deprecating at some point and we will give a 1 year notice before discontinuing support.
Lipid Forcefield Development: Julianne
MG: It looks like lipid simulations with Sage are collapsing
JH: we think that is a feature of the force field, but it does look like its collapsed. It seems that the way we equilibrate is what flattens it.
JW: Do the CHARMM and Amber forcefields (with very difference behavior) have similar performance with the metrics you alluded to?
JH: We will get to that
JW: Was this the starting conformation? Are they difference between forcefields?
MRS: They start from the same conformation
MG: Do you have an exp result for pentadecane
JH: No, but for hexadecane it’s ~0.3
TG: Did you evaluate the alkane diffusion coefficient with CHARMM and Amber?
MRS: is more complicated.
JH: Using Charm gui to model alkanes is more involved
LW: What do the uncertainty intervals represent?
JH: Standard errors. For heats of vaporization this is the standard deviation for 1 simulation. I ran triplicates for density and diffusion coefficients.
JW: Was this the lipid MAPS optimization data set?
JH: No, this is only including changes to torsionsMG:
(JW later sated his curiosity, it’s this dataset)
MG: Mobility was not affected by the non bonded parameters of those head groups. The attractive forces between the alkanes is not that great and your data suggest that those are very good so I'm thinking that the issue is the stickiness of the LJ parameters of your head groups. Maybe the charges too?
MS: We don't want to change the charges we don't have “levers” to doing that. To change epsillons we have to figure out what the data is.
MG: There is some ion data.
LW: Have you considered running data using non proper angles instead?
JH: Getting indexing error with openff, I’ll follow up with you.
MRS: when running at lower temperatures sage doesn't go to gel phase?
JH: No
JW: Do the simulations with Sage, CHARMM and Amber use difference water models?
JH: Amber uses tip3p. I'm pretty sure they both use tip3p parameters for lipids simulations
MT: you added molecules into the sage qm training data set. what chemistries did you add and how many? Did you change the head groups represented?
JH: I had 63 molecules for the torsion drive data for phosphates set and about 192 torsion drives for alkanes
MT: how does all of those compare to the data that you are training to versus what is in the standard dataset for Sage?
JH: When refitting sage it uses around like 900 diff molecules to train on and i might be adding 100 or 200
MT: Adding more phosphates didn't seem to make them budge much?
JH: No,
MS: Torsions are really not moving much, but they are matching better.
DM: What if we zero all the torsions?
MRS: My instinct is that they are all regularized too much
JH: I did not change the default on regularization
LW: Try rerunning this with .. torsions. Torsions would have to change a lot to compensate for the electrostatic effect
MRS turn down regularization to see if that allows a better fit. I like the idea of getting MM match QM
DM: Do something extreme so test if torsions are the problem.
JH: Would we only be changing the QM related torsions?
MRS: One of the reasons the alkane could be slow if the QC torsions are too high. If all the torsions are all 1k/cal too high that would slow down the diffusion.
MT: the msd plots you showed had an order of magnitude difference. with the alkane the errors were around 10-20% thats why we think its related to head groups
MG: That’s why I think it’s the head group
MS: I agree that could be the parameters or the geometry
DM: If we do something to match?
MG: Reduce sigma values?
MS: We need to get torsions to match qm
TG: Do you know if the periodicity is assigned to those
JH: No answer right now
TG: Sage doesn't do gradients very well so with this dynamic stuff the thermostat might be working too hard. you have rather sharp displacements in the molecules that are really far off. Velcro effect.
MRS: I'm skeptical because the energies should “jiggle” enough
MG: That would break stat mech
MS: Is this running with SD thermostat?
JH: No, md. thermostat parinello raman
DM: Tested that everything was run with same thermostat?
JH: All mdps were run the same for the trajectories shown
MT: about charges, you used NAGL instead of am1bcc, so it’s not Sage (strictly speaking). have you checked how different those are?
JH: I ran sage with am1bcc charges and it didn't look there was a lot of difference. the per atom charges were numerical similar. from that i assumed charges were not making a lot of a difference
JH: Thanks for all this feedback, I can try some more extreme changes and look at the non bonded.
JW: Do we want to look at this projects and determine metrics? For whether we want this to become a flagship forcefield line?
DM: Michael, we need to actually start the focus team, we have others interested in joining. How does this interact with the road map. we are already putting effort in this direction so there is available infrastructure to support, but we don't have a lot of idea of what you are doing.
JH: I'm fine to continuing independently with the occasional help, you all have been really responsive.
JW: it's just a matter of process to include to mainland ff.
DM: It seems like we’ve spent a few months on torsions and found that that isn’t it. Can we more dramatically change some knobs to figure out what needs to be fixed.
JE: working on ionic liquids? Is that related?
MS: Danny Cole and Jorge Miguel and I put in a proposal that went in in November. we are holding off until we hear back on that funding. We will be able to address questions on high charged systems
JE: If thats what we need to tweak to get lipids to work, we also need to do that in a context that works for ionic liquids
MS: Ionic liquids for sure need big tweak in potentials. Charge density in these is way higher, 12-6 potential will not work.
DM: Moderna needs summary of updates in August. we can do that already with what you presented.
DM: Lily and infra team: if they needed something to fit charges on top of NAGL?
LW: you can
JW: Putting into release ff would be complicated to do
DM: We can't treat charges bc we are using nagl. is this something that we are able to do right now?
Discussion for openff 3.0
DM: In some way you have a handle to do this
MT: They can use library, pre-set charges.
JE: If all these problems were figured out, what’s next in terms of OpenFF?
DM: We had a discussion with Moderna about how many lipids does it have to have to “be a lipid forcefield” the answer is not many, just the greatest hits. We can just improve our current forcefield and give it to them.
JW: JH has done some great work, so it should go through our checks and be approved easily.
LW: The end goal would one to pass through the benchmarks and release it.
JE: Is there a level of customization in which we can support.
LW havent thought about that yet. We keep calling OpenFF 3.0 Rosemary and the protein FF Rosemary.
JW: If we reach the crossroads in which with library charges it looks good but with NAGL charges it looks bad … comparing these charges will be a decision for determining the model
JH: Will the lipid forcefield and protein forcefield diverge at this point then?
DM: Yes there would be some work but the bar is low, Schrödinger’s conclusion is that you tune for the application and it’s not general … idk what we would run to make sure the protein and lipid ffs work well together
MG: If phosphate needs to be adjusted it may be affecting elsewhere
MS: The torsions we are getting are not affecting other torsions….
MS: We don't know what we have to do to make sure that they are consistent
JE: It would be cool if Rosemary had all the parameters to work for all biomolecules. of we can release lipids and proteins at the same time for a major victory.
MRS: lipids and proteins, not RNA
We can rename FF-Release to a Lipid focus group or attend the bio-polymers meeting.
Let’s have this topic join the bio-polymer meetings.
JW: Que no polymers tambien?
MT: Let’s reconsider that.
Interchange: Matt Thompson
Is “convert to smee system” a coherent ask?
MT – I don’t currently know what this means, but I think this would make sense when I look into it.
What does “backdoors for low accuracy transformations” mean?
MT – Probably less strict compatibility checking for loading/combining components
Ensure charge neutrality from AM1BCC?
DM – This may have to do with ensuring that there isn’t any fraction of a charge left on molecule after AM1BCC charge assignment due to rounding on individual atoms.
DM – Re: Examples - Long ago I read a covalently modified peptide script that people still apparently use. Could be good to jump on a call with VVoelz to ask what they’re doing with modified peptides and tell them about our PTM capabilities and using interchange.
JW – Surprising that people WANT a pypi package. Our ad board pretty unanimously didnt
TB – It’d be easier for me to do my research with a pypi package. OpenFF is my only conda dep right now and it makes installation clunky
AF – Could whole openff stack go to pypi?
Most of it, yes. Bespokefit and rechange and stuff would need to stay on conda
DM – Re: adoption, MD format converters generally gain users when the last thing they used gets dropped, or the new one provides functioanlity they need
from_gromacs is already implemented, but is experimental. Importing from other ecosystems in ways that might not involve any SMIRNOFF components at all could provide value as a gateway drug.
DM – Would be good to do publicity - Like, after adding roundtrip tests and combine - could be a good blog post
JE – Worth tagging 1.0 roadmap? Or do you not expect to get useful/quality feedback?
MT – Current feedback hasn’t been overwhelming. I might want to coordinate with the protein FF release, since there may be a lot of performance improvements/accuracy tests that will be basically necessary to use it.
JW – I’m in favor of publicly posting the roadmap as a way to get feedback
MT – I expect some feedback may be weird/out of scope, so as long as we can cordon that somewhat it should be workable.
JW – I like the idea of a JOSS paper a lot
PB – Do people still use PMX to make gromacs inputs? If so, is that more of openFE’s stuff
JW – Hybrid molecules have been pretty solidly seen as out of scope for Interchange.
(support for hybrid molecules)
MS – The way that GROMACS and OpenMM represent FE calcs is different - So question is how people using interchange would zhuzh the gromacs/openmm outputs into their FE packages infput format.
Packmol discussion
JW – I think we can draw a line between using packmol for evaluator setup is essential, pontibus is mostly essential, and helping users solvate protein-ligand systems is out of scope.
TB – I like using the packmol wrapper and would like it to remain available and become more discoverable
DM –
JW –
MT – I’m thinking of wrapping a subset of packmol, so like not micelles/bilayers. But packing boxes of solvent. OpenFE’s concern was that we had two private wrappers that seemed to do similar things but did different things.
DM: I’m concerned about endorsing PDBFixer. Their examples use a lot of things like OMMFFs that we don’t want to be pushing users towards. I would be after examples that keep me in OFF-land. I’m not ver persuaded by the concern that users will want more than we want to support.
JW: PDBFixer does a much better job of a water box than Packmol. A compromise could be making it publically available and adding caveats for packing water around a protein; it’s a research method and you may need to equilibrate.
DM: is it worth treating water as an exception?
MT: (shows functions in _packmol.py); these are what he wants to be public.
DM: does solvate_topology use pre-equilbrated water?
MT: like the carve-out method? No.
MT: it would be useful to have an example of PDBFixer for water solvation. We do have it already inside another example.
JE: does it meet the needs of pontibus?
MT: yes
TB: these are what I was referring to. I’m not assuming the outputs are physically plausible, I just need something to pack boxes. My proposal was making it more public and discoverable. I’d like it to keep existing. Note, I’ve run into some issues with solvate_topology with residue naming as it just adds a water molecule.
MT: we can’t have this be public and tell users not to use it. Not making this public more or less necessitates telling people they shouldn’t use it. We have a number of examples that use this method. Also, it would be hard to deal with user requests.
JW: I’m concerned that this doesn’t perform well in all cases; it’s a yellow in the traffic light system. We would need to add a number of caveats for solvating proteins or protein-ligand complexes; they would not be at the right density, not well packed, etc. But I’m in favour of making them public and loudly documenting this.
DM: I think water deserves these loud warnings. Typical workflows take pre-equilibrated water and just carve out spots for your protein, so the bulk part of it is broadly correct; the local part just really needs a few ps to equilibrate. Here we’re probably looking at ns+ to equilibrate. So water would need disclaimers.
MT: We should absolutely document differences in expected behaviour and what you actually get. What I’m uncomfortable with is putting a caution light on this as a whole. I find tagging things with “experimental”, which is essentially a red light, easier to deal with.
JW: I think that’s a good system for red light stuff.
DM: One other thing that might be worth doing is linking a list of solvents we routinely pack with solvate_topology_nonwater
LW – Even the ones that we commonly use are packed very badly and require equil
MS: the number of things we can pack is quite large, I’ve used it to pack ionic liquids etc.
MT: we should document as much as possible and add warnings for high-viscosity compounds.
PB – System dependent issues…
MS: should maybe just put out some general warnings and get users to check their outputs.
JM: I think the experiment/production split is useful for notebooks, but we have warnings all over the place in the code so it doesn’t really mean anything. My other suggestion is that we could include some equilibration as part of the method. Not sure this is a good idea. Also, if we make this public I’d like to take another pass at tweaking it and ensuring the arguments are correct and well-named.
MT: to clarify, JM’s correct that there’s warnings everywhere and probably everyone ignores them. I have a decorator in Interchange for experimental features, where if you don’t set a particular environment variable you flat out can’t use it.
JM: yes, agree, MT’s approach is very different from the toolkit’s.
JW: the toolkit’s issue arises from neglecting older documentation issues and never cleaning things up; in favour of using MT’s approach going forward.
JW: I’m not in favour of adding equilibration in. We should tell users not to use it in production before some equilibration. Users will be the experts here.
JE: Agree. Philosophically, Interchange is the interoperability tool that sits before MD.
JM: makes sense
MT: Agree that equilibrating by default is probably not we want, but see some potential here in e.g. making an example notebook.
LW – Agree with TB that packmol method is increbily useful. I don’t want an equilibration method. I would like things to be more findable/usable.
PB – I wonder why we need to tell the user they may have a badly packaged box.
JW: our users won’t be expert users. They’ll be from industry who need results ASAP and take the shortest route to results. This would result in a bad surprise.
PB: aren’t there better methods then?
JW: yes, pre-equilibrated boxes are much better.
TB: it’s a question about audience; slightly surprised you need to even warn people. I took it as a given that of course you need to do some equilibration later.
JM: we could also have an API that packs a very small box, equilibrates it, tiles and uses that to solvate.
JW: that sounds like a good algorithm if we want to invest the effort, but I don’t think we want to invest the effort.
TB: there’s also the question of what if I want to use a really different water model. Seems to make sense to outsource that to the user.
MT: my proposal is to do some tests, do some refinement, then add docs and warnings. I don’t think the issues that have been raised are common or bad enough to stop people from using this method.
JW – I agree with this proposal
Docs / PTM Roadmapping