Item | Presenter | Notes |
|---|
Updates from MolSSI | Ben | BP: In terms of production, nothing new. In plain Dataset, status gives lots of NaNs for INCOMPLETE status. One thing added that you might be interested in: if there’s an InternalServerErrror, this will be stored in the DB in a table; if you’re running your own server this is queryable using FractalClient. On public instance, allows me to find these without trawling the logs. Really been doing a ton of backend work, simplifying things, hopefully making it faster
|
Validation clarity
| Simon
| |
| | |
How to add more compute specs to an existing dataset | Pavan | |
Can I run a set of hessians similar to adding compute spec? | Trevor | DD – Not really, since the starting molecules are not the same as the initial dataset. I think we could implement this as a Github Action that launches hessians as the optimizations complete. So, each time the action runs, it would take the completed optimizations and start a new hessian for it. TG – This sounds great, though it conflicts a bit with the idea of versioning in the dataset standards DD – You could say that “the hessian dataset is tied to its optimization” JW – Could also have the action populate another dataset with -hessians appended. TG + DD – That’s how we currently do it. DD – Could do it either way, but I’m somewhat partial to extending the existing dataset TG – I feel weird about messing with state after creation, but we already do that a lot, so maybe that’s not too bad SB – From the researcher’s side, I think it’s OK to have these state changes. We are switching to a model of recording the state of the dataset by enumerating the record IDs that we pull down.
|
Hessian convergence | Horton | JH – We’ve previously run hessians to “gau” convergence, but I think we’ll want to run to “gau tight”, also want to make integration grid fineness higher. In my experience this is necessary to get good results. TG – That’s not a setting in the hessians, that’s a setting in the optimizations. But in the big picture, this will depend on whether our hessians contain imaginary frequencies. Imaginary frequencies shouldn’t exist if we’ve really converged. DD – HJ is submitting a hessian job right now – Could one of you take a look at whether she’s doing this appropriately. TG – Not sure that HJ’s dataset will require this. DD – So, let’s not change HJ’s dataset for now, and investigate the outputs to see whether they’re visibly bad. TG – JH, is convergence criteria set in GeomeTRIC? JH+DD – Yes TG – If we’re going to put the convergence criteria tigher, should we make the wavefunctions tighter as well? JH - Definitely agree on a finer grid, that should help convergence.
JW – Submit one or two datasets (with original vs. tight criteria)? DD – I’d think we should do two, that way we can study the result of the changes. SB – HJ will submit a new torsiondrive dataset for Sage, that should be very high priority. After that there will be a hessian set, and I’m not sure which molecules will be in that DD – So, let’s do one torsiondrive dataset with tight, and the other with the standard. The standard one will be high priority initially. Once it’s done, the tight-criteria one will become high priority.
|
Irregularities between optimizations generated from torsiondrives | Simon Boothroyd | SB – Issue 540 of QCF SB – Not sure whether this is outdated – would like to ensure that this isn’t still happening DD – Main issue here is “atomicity” regarding how the manager talks to the server, and the server talks to the database. There are opportunities for race conditions in these two paths. So this is a situaiton where data that should have gone into the history gets dropped, and causes a cascade of dropped info. So this may go away with our current updates. This is a hard issue, because it’s a random race condition, so it’ll be hard to reproduce and measure. But these things should cease to happen over time as BP does work. SB – This isn’t a blocker, but I’m worried about having a different number of entries and minimizations, and getting something like off-by-one indexing errors, or fitting to the wrong minimum. We don’t currently have an automated way to filter out these cases. DD – Thinking about it, the only way to check for this would be to parse the stdout of geometric and see if it matches the number of optimization steps. SB – For now we’re going to take the results at face value. I think a fix for this would be on the QCF side. TG – In my experience, the thing that breaks is the table lookup, where the optimizations worked, but when the torsiondrive tried to collect everything, the table indexing is off. So I do manual energy checking myself to detect this. DD – If there is an incomplete history, is the final molecule correct? SB – Do the final molecule and the final energy come from the same thing? TG – I think the final molecule+energy is legit, but the torsiondrive grid point table doesn’t report the actual minimum. SB – So, that’s reassuring. Even if the structure isn’t the true minimum, the returned geometry DOES correspond to the appropriate energy.
DD – Does this always show up as an error? Or is it ever silent? TG – Both. In some cases, it’s an indexerror in the lookup. But sometimes it is silent. SB – Yeah, there are two failure modes – Some silent, some loud. TG – Best way to debug it is to see “who ran it”? When Levi saw this, it was coming from a dirty branch of some QC* (QCEngine?) deps. This was from a long time ago.
Loud case: Silent case: TG – This appeared when Pavan ran a query that returned None – I’llt ry to find this in Slack. td.dict()['minimum_positions'] will be incorrect
DD – This looks like a race condition
|