2020-11-17 QCFractal Users Meeting notes

Updates from MolSSI

• BP – Some troble with update+migration last week. Everything is up and running now.

• BP – No upcoming plans. Though will update server more frequently. It turns out I can upgrade server without requiring updates to managers.

  • JW – Would this cause server downtime?

  • BP – No, that’s database migrations, and we won’t do those frequently

• BP – Something is downloading lots of jobs continuously

  • TG – That’s me. I iterate over jobs and look at the STDOUT for errors

  • DD – We were getting a higher error rate because of a dependency problem on Lilac. I worked with Chodera on this, turns out it was a problem with our “everything bagel” (QM+MM+ML) conda environment for QCFractal managers. Specific problem was with data classes, unclear what the exact problem was.

  • DD – To prevent this from happening again, we are looking at implementing “canary tests” for these conda environments.

• BP – MolSSI wants feedback about what you’d want in a job management dashboard. Things like: active clusters, number of hosts, etc. Goal would be to manage compute and see job status. What sorts of information should be shown, in what kind of plot (bar, pie, etc?)

  • DD – I’ll post this on QCFractal

Queue/Manager status

• DD - Managers running at PRP and UCI. Lilac managers are running but need to be updated.

• DD – 8 datasets running, 3 queued for submission. Mix of ANI, QM

• TG – My managers are mostly busy, there are some lulls when downloading/re-evaluating STDOUT

User Questions

• (General) – We found code to do wavefunction reconstruction, it’s available in psi4 but not well documented. We’re going to make it accessible through psi4 and recharge, and will make it available elsewhere after we’ve tested it a bit.

• HJ – If I want to submit optimization datasets, which starts from the frames in a torsion scan, for example spaced 15 degrees/24 steps around a torsion, what’s the best way to do that?

  • TG – If it’s the same molecule with different geometry, they should be separate entries.

  • TG – Optimizations do one molecule per optimization.

  • DD – I’d treat this as one molecule with 24 conformers.

  • TG – What should happen is QCSubmit will detect it’s the same molecule and will append them all as conformers into the same molecule.

  • DD – Is there any other benefit to stacking the conformers into the same molecule?

  • HJ – I’ll plan on loading the input conformers into QCSubmit and trust its built-in deduplication

• HJ – What are the common causes of errors (like, in error cycling)?

  • DD – QM convergence errors, SCF errors, geomeTRIC convergence errors, lack of memory

  • HJ – Is there documentation of these that I can check? For my own purposes I’m running local QCEngine jobs and want to know how to diagnose my errors.

  • TG – I have a script that diagnoses errors on the fly during local runs. I’ll send it to you.

  • DD – I’m running the Ulrich’s/dev2 dataset locally and diagnosing local errors as well.

  • JH – What kinds of errors are you seeing? I’m seeing at least one molecule that having geo. conv. errors.

  • HJ – I’m seeing geometry convergence errors

• HJ – When I submit to qca-dataset-submission, who should I tag as reviewer?

  • DD – When it’s ready for review, make a comment tagging @David Dotson @Joshua Horton @Trevor Gokey and we can assign whoever is most available.

• PB – When is memory set? If we’re going to do a large basis set/molecule, is there a tag for that?

  • DD – There will be a higher memory queue where molecules that have out-of-memory errors get sent

  • DD – This is possible due the BP’s recent implementation of task modification

  • TG – Is there a way to see how many jobs are in a compute tag?

  • BP – Yes. Also excited about having defunct tags.

  • DD – We will want to avoid proliferation of tags, so that we don’t end up having workload/worker imbalances.

• PB – Status of JM submission?

  • JH – There was something funny about previous phenyl set, where molecules with iodine had high SCF convergence error rates.

    • DD – I think this set is ready for submission.

    • JH – Could we confirm that the iodine-containing molecules that we removed from this set are necessary? We’ve seen these molecules before and they have never converged. So let’s check with JM and ensure that these molecules are OK to exclude.

    • TG – It’s possible that starting from a different initial geometry will resolve these convergence errors.

    • JH – A different input geometry might work for optimizations, but not torsion drives, since we can’t control the intermediate geometries in those.

    • JW – So for “whether to submit this dataset with tricky molecules removed”, it seems like the answer is “Yes”. In the longer-run, we might consider if there’s a way to avoid these convergence errors in torsiondrive datasets by more cleverly choosing intermediate geometries.

• JH – HJ’s upcoming submission also totally lacks molecules with iodine.

  • HJ – That’s correct and intentional.

• PB – Can the number of tasks be less than the number of unique molecules?

  • TG – Not really.

  • DD – In theory, yes, but one would have to submit molecules with no task.

  • PB –

    • (Ctrl-F 1043, 1041 )

  • TG – This may be a case of automatic deduplication. This might be based on identical InChI, or identical generated conformers.

  • JH – Could have different InChI, but same QCArchive conformer

  • DD – When this submission goes to the public QCArchive, we’ll “lose” two molecules. We should figure out which molecules these are and whether they’re important

    • JH – Can look at which molecules would get submitted and hashing all of them, then finding which hash is missing.

    • JH will work with PB to make a script for this.