2020-06-19 QCF Submission Meeting notes

Datasets in the PR queue

• Rowley Biaryl torsiondrive dataset: Trevor working on initial validation with local QCFractal snowflake instance

  • @Trevor Gokey and @Joshua Horton will work together in a tight iteration loop to get this dataset into a state that is ready for submission to prod QCA

  • As a bonus, will develop out our CI validation approach

• DANCE dataset needs feedback from @Jessica Maat (Deactivated) since Bryon may now be graduated

• @Joshua Horton will work on getting Genentech dataset ready with QCSubmit; may merge into a single dataset submission if this makes sense to him; this dataset is desired but relatively low priority

Process improvement

• Could set up CI on qca-dataset-submission for datasets using fractal snowflake + QCSubmit

• @Trevor Gokey developing validation functions for datasets that could be run with our CI on new submissions; this will really help us handle larger and more complicated datasets that are on the horizon so we identify problems quickly and before submission, using costly compute resources

• Notes on areas for QCSubmit to check (for @Joshua Horton):

  • Indices between QCA json ordering and mapped smiles ordering

  • Generate 2D images of molecules with torsions highlighted

  • Exposing QCSubmit validators to reuse for user to check/test integrity before submission to main servers

    • Check torsion connectivity

    • Any other checks (“filters”) already in the QCSubmit workflows

Current issues

• Ben: wrt the INCOMPLETE issues:

  • the Psi4 PR addressing the deadlock state has been merged; Ben will get info on when release will happen from Lori

  • There may also be an issue at the Manager<->Server interface where the Manager doesn’t submit complete or consistent information to the Server; Ben will create an anchor issue; not exactly clear if the root cause is actually in the Engine layer

  • The above issue appears distinct and separable from issue where Server can drop updates to result statuses if it chokes on what it receives from a manager

• David: wrt the INCOMPLETE issues:

  • looking to simplify where we can; if we are running single jobs on a manager, would reduce one process boundary by having a synchronous adapter that runs in the same process as the Manager itself

  • we still need a remediation pathway for clearing existing INCOMPLETEs that are gumming up the works while we solution for the root cause

• QCSubmit dependency versions/conda environment needs to be addressed for installation. Versions that Trevor had to deal with to install QCSubmit and get Biaryl dataset submitted and running locally:

  • py-cpuinfo<6

  • master openforcefield

  • pydantic 1.5

  • qcportal (cannot use qcfractal.interface client)

  • qcfractal > 12

  • basis_set_exchange

  • xmltodict, openmmtools were mysteriously missing from env