2022-07-12 Protein-ligand benchmarks meeting notes

Storage and serialization working group update

David W.H. Swenson

• Slides

• DS – Not sure if we need to have a working group on this - Probably doesn’t require multiple meetings, and we don’t need to use that much of peoples' time.

• DS – Mostly been focusing on the per-edge data storage, which includes raw simulation data

  • labels look like file paths; this gives a standard way within gufe for where serialized versions of different classes are stored within a ResultStore

  • We’ve chosen to store components in native formats where possible (e.g. SDF for small molecules)

  • JW – “Scientific context” in file path; can you give more detail?

    • DS – protocol settings will be encoded by the transformation id; that can be used to identify a JSON file giving full transformation/protocol settings in /setup/transformations/{serialized-settings.json}

    • JW – this seems like we have things in 3 different places; may make it hard to find what you want within the data?

      • DS – we’re trying to avoid duplication of data storage; this design is trying to avoid that (this is similar to normalized relational database form of data)

    • LN - Why not instead of having this key system, with hashing and serialization, you use a DB system (e.g. Postgres)?

      • DS – There are a few issues with this - You need to maintain a database schema and know which tables you’ll have. But stoage is kind of “dumb” - It’s just bytes, and key-value pairs. Whereas a database would be kinda strict and brittle. So this would be more natural for a user.

        • DS – DB would need to serialize in some cases large files

      • LN – You’d mentioned a few storage mediums - S3 is one of them, but what about the others? Will the labels be persistent across these data stores?

        • DS – That’s exactly what we have implemented. So there will be a memory backend (string keys in a dict), storage backend (where this is a proper file path) and an S3 version. And people can implement other backends - As long as they can transform this key into something suitable for their storage format

    • IP – As far as I can see, these JSON files are pretty static. Are you anticipating these being written/modified frequently and being pretty large?

      • DS – We’re expecting each file “chunk” to be separate. So the files will be written once by the simulation and then never again. If you wanted, you COULD make a backend that stores everything via JSON, in which case you’d have a massive JSON. But that’s not on our roadmap right now.

      • IP – Ok, because appending JSON requires parsing the entire thing.

      • DS – Right, so we’re not planning on doing that. An earlier iteration of this plan used a giant JSON, which may be what you’re referring to, but that won’t go into production

ProtocolSettings taxonomy working group update

Mike Henry

• I’ll be meeting with BR tomorrow, to get more feedback. Already got great feedback, expecting to have a good conversation.

• MH – There’s no way to validate a SMIRNOFF force field using an xml schema.

  • JC – Could you providemore detail?

  • MH – There’s no XML schema to validate a SMIRNOFF force field.

  • JC – Can just use the ForceField class constructor as a validator.

  • MH – That’s right, this isn’t a blocker and I can try to parse using the toolkit.

  • JC – A SMIRNOFF force field should specify energy completely. The thought is that other FFs don’t, and we want to make sure that OTHER FFs have enough info to specify energy. So we should focus on how to specify additional info for FFs like GAFF and ff14SB.

    • e.g. gaff and amber14sb needs some additional information; perhaps implemented as an AmberHarness that bundles the settings along with this supplemental information

  • MH – What I was hoping to not have to trust the user that the SMIRNOFF FF they provide is valid. But it looks like there’s no easier way to do this than initializing a FF from the source.

  • • JC – throw it into the parser is the best answer for this

  • IA – We can’t only support OpenFF and GROMACS - What if a user wants to use CHARMM?

    • JC – There’s no open-source parameter assignment engine for CHARMM

    • IA – Even if it’s not CHARMM, we want to keep the door open to other user sources of force fields/parameters.

    • JC – There's no realistic alternate small molecule parameterization pathway other than antechamber/leap. I spoke extensively with JSwails and we found that the only way we could possibly support other stuff would be with a series of docker containers.

    • IA – I’ll need to think more about this, so I’ll follow up. But I want to avoid the situation where a researcher build a new FF and tools and wants to pass them in, but we reject that.

    • JC – We need to recognize that there’s a ton of effort required to make these pipelines work. The best thing we can reasonably do is provide a well-defined destination format.

    • MH – I want to avoid a situation where there’s a mixing rule specified in a force field, but then it gets overwritten by their runtime settings. So, like, if there are two conflicting sources of FF info, which one wins? And how do we document/communicate that?

    • LN – About the only way that you can make this usage safe is by making anything smelly throw a useful error. And this will be a big time commitment. JC’s idea for a containerized “kit” is probably the best compromise here. Then if there’s missing info or info getting overwritten the software should emit lots of errors/warnings.

  • DD – The first FFs that we care about are openFF force fields and GAFF and AMBER ff14sb. For protein-ligand benchmarks, are there others?

    • JW – We should also consider previous AMBER ffs

    • IA – From OpenFE’s perspective, when we talk about “AMBER FFs” we will also want nucleic acid and lipid FFs.

    • JC – I maintain openmmforcefields and things like lipids are problematic because of nonstandard naming conventions. So it may be better to have something minimally defined and fenced off called a “force field”, which may be amber/charmm/openff on the backend, and let them handle their own validation. But I really don’t think that this is feasible at a large scope, it’s deeply complex and we can’t handle a lot of additional FFs without hitting VERY DEEP complexity that we can’t afford.

  • MH – This sounds like a good plan.

  • JC – You might want to talk with someone from GROMACS to gather more details. For different formats there will be different information contents required. But you can think of things in terms of their openmmforcefields inputs (like, a string FF identifier and a dict of kwargs) then you can flesh out the requirements.

• JW – I do have experience porting FFs to SMIRNOFF formats; that is a possible way to go about this

  • is a tedious process,

  • MH – Understood. I’ll keep that in mind as an option.

• DD – Could you open a PR to GUFE to outline the structure of these objects?

  • MH – Sure, I’ll do that soon after I talk with BRies.

protein-ligand-benchmark - 0.3.0 update

David Dotson

• blocked by protein-ligand-benchmark#52 - IA update

  • IA – On the PDB front, I’m in “icode hell”. I’ve opened #58 which tried to fix things. There are issues with thrombin, since it has altlocs and noncontiguous residue numbers. Moving forward with the assumptions that these are intentional.

  • IA – There’s a question here, following discussion with JC and IP, about interpretation of structures. The files so far have been prepared using Schrodinger to identify disulfide bridges. So there are SSBOND entries. But other tools ignore these/reinterpret them. This raises the question of whether we should replace CYS with CYX to help those other tools understand.

  • JC – CYX isn’t a valid PDB residue name. Did you have a chance to try out the normal AMBER prep process?

  • IA – Yes, I tried that. I had to skip using reduce since it does a lot of interpretation itself. So I’m not sure what’s the best way to proceed.

  • JC – I’d like to find a way to remain compliant with the PDB standard and also be readable by these tools. Maybe we could get pdb4amber to be updated to do the right thing in this situation.

    • OpenMM PDBFile uses CONNECT, doesn’t use SSBOND entries; detects disulfides by proximity

      • that may be poor behavior; we can have that fixed upstream

    • DD – Do you think there’s a path forward even if we can get things pushed upstream quickly? I want to make sure we’re not blocked by big external changes.

    • IA – As long as OpenMM can be made to behave correctly, we shouldn’t be blocked.

    • DD – pdb4amber is a public repo and we can open a PR for this.

    • JC – Could we get access to the amber source repo? BTW, pdb4amber source is no longer public

  • JW – I used to have access but I can’t find my credentials now (may have been connected to my UC Irvine persona)

    • http://gitlab.ambermd.org

    • (General) – We could have representatives from different stakeholders here join

• DD – discussed the value of resurrecting original networks where possible. Some targets are impossible to use original networks due to undockable structures - seeking formal decision

  • JC – I don’t think this is critical, but it would just need to be done for original JACS set

    • DD – Thrombin and TNK2 networks could be fully reconstructed, but the others weren’t

    • JW + JC + IA – We approve not reproducing the old networks

Gromacs support - OpenFF needs followup

David Dotson

• what was the conclusion on OpenFF’s needs for gromacs support? Near term?

  • JW – I approve not having GROMACS support in the intiial release.

  • JC – We should be able to rerun the old benchmarks.

John Chodera

• advantages of using data across transformation runs for DG calculation vs. treating them as separate samples

fah-alchemy - current board status

David Dotson

• fah-alchemy : Phase 1 - MVP

• 7 weeks out from 9/1 deadline for biopolymer benchmarking