DD – Planning to do this like a more formal project later this year/next year. Hoping to get feedback on CLI entry points and other details to ensure we make something useful.
| Github link macro |
|---|
| link | https://github.com/openforcefield/openff-benchmark |
|---|
|
DD – The season 1 machinery is very purpose-built.
JW – We’ve had three goals:
1. Benchmark our FF against other FFs
2. Make CLI tools that users want
3. Have a slick web browser/visualization tool that hooks up to QCArchive and lets people interactively see the results of benchmarking
JW – We’ve tried various times at going after these goals and failed due to a lack of defined scope and lack of staekholders. But the “industry benchmarking season 1” project actually succeeded at engaging with users and keeping to a project plan, and accomplished goal #1 above.
DD - we expect user to write their own script and uses OpenFF components.
DD – No GUIs, but maybe expanding past the Python library to a CLI. In a season 2, we’d focus on expanding the CLI to handle more user needs, but we still wouldn’t make a GUI or a mouse-based app.
DH – from previous discussions, we’d said that the next season would be more 1:1 with partners to develop customized tools. One risk is that partners may be in an advantage position, becauase we could basically end up making custom steps for their specific workflows.
DD - We could keep from getting absorbed into a single partner’s org in this process by following the project planning workflow, and requiring stakeholders to follow a defined structure for their input.
DN - Great if we can do a meeting where people can tell what they expect for the product, to have a clear direction to follow.
DH - sometimes partners don’t state what they expect, so also good if we steer in the direction from the OpenFF point of view.
JW - like the idea to have written statement for communication and expectations. In the previous season we learned how important it is to have clear communication, and how much value we can add by making every message thorough.
DN - We should educate stakeholders about what to expect from a benchmarking product and the different phases, so expectations would be aligned with less frustration from our side.
JW - Timing of season 2 will depend upon PL-benchmarking, so a bit uncertain. Thus the starting date might be quite flexible.
DD - Right now the focus is on the manuscript, we would like season 2 to be quite different from season 1. Do you think there’s urgency from the partners to do a season 2?
DH – I don’t think there’s a sense of urgency for season 2. People aren’t really using it for internal work.
DD – Then the CLI would seem to be higher priority, to get our things into their workflows at all. So if we were trying to decide on whether to initiate a season 2 campaign in the fall, or a product development project for the CLI, then the CLI would be higher value. This is because we need to get our tools in use if we want to secure our funding/support stream.
DH – What would CLI actually mean? Like, specific functionality?
DD – For example, taking a set of molecules and producing parameterized systems for a variety of MD engines. Once we have a better understanding of industry workflows we can expand to meet their needs.
DH – So, the functionality is not really known right now, but you’d find it out.
DD - The CLI is a product-project, so we will ask partners to engage and give their feedback.
JW - Do Janssen use binding FE calculation? Because I have this idea that we shouldn’t make a CLI since OpenFE will do it for us. What would be a good use of time would be to coordinate with them rather than overlap.
DN - DH said that running FE calcs is possible if we have the pdb structure, but a lot of mols they don’t because they are instrinsically disordered. As postdoc I realized that I could go from conformation to conformation with a low barrier. Would be nice if we could allow this kind of calculations.
JW - What fraction of your project would benefit from having calculations that accounts for highly disordered proteins?
JW – A CLI tool that gives people parameterized ligands wouldn’t be useful in isolation
DD – We could make something like the GROMACS suite, where people come with a solvated box with a protein+ligand, and we could assign parameters to that.
DD – DN, what would you think of us doing a joint CLI project with OpenFE?
DN – It would be possible to coordinate the projects, and it would be mutually beneficial for both initiatives.
DD – Would it be better to have the projects work together, or have their sepaate efforts coordinated by you, DN?
DN – I’m happy to do the coordination, with infrequent direct coordination meetings. So tasks could be assigned to the different teams and people could work on them separately.
DD are they planning to develop CLI tools in the near future?
DN probably in September.