2022-01-19 Industry benchmarks meeting notes

Manuscript

• DH – have everything we need, data is there; will send out to partners after another round of reading

  • had one read-through already with comments to LD

• DH – one note: we are trying to make clear the selection criteria; may be the at some partners didn’t really follow too closely

  • We did issue this originally for both public and private molecules: Call for Public Compounds

  • And the validation step combined with what each partner chose functions as the selection criteria

Partner benchmarking 2022

• DD – what would you like to see for benchmarking in 2022?

• DH – initially thought it would be good to repeat the benchmark every year, but not sure we would get more useful conclusions by just repeating at the moment

  • might need to do something more specific, like looking at one compound pattern, e.g. charged molecules, biaryls, torsiondrives, etc.

  • something to probe more non-bonded interactions

  • in short, should be something different than Season 1

• DH – know from another partner that it is important we show that OpenFF has made progress, use of the toolkit internally

  • probably a better investment in time from partners this year to get OpenFF tooling applied by partners, integrated into actual workflows

  • if a second benchmarking season leads to a workflow, that would be a positive outcome that gives us both

    • DD – this is key; a season 2 is an opportunity to get internal use widespread among partners, and forces development team to not build bespoke tools to support a protocol, but instead engineer the components needed for general use that are then used in a protocol

• JW – I wonder which format for these tools makes sense?

  • we have Python library components, but then also need CLI-based tooling

  • could the steps talk to each other with serialized OFFmols, but have users input molecules or output molecules as SDF

• JW – benchmarking v2 is probably 3rd or 4th in big deliverables this year

  • we need to get rosemary out first, though

  • but do want to draw a distinction between benchmarking and CLI tools

  • if we make CLI tools for benchmarking v2, it becomes easier, but perhaps overfit to the problem

• DH – perhaps not the prerogative of OpenFF to build tools? Developing library components priority, partners can use to build tools

  • DD – but if they don’t know how to build tools, this presents a problem for all parties

  • DD – I had a good time working with XLucas, and I think this was productive. I wonder if we want to have a tooling effort where we work WITH the partners to design a CLI. By actively engaging, we can get direct feedback on which functionality is most useful. I’d be interested in that kind of work.

  • DH – That’s a good idea. It could be framed as a benchmark, but focused on one partner.

  • DD – This would be great

• JW – I’d be interested in having the infrastructure team do some limited pharma user support/development, but I’m not sure when we’ll have time. How is timeline looking for your F@H work?

  • DD – I could spearhead this in late 2022, run it similar to the benchmarking project. Stated goal would be to do “benchmarking season 2”, but really the design would emphasize the reusability of components and aim for people to adopt them for internal use.