2023-11-14 alchemiscale : dev group meeting notes

living networks

• JC : moving toward a consistent strategy of retrospective analysis and prospective use in the same AlchemicalNetwork

  • JC : what is the extend of scope for this?

  • DD : we intend to touch whatever we need to to enable this usage pattern

• JC : RG do you see any places we can contribute to the ecosystem instead of shoving things asapdiscovery?

  • RG : cinnabar probably of greatest interest in performing analysis

    • currently leaning on networkx, but idea is you can swap out a network graph library if needed for larger networks

  • JC : What is primary API for using cinnabar object models? Via API for cinnabar objects, or via {to|from}_{networkx|...}

    • RG : To avoid reinventing the API for graphs, aim to use {to|from}_{networkx|...}

    • JC: Need to standardize naming of node/edge data and units, since these are critical to operation of DiffNet/NetBFE and was poorly documented, leading to errors

• DD: does gufe have strategy in the data model

  • RG: I’ve banned strategy as a word since it isn’t clear what we are doing, lots of things have strategy

  • DD: Alchemistry strategy means “how we allocate compute power on a network (how much we compute do we spend on a leg)” This is what we want to work on next in 2024, this will touch things like cinnabar since it needs to be able to evaluate the current estimates to make choices from the uncertainties to decide what to run

    • JC: we will want to use weights so that we can do this in an async manner instead of priority since then different parts can be running and update weights

    • DD: Yes weights can be adjusted so if something needs to be done “sooner” we can weight it higher

    • JC: Some of the logic for this is re-useable, is there a way to re-use some of these components

      • MH : might need to create an object model for weights that these Strategys operate with

        • JC : it may make sense for this to live in cinnabar then?

        • MH: I think we can add weights to our cinnabar object model and use that

        • JC: Is cinnabar a factory model

        • RG: Yes, would you have a network where not every edge is computed?

        • JC: That would be possible so we should support that <I think I captured this correctly>

        • DD: Since cinnabar has this object model, given an alchemical network with results, could it produce edge weights, so does it make sense for it be a strategy engine?

        • JC: The object model is capable of storing this, but another package will execute it and then can update it

        • DD: So alchemiscale will have the bits that knows how to use the weight outputs from cinnabar

        • IK: Who computes the weights? Which package has that function

        • JC: cinnabar will have the object model and some implementation for calculating weights, but other packages can use the same object model and then do their own thing but use the same API, so we need a base class and one implementation, then we are off to the races, so it will be captured in the cinnabar API, but other packages can implement it

        • RG: I will have this done in two weeks by the next meeting

        • JC & DD: Are you sure about that? Only do it if it makes sense for your timeline

        • RG: <nods affirmatively>

        • DD: RG do you have what I need?

        • RG: I will ping DD and JC for review

DD: We have user identity, compute identity and it would be nice to have an admin idenity that gives and admin api and client, where you might be managing one (or more than one) alchemical instance and have some nice api points to add guard rails. Right now we use scopes to control access, so right now we can control compute access by spinning up workers that target the scope, so other orgs can use the alchemiscale instance without starving compute resoruces. This doesn’t meter service, so it doesn’t help if the worker has both scopes, since it will just grab jobs and not track which ones it runs

• DD & MH this will probably have to happen server side since the workers don’t keep the state

• JC: Something like just preventing one group/scope dominating compute resources would be the minium, it doesn’t have to be fancy right now

• DD: We do have the notion of a weight we an task is actioned on a task hub. tasks to have a priority exposed on the task, once we have it implemented, we could allow users to set the priority higher and then they will strictly pick the higher priority tasks, maybe on the scope level we can set what the max priority can be. So the ASAP instance could allow their tasks to be higher.

• JC: We need to still do something to make sure that people always get some share of the compute, even if the other group has priority

• DD: Okay we have another mechanism that might work, there is a weight on the task hub, so an alchemical instance could have many task hubs going, we could make a task hub have higher weights

• MH: Just to clarify, will resources get shared or if a task hub has a higher weight, will it always get the compute before a task hub of lower weight?

• DD : could create a “fair share” service that operates on TaskHub weight based on utilization