Versions Compared

Old Version 1

changes.mady.by.user Jeffrey Wagner

Saved on

compared with

New Version Current

changes.mady.by.user Jeffrey Wagner

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Participants

Discussion topics

Item

Notes

General updates

  • JW –

    • Something about our docker subscription lapsed, so I’ll need to move the QC worker docker images and conda env yamls to somewhere else - likely ghcr (github container registry?). We currently use dockerhub. Happy to do a screenshare on that if you’d like to see how it works.

      • BW – I’d be interested to see how this works. I have an image for a different project that I’ve put on dockerhub. But it looks like the limits are low.

    • Next Monday is labor day, then the two mondays after that I’m in europe. Could we cancel and then schedule a meeting if needed?

      • BW – Yes, let’s cancel and we can reschedule if needed.

  • BW –

    • Benchmarking runners/actions

    • JW – We can discuss major design stuff/decisions at the internal benchmarking meeting, but what are some details that we can work out now?

    • BW – Migration to OFF GH org and final naming?

      • JW – Yes to migration, maybe we call it “yammbs-dataset-submission”

      • BW – Making this now

    • BW – How to use OE license in secrets?

      • JW – Oh, I need to enable access to the org license (done)

    • Call for user stories for Weds?

    • Weds meeting items:

      • Provenance requirements?

      • Set a timeline for versioning/stability? (eg “this is unstable for the first 3 months, don’t expect to be able to reproduce results from that time…”)

      • Should zenodo records document how to use themselves? (eg “create an env from this yaml gere by running …, then load the sqlite file here…”)

      • Format of uploads? (one big zip file? A bunch of loose files?) BW will check whether it’s possible to fetch a single file from a larger zenodo record)

      • Manual review before zenodo publication? (Automation can create a draft and require human review before cutting)

      • Create an index mapping submissions/folders/PRs to zenodo record?

        • JW – Could have a table in the GH repo where each submission is linked to its PR, folder, and zenodo. And links from zenodo back to the GH repo.

        • BW – Maybe jinja could update this table. Would be tough to have it in the main README without rewriting it each time. So maybe a big index in a different folder/file.

      • Validation machinery - make sure input is valid-ish

      • How to trigger dataset computation and upload?

        • GH comments (current version)

        • PR merging (would require a rework)

        • Relabeling (what qca-dataset-submission does)

      • Processes for…

        • Adding new QC datasets

        • Approving behavior changes

      • Deduplication logic/what to do if files would be overwritten in GH

      • If we use Zenodo, what should actually be committed to GH in a PR (or do we just use PRs for decision/review process?)

      • What’s the lifecycle?

        • User submits PR

        • PR gets approved by human

        • CI 1 is kicked off using some human trigger

        • CI 1 runs YAMMBS job

        • CI 1 prepares zenodo draft

        • CI 1 updates index of jobs

        • CI 1 finishes, comments on PR to tell user next step

        • The draft is somehow published

          • Human goes to zenodo and publishes?

          • Triggered by merging the PR?

          • Triggered by comment?

    • BW will think through this workflow some more and put a diagram together for weds

    • BW will prepare a test run and zenodo draft for wednesday

Trello

https://trello.com/b/dzvFZnv4/infrastructure

Action items

  •  

Decisions