2020-06-02 Wagner Dotson PRP tutorial Meeting notes

Date

Jun 2, 2020

Participants

@Jeffrey Wagner
@David Dotson

Discussion topics

Notes

Notes

Getting added to namespace
- https://nautilus.optiputer.net/
- David registered using david.dotson@openforcefield.org
- Jeff added david to openforcefield namespace
- David joined rocketchat
Inside deployment.yaml
- Tolerations are special resources that are preemptible
Deployment types
- “Deployment2” → pod name openff-qca2 → 32 cores, 48 GB RAM
- “Deployment3” → pod name openff-qca3 → 16 cores, 24 GB RAM (easier to schedule, but too few resources for jobs like hessians w/ >25 heavy atoms)
- If changing manager/deployment CPUs or RAM, coordinate changes in both manager.yaml and deployment.yaml, and then delete and remake appropriate manager.yaml secret
- Deployments have tons of tolerations allowing them to run on pre-emptible resources that normal jobs can’t use, also have nice priority. This means our jobs will die more often, but we should be comfortable requesting tons of resources. Just make sure we use what we request, or scale down if not!
Managers run this docker image: https://hub.docker.com/r/molssi/qcarchive_worker_openff
Magic incantations
- python cleanup_idle_jobs.py “<paste the grafana resource table here>”
- kubectl create secret generic manager2-config-yaml --from-file=manager.yaml
- kubectl apply -f deployment.yaml
- kubectl delete secret manager2-config-yaml
- kubectl get deployment openff-qca2
- kubectl delete pods openff-qca2-68d8767d74-hqrwl openff-qca2-68d8767d74-9jf8q openff-qca2-68d8767d74-w8jkl
  - cleanup_idle_jobs.py will provide this list if you paste in the table from the grafana monitoring page (lists all pods with less than 1 CPU used)
  - In a deployment, this command has the effect of restarting (not permanently deleteing) the selected pods. They will be assigned a new UUID-ish name, but pods will be spun up again to reach the deployment’s replcas request
  - If you run scale about 15 seconds after this, the pods that were restarting will be selected for removal
- kubectl scale --replicas=10 deployment/openff-qca2
- kubectl logs openff-qca2-68d8767d74-f5q47
Montitoring dashboard
- https://grafana.nautilus.optiputer.net/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&refresh=30m&var-datasource=default&var-cluster=&var-namespace=openforcefield&from=now-3h&to=now
kubectl scale may have an autoscale option based on utilization
Maybe an API to get the same info (I though they were going to shut down grafana):
- JeffWagner12:42 PM
  Hey all, I've been poking around with Thanos, seeing if I can replace my favorite grafana monitoring graphs. Just made some progress in reproducing the cumulative CPU usage plot [1]. It's a bit finnicky, so sometime I have to retry/refresh a few times to get it.
  I found the autocomplete function to be useful in learning how I can construct Thanos queries. There are some general cookbooks [2], but each deployment seems to use its own keywords, so I had to do some digging. You can get a hint at the syntax by, for example, running `{namespace="openforcefield" }` as a query, selecting to see the results in the console, and looking at the keywords that the results use. [3]
  [1] https://thanos.nautilus.optiputer.net/graph?g0.range_input=6h&g0.step_input=150&g0.stacked=1&g0.moment_input=2020-01-14 20:29:34&g0.max_source_resolution=0s&g0.expr=node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{namespace="openforcefield",prometheus="monitoring/k8s"}&g0.tab=0
  [2] https://prometheus.io/docs/prometheus/latest/querying/examples/
  [3] https://thanos.nautilus.optiputer.net/graph?g0.range_input=6h&g0.step_input=150&g0.moment_input=2020-01-14 20:35:50&g0.max_source_resolution=0s&g0.expr={namespace="openforcefield"}&g0.tab=1

deployment2 files:

manager.yaml

common:
 adapter: pool
 tasks_per_worker: 2
 cores_per_worker: 32 # change this 
 memory_per_worker: 48 # in GB
 max_workers: 1 # keep this at 1
 scratch_directory: "$TMPDIR"

server:
 fractal_uri: api.qcarchive.molssi.org:443
 username: mobleylab3
 password: <Contact wagner or dotson>
 verify: True

manager:
 manager_name: PacificResearchPlatform2
 queue_tag: null
 log_file_prefix: null
 update_frequency: 30
 test: False

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: openff-qca2
  labels:
    k8s-app: openff-qca
spec:
  replicas: 75
  selector:
    matchLabels:
      k8s-app: openff-qca
  template:
    metadata:
      labels:
        k8s-app: openff-qca
    spec:
      priorityClassName: nice
      tolerations:
        - key: "nautilus.io/suncave"
          operator: "Exists"
          effect: "NoSchedule"
        - key: "nautilus.io/wave"
          operator: "Exists"
          effect: "NoSchedule"
        - key: "nautilus.io/science-dmz"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
      - image: molssi/qcarchive_worker_openff
        name: openff-pod
        resources:
          limits:
            cpu: 32
            memory: 48Gi
            #memory: 30Gi
          requests:
            cpu: 32
            memory: 48Gi
            #memory: 30Gi
        volumeMounts:
          - name: manager-config-secret
            mountPath: "/etc/qcfractal-manager"
            readOnly: true
      volumes:
        - name: manager-config-secret
          secret:
            secretName: manager2-config-yaml

Meetings

2020-06-02 Wagner Dotson PRP tutorial Meeting notes

Date

Participants

Discussion topics

Action items

Decisions