Page Comparison

Intro

We keep getting weird issues with Psi4, which are only magnified by bigger molecules. Here, I examine how psi4 is utilizing memory to see if there are any glaring issues.

These experiments are done using an optimization that consistently fails using normal procedures. The optimization in QCArchive is 34754174. The “molecule” in question is:

Procedures/Scripts

Using psi4 1.4a3.dev63+afa0c21

I run the following in two terminals, one is the main psi4 running:

Code Block
bash run-local.sh 34754174

and the other processes the output and generates a plot:

Code Block
bash makeplot.sh 34754174

this will generate everything in a folder named 34754174. For the impatient, my plotting terminal actually runs the disk checker (see below), and the plotter periodically:

Code Block

language	bash

while sleep 60 ; do bash makeplot.sh 34754174 ; done &
while sleep 1; do
    if [ ! -e 34754174/*_psi_scratch ]; then
        continue;
    fi
    echo -n "$(date +%s ) "; du -s 34754174/*_psi_scratch | awk '{print $1}';
done >> 34754174/disk.dat &

In effect, running the first and last box in separate terminals should get you the plots I show below. The plot is 34754174/mem.png.

Running

Using this modified script from David Dotson to run a task (name it run-local.py):

Code Block

language	py

#!/usr/bin/env python

import sys
import os
import pprint
from qcfractal.interface import FractalClient
from qcengine import compute_procedure

id = sys.argv[1]

# os.makedirs(id, exist_ok=True)

cl = FractalClient.from_file()
task = cl.query_tasks(base_result=id)[0]
print(task)

with open('task.json', 'w') as f:
    f.write(pprint.pformat(task.json()))

task.spec.args[0]['input_specification']['keywords']['scf_mem_safety_factor'] = 0.5
# task.spec.args[0]['input_specification']['keywords']['df_ints_num_threads'] = 1
# task.spec.args[0]['input_specification']['keywords']['diis'] = True
# task.spec.args[0]['input_specification']['model']['method'] = 'mp2'

result = compute_procedure(*task.spec.args, local_options={"memory": "16.0", "ncores": 8, "scratch_directory": os.getcwd() })
print("Computation done.")
print(result)

with open('result.json', 'w') as f:
    f.write(pprint.pformat(result.json()))

Noting that I added local_options={"memory": "16.0", "ncores": 8, "scratch_directory": os.getcwd() } to be able to control the cpu/mem given to psi4 and the scratch directory since my local machine uses a tmpfs as /tmp. There is also some record of how/what settings I was testing.

I drive this python script with the following bash script, which does the initialization and memory recording (name this run-local.sh):

Code Block

#!/bin/bash
id=$1
d=$PWD

finish() {
	cd $d
	kill $(jobs -p)
}
export -f finish

mkdir -p $id
cd $id

trap finish SIGINT 
trap finish SIGCHLD 
python3 -u ../run-local.py $id &
pidstat -p ALL -l -C "$id" -r -h -H -u 1  | grep python | tee stat.log

Ideally the traps are there to stop pidstat once the background psi4 job finishes, but it’s a bit finicky (and I haven’t had too many jobs run until completion ). In any case, the pidstat command will create a memory log of the main process (e.g. the qcengine/geomeTRIC process) and the psi4 subprocess calls.

Plotting

I then have a similar setup for driving the plots. Here is the bash driver (name this makeplot.sh):

Code Block

#!/bin/bash
id=$1
d=$PWD
cd $1

awk '/psi4/ {print $1 " " $13}' stat.log  | head -n -1 > psi4.dat
awk '/run-local/ {print $1 " " $13}' stat.log  | head -n -1 > main.dat
awk '/psi4/ {print $1 " " $22}' stat.log  | tr -d "[A-Z]" | head -n -1 > memlimit.dat

if [ -f out.log ] ; then
        awk '/Cache is/ {print $1 " " $4}' out.log  > cache.dat
fi

python3 $d/plot.py

cd $d

I do the head -n -1 funny business in case buffering produced an incomplete last line, which would make the numpy loader bork.

The python plotter (name this plot.py):

Code Block

language	py

import matplotlib.pyplot as plt
import numpy as np
import os

fig = plt.figure(figsize=(12*4, 5), dpi=200)
ax = fig.add_subplot(111)

x, y = np.loadtxt("main.dat").T
xmin = x.min()

ax.plot(x-xmin, y / 1024 ** 2, "b", lw=0.3, label="geometric")

x, y = np.loadtxt("psi4.dat").T
ax.plot(x-xmin, y / 1024 ** 2, "r", lw=0.3, label="psi4")

x, y = np.loadtxt("memlimit.dat").T
ax.plot(x-xmin, y, "k", label="mem specified")

if os.path.exists("cache.dat"):
    x, y = np.loadtxt("cache.dat").T
    try:
        size_x = len(x)
        if size_x > 0:
            ax.plot(x-xmin, y / 1024 ** 3, "g", lw=0.3, label="B cache")
    except Exception:
        pass
if os.path.exists("disk.dat"):
    x, y = np.loadtxt("disk.dat").T
    try:
        size_x = len(x)
        if size_x > 0:
            ax.plot(x - xmin, y / 1024 ** 2, "g", lw=0.3, label="Disk space")
    except Exception:
        pass
ax.set_xlabel("Time (seconds)")
ax.set_ylabel("Memory (GB)")
ax.legend(loc=3)
ax.tick_params(labeltop=False, labelright=True)
ax.grid(axis='y')
fig.tight_layout()
fig.savefig("mem.png")

As can be seen from the plotter, I decided to record disk usage of the psi4 scratch about half way through the experiments. To enable this, I ran this in the background:

Code Block

language	bash

while sleep 1; do
    if [ ! -e 34754174/*_psi_scratch ]; then
        continue;
    fi
    echo -n "$(date +%s ) "; du -s 34754174/*_psi_scratch | awk '{print $1}';
done >> 34754174/disk.dat &

Not pretty since I wrote it as a quick one-liner, but works.

I originally wanted to look into the B matrix cache of geomeTRIC, since it is unbounded and for some other experimental jobs I ran into the warning seen below. I used a locally modifed geomeTRIC, which prints out the B matrix cache size whenever it is accessed (https://github.com/leeping/geomeTRIC/blob/master/geometric/internal.py#L1761):

Code Block

language	py

    def wilsonB(self, xyz):
        """
        Given Cartesian coordinates xyz, return the Wilson B-matrix
        given by dq_i/dx_j where x is flattened (i.e. x1, y1, z1, x2, y2, z2)
        """
        global CacheWarning
        t0 = time.time()
        xhash = hash(xyz.tostring())
        ht = time.time() - t0
        if xhash in self.stored_wilsonB:
            ans = self.stored_wilsonB[xhash]
            return ans
        WilsonB = []
        Der = self.derivatives(xyz)
        for i in range(Der.shape[0]):
            WilsonB.append(Der[i].flatten())
        self.stored_wilsonB[xhash] = np.array(WilsonB)
        ##################################################################
        mem = 0.0
        for stored,mat in self.stored_wilsonB.items():
            mem += mat.itemsize * mat.size
        logger.info("{:20.4f} Cache is {:f}\n".format(time.time(), mem))
        ##################################################################
        if len(self.stored_wilsonB) > 1000 and not CacheWarning:
            logger.warning("\x1b[91mWarning: more than 1000 B-matrices stored, memory leaks likely\x1b[0m\n")
            CacheWarning = True
        ans = np.array(WilsonB)
        return ans

However, since I am able to roughly account for this by recording the memory of the entire QCEngine/geomeTRIC process in these experiments, I avoid its use.

Segfault hunting

This script will run many psi4 processes in parallel (here -P 8), and kill them if they run for more than 5 seconds:

Code Block

#!/bin/bash
id=$1
d=$PWD

finish() {
	x=$(jobs -p)
	[ "$x" ] && { echo $x | xargs -n 1 pkill -P ; }
}

run() {
	id=$1
	i=$2
	rm -rf ${id}.${i}
	mkdir -p ${id}.${i}
	cd ${id}.${i}

	trap finish SIGINT

	python3 -u ../run-local.py $id &> output.log &
	sleep 5

	x=$(jobs -p)
	[ "$x" ] && { echo $x | xargs -n 1 pkill -P ; kill $x ; }

	rm -rf tmp*
	echo "$(date) ${id}.${i} done; failed? $(if grep -q Failed output.log ; then echo -n yes ; else echo -n no ; fi )"
	cd ../
}
export -f finish
export -f run

seq 10000 | xargs -P 8 -n 1 -I {} sh -c "run $1 {}"

Some magic had to be done to kill the psi4 subprocess as it wasn’t working with the older version. Should update the other script to use pkill -P which says to kill the children of the process. It is still finicky for whatever reason at some times, so I have the mop command below.

This is my psi4 mop since it appears some psi4 commands don’t get cleaned up properly:

Code Block

language	bash

while sleep 5 ; do 
    ps aux | grep psi4 | grep msgpack | sed "s/:/ /g" | awk '{if ($11 > 0) print $2 }' | xargs -r kill -9
done

which basically parses ps and kills any psi4 process that has run longer than a minute (noting that this is CPU time, not wall time). Not sure if there is a better way to query runtime on processes.

Finally, I run

Code Block
bash run-local.sh 34754174 \|& tee segfaults.log &

and then count the number of failures via

Code Block
grep -c yes segfaults.log # for the ones that fail wc -l < segfaults.log # for the total tries

Experiments

SCF_MEM_SAFETY_FACTOR

Setting SCF_MEM_SAFETY_FACTOR is affected by number of cores. If I set 0.38 for 2 cores, the memory usage will be higher if I run the same thing with 8 cores.

This is the error if the safety factor is set too low (.20 when mem=2GB):

Code Block

FailedOperation(error=ComputeError(error_type='unknown', error_message='geomeTRIC run_json error:\nTraceback (most recent call last):\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/run_json.py", line 214, in geometric_run_json\n    geometric.optimize.Optimize(coords, M, IC, engine, None, params)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1331, in Optimize\n    return optimizer.optimizeGeometry()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1293, in optimizeGeometry\n    self.calcEnergyForce()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1002, in calcEnergyForce\n    spcalc = self.engine.calc(self.X, self.dirname)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 874, in calc\n    return self.calc_new(coords, dirname)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 866, in calc_new\n    raise QCEngineAPIEngineError("QCEngineAPI computation did not execute correctly. Message: " + ret["error"]["error_message"])\ngeometric.errors.QCEngineAPIEngineError: QCEngineAPI computation did not execute correctly. Message: QCEngine Unknown Error: Traceback (most recent call last):\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/schema_wrapper.py", line 411, in run_qcschema\n    ret_data = run_json_qcschema(input_model.dict(), clean, False, keep_wfn=keep_wfn)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/schema_wrapper.py", line 558, in run_json_qcschema\n    val, wfn = methods_dict_[json_data["driver"]](method, **kwargs)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/driver.py", line 717, in gradient\n    wfn = procedures[\'gradient\'][lowername](lowername, molecule=molecule, **kwargs)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/proc.py", line 2383, in run_scf_gradient\n    ref_wfn = run_scf(name, **kwargs)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/proc.py", line 2288, in run_scf\n    scf_wfn = scf_helper(name, post_scf=False, **kwargs)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/proc.py", line 1568, in scf_helper\n    e_scf = scf_wfn.compute_energy()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 83, in scf_compute_energy\n    self.initialize()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 193, in scf_initialize\n    self.initialize_jk(self.memory_jk_, jk=jk)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 128, in initialize_jk\n    jk.initialize()\nRuntimeError: \nFatal Error: SCF::DF: Disk based algorithm requires 2 (A|B) fitting metrics and an (A|mn) chunk on core.\n         This is 2Q^2 + QNP doubles, where Q is the auxiliary basis size, N is the\n         primary basis size, and P is the maximum number of functions in a primary shell.\n         For this problem, that is 374074280 bytes before taxes,534391828 bytes after taxes. \n\nError occurred in file: /scratch/psilocaluser/conda-builds/psi4-multiout_1609773118680/work/psi4/src/psi4/libfock/DiskDFJK.cc on line: 727\nThe most recent 5 function calls were:\n\npsi::DiskDFJK::preiterations()\n\n\n'))

The formatted error is:

Code Block

geomeTRIC run_json error:
Traceback (most recent call last):
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/run_json.py", line 214, in geometric_run_json
    geometric.optimize.Optimize(coords, M, IC, engine, None, params)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1331, in Optimize
    return optimizer.optimizeGeometry()
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1293, in optimizeGeometry
    self.calcEnergyForce()
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1002, in calcEnergyForce
    spcalc = self.engine.calc(self.X, self.dirname)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 874, in calc
    return self.calc_new(coords, dirname)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 866, in calc_new
    raise QCEngineAPIEngineError("QCEngineAPI computation did not execute correctly. Message: " + ret["error"]["error_message"])
geometric.errors.QCEngineAPIEngineError: QCEngineAPI computation did not execute correctly. Message: QCEngine Unknown Error: Traceback (most recent call last):
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/schema_wrapper.py", line 411, in run_qcschema
    ret_data = run_json_qcschema(input_model.dict(), clean, False, keep_wfn=keep_wfn)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/schema_wrapper.py", line 558, in run_json_qcschema
    val, wfn = methods_dict_[json_data["driver"]](method, **kwargs)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/driver.py", line 717, in gradient
    wfn = procedures['gradient'][lowername](lowername, molecule=molecule, **kwargs)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/proc.py", line 2383, in run_scf_gradient
    ref_wfn = run_scf(name, **kwargs)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/proc.py", line 2288, in run_scf
    scf_wfn = scf_helper(name, post_scf=False, **kwargs)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/proc.py", line 1568, in scf_helper
    e_scf = scf_wfn.compute_energy()
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 83, in scf_compute_energy
    self.initialize()
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 193, in scf_initialize
    self.initialize_jk(self.memory_jk_, jk=jk)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib//python3.7/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 128, in initialize_jk
    jk.initialize()
RuntimeError: 
Fatal Error: SCF::DF: Disk based algorithm requires 2 (A|B) fitting metrics and an (A|mn) chunk on core.
         This is 2Q^2 + QNP doubles, where Q is the auxiliary basis size, N is the
         primary basis size, and P is the maximum number of functions in a primary shell.
         For this problem, that is 374074280 bytes before taxes,534391828 bytes after taxes. 

Error occurred in file: /scratch/psilocaluser/conda-builds/psi4-multiout_1609773118680/work/psi4/src/psi4/libfock/DiskDFJK.cc on line: 727
The most recent 5 function calls were:

psi::DiskDFJK::preiterations()

0.25 works, but goes beyond the 2GB limit.

Here is an example using a safety factor of .20, and .75 (the default) with 4GB

DIIS

Now, I know DIIS saves state, so it might be the cause of the ramp. Turning off, we see exactly no change in memory behavior, other than the iterations are slower and more are needed:

DFT vs HF

Now, B3LYP/DZVP should be pretty stable; is the dispersion affecting this at all? Tried it, no. Just for fun; does using HF change anything? YES!

How about wb97x?

Yes, there is a ramp up (this one failed due to no disk space; currently working with a 18 GB space). However the memory requirements seem to be somewhat higher than b3lyp.

How about blyp? Yes

How about M06? Yes

How about TPSS? Yes

How about MP2? No (but the memory jumps way up after the HF finishes; ran out of disk space causing it to crash)

DF_INTS_NUM_THREADS

What about DF_INTS_NUM_THREADS since the documentation says it could affect memory issues? Setting to 1 does seem to affect memory (note that all calculations are done using 8 cores):

There seems to only be a constant offset; the difference in the internal iterations is about the same as previous experiments.

Direct vs DF

Tried using SCF_TYPE as DIRECT , which turns off density fitting, recommended by Ben Pritchard. Documentation is https://psicode.org/psi4manual/master/scf.html#eri-algorithms

It is slow, and I am not patient enough to let this finish on my desktop as I have other things in the queue:

We see a slight overall ramp which is consistent with the CORE algorithm shown below. Will try again later when things are idle.

Let a full DIRECT run finish

Full Runs

CORE vs DISK algorithm

At this point; I am not sure what other options could affect memory. There seems to be a consistent ramp in memory for DFT; not sure if this is normal or not.

Now, let’s try to take the optimization out many steps using typical settings of 8 cores, 30GB. This was using the CORE algorithm, meaning it could be done completely in memory. We have:

and it does fail after 76 iterations with:

Code Block

FailedOperation(error=ComputeError(error_type='unknown', error_message='geomeTRIC run_json error:\nTraceback (most recent call last):\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/run_json.py", line 214, in geometric_run_json\n    geometric.optimize.Optimize(coords, M, IC, engine, None, params)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1331, in Optimize\n    return optimizer.optimizeGeometry()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1298, in optimizeGeometry\n    self.calcEnergyForce()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1002, in calcEnergyForce\n    spcalc = self.engine.calc(self.X, self.dirname)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 874, in calc\n    return self.calc_new(coords, dirname)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 866, in calc_new\n    raise QCEngineAPIEngineError("QCEngineAPI computation did not execute correctly. Message: " + ret["error"]["error_message"])\ngeometric.errors.QCEngineAPIEngineError: QCEngineAPI computation did not execute correctly. Message: QCEngine Unknown Error: Unknown error, error message is not found\n'))

Code Block

geomeTRIC run_json error:
Traceback (most recent call last):
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/run_json.py", line 214, in geometric_run_json
    geometric.optimize.Optimize(coords, M, IC, engine, None, params)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1331, in Optimize
    return optimizer.optimizeGeometry()
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1298, in optimizeGeometry
    self.calcEnergyForce()
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1002, in calcEnergyForce
    spcalc = self.engine.calc(self.X, self.dirname)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 874, in calc
    return self.calc_new(coords, dirname)
  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 866, in calc_new
    raise QCEngineAPIEngineError("QCEngineAPI computation did not execute correctly. Message: " + ret["error"]["error_message"])
geometric.errors.QCEngineAPIEngineError: QCEngineAPI computation did not execute correctly. Message: QCEngine Unknown Error: Unknown error, error message is not found

No memory errors. My machine has 31GB of memory and additional swap space, so if all is well there should have been enough memory to keep going even if virtual memory grabbed more than 31GB (I note that there is ~2GB more virtual than resident on average, so nothing crazy going on here). Let's try using the DISK algorithm by setting the safety factor lower to .5 and memory to 16 GB. Realistically, this should have zero memory issues; even virtual memory shouldn’t hit the 16 GB limit. Here is what I have so far:

Died at 94, RIP.

Also, oddly, the memory spikes at iteration 75; this spike may have been the cause for the other failure where the memory fluctuations are much higher. Also notice that the memory tends to decrease in this run, whereas it tended to increase in the CORE algorithm. Quite a difference in behavior.

Checking the output, it looks like everything is perfect, but the energy of the last psi4 run is None and there is no output from it. Everything seems to point at a psi4 calculation failing to start. Will try plugging in the last schema manually and seeing if psi4 accepts it. Just looking at the last structure shows nothing remarkable.

It does finish successfully using HF:

Trying to load the last qcschema molecule into the task and running it, with some print statements in the QCE Psi4 harness. Noting that execute is full of context managers and no exception handling. First two iterations went swimmingly, will see how long it lasts.

Plot the results of the complete optimization starting from the failed one above (iteration 94)

Segfaults

After spending enough time on this, I noticed that there was a random crash upon starting a job. To investigate, I took the problem input from above (at iteration 94), and ran it for 5 seconds, 10000 times and recorded how many times it segfaulted. It segfaulted 62/10000 times so far (1%) but was near 3% at certain times. Something tells me it is thread related since there was a period of time when too many psi4 jobs were running and casing them all to run at 50% CPU. Anywho, I ran this after modifying QCEngine to dump any and all contents from the Psi4Harness https://github.com/MolSSI/QCEngine/blob/master/qcengine/programs/psi4.py#L217

Code Block

                    run_cmd = [
                        which("psi4"),
                        "--scratch",
                        str(tmpdir),
                        "--nthread",
                        str(config.ncores),
                        "--memory",
                        f"{config.memory}GB",
                        "--qcschema",
                        "data.msgpack",
                    ]
                    input_files = {"data.msgpack": input_model.serialize("msgpack-ext")}
                    try:
                        success, output = execute(
                            run_cmd, input_files, ["data.msgpack"], as_binary=["data.msgpack"], scratch_directory=tmpdir
                        )

                        print("SUCCESS IS", success)
                        print("OUTPUT IS\n", output)
                    except Exception as e:
                        print("BORK")
                        breakpoint()
                    if success:
                        output_data = deserialize(output["outfiles"]["data.msgpack"], "msgpack-ext")
                    else:
                        output_data = input_model.dict()

                    print("\n\nOUTPUT DATA IS\n", output_data)

It should be pretty clear which uncultured barbarian lines I added The point is the dump the output that psi4 outputs directly, then, just because I have no idea what is going on, check it after QCEngine has had a chance to get its hands on it. In this case, I was wise to check both versions:

Code Block

SUCCESS IS True
OUTPUT IS
 {'proc': <subprocess.Popen object at 0x7f0a78fe36d0>, 'stdout': '', 'stderr': 'forrtl: severe (174): SIGSEGV, segmentation fault occurred\nImage              PC                Routine            Line        Source             \nlibgdma.so         00007F60FDA7D0DC  for__signal_handl     Unknown  Unknown\nlibpthread-2.32.s  00007F61046D7830  Unknown               Unknown  Unknown\nlibstdc++.so.6.0.  00007F609C4B68C9  Unkno
wn               Unknown  Unknown\nlibstdc++.so.6.0.  00007F609C4B6AF8  _ZSt29_Rb_tree_in     Unknown  Unknown\ncore.cpython-37m-  00007F61007D16AE  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F61007E2542  Unknown               Unknown  Unknown\nlibomp.so          00007F609C60E153  __kmp_invoke_micr     Unknown  Unknown\nlibomp.so          00007F609C5B0CED  Unknown               Unknown  Unknown\nlibom
p.so          00007F609C5B4CF6  __kmp_fork_call       Unknown  Unknown\nlibomp.so          00007F609C5A39E2  __kmpc_fork_call      Unknown  Unknown\ncore.cpython-37m-  00007F61007E10F5  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F61007F1BA0  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F61007F0587  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F61009C9969  _ZN3psi5
VBase10in     Unknown  Unknown\ncore.cpython-37m-  00007F61009C86A9  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F6102540A4E  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F610252878D  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F610254DC43  Unknown               Unknown  Unknown\ncore.cpython-37m-  00007F60FEA88E8A  Unknown               Unknown  Unknown\ncore.cpy
thon-37m-  00007F60FE6CF66E  Unknown               Unknown  Unknown\npython3.7          0000559523213504  _PyMethodDef_RawF     Unknown  Unknown\npython3.7          0000559523215B9A  Unknown               Unknown  Unknown\npython3.7          00005595231F3191  PyObject_Call         Unknown  Unknown\npython3.7          00005595232525A4  Unknown               Unknown  Unknown\npython3.7          00005595231F2A7A  Unknown    
           Unknown  Unknown\ncore.cpython-37m-  00007F60FE6C95BB  Unknown               Unknown  Unknown\npython3.7          000055952322DF3B  _PyObject_FastCal     Unknown  Unknown\npython3.7          000055952322EA79  Unknown               Unknown  Unknown\npython3.7          00005595232A174A  _PyEval_EvalFrame     Unknown  Unknown\npython3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown\npython3.7  
        00005595231E21F0  _PyFunction_FastC     Unknown  Unknown\npython3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown\npython3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown\npython3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown\npython3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown\npython3.7          00005595231E0932  _PyEval_EvalCo
deW     Unknown  Unknown\npython3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown\npython3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown\npython3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown\npython3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown\npython3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown\npython3.7     
     00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown\npython3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown\npython3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown\npython3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown\npython3.7          000055952322C700  _PyFunction_FastC     Unknown  Unknown\npython3.7          000055952322E8E8  Unknown          
     Unknown  Unknown\npython3.7          000055952329DF95  _PyEval_EvalFrame     Unknown  Unknown\npython3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown\npython3.7          000055952322C700  _PyFunction_FastC     Unknown  Unknown\npython3.7          000055952322E8E8  Unknown               Unknown  Unknown\npython3.7          00005595232A174A  _PyEval_EvalFrame     Unknown  Unknown\npython3.7        
  00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown\npython3.7          00005595231E1B49  PyEval_EvalCodeEx     Unknown  Unknown\npython3.7          00005595232C1C2B  PyEval_EvalCode       Unknown  Unknown\npython3.7          000055952332E3FF  Unknown               Unknown  Unknown\npython3.7          00005595233389C7  PyRun_FileExFlags     Unknown  Unknown\npython3.7          0000559523338B9C  PyRun_SimpleFileE   
  Unknown  Unknown\npython3.7          0000559523339129  Unknown               Unknown  Unknown\npython3.7          000055952333927C  _Py_UnixMain          Unknown  Unknown\nlibc-2.32.so       00007F61043E2EBB  __libc_start_main     Unknown  Unknown\npython3.7          00005595232ADC99  Unknown               Unknown  Unknown\ndouble free or corruption (fasttop)\n', 'outfiles': {{{msgpack binary stuff}}}
  
  OUTPUT DATA IS
 {'id': None, 'schema_name': 'qcschema_input', 'schema_version': 1, 'molecule': {'schema_name': 'qcschema_molecule', 'schema_version': 2, 'validated': True, 'symbols': array(['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'S', 'S', 'C',
       'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'N', 'N', 'N', 'N',
       'N', 'N', 'N', 'N', 'N', 'N', 'C', 'C', 'C', 'C', 'C', 'C', 'C',
       'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'H',
       'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H',
       'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H',
       'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H',
       'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H'], dtype='<U1'), 'geometry': array([[-3.81401710e+00,  4.52186150e+00, -8.71215711e+00],
       [ 7.04721078e+00,  6.06583650e+00, -7.69161444e+00],
       [ 5.00674514e+00, -9.58739849e+00,  9.73687713e+00],
       [-1.17332843e+01, -8.80603461e-01, -6.73137831e+00],
       [-5.08855645e+00, -6.46906296e-01, -5.05759736e+00],
       [ 6.45242066e+00, -5.57460393e+00,  3.03384361e+00],
       [-3.53073787e+00, -7.09759523e+00,  5.59306174e+00],
       [ 2.35267227e+00,  4.72746885e+00, -3.32531886e-01],
       [ 9.27238920e+00,  1.22310123e-01,  3.52753032e+00],
       [-7.44822757e+00, -7.75141307e-01,  1.96138265e+00],
       [ 9.48349600e-01, -2.50327718e+00,  3.08054417e+00],
       [-1.22971885e+00,  3.85946463e-01,  4.67143210e+00],
       [-5.52365198e+00,  3.21701062e+00, -9.60957029e+00],
       [ 4.90541071e+00,  6.60231010e+00, -6.95805669e+00],
       [ 3.37044220e+00, -8.01013092e+00,  9.18668615e+00],
       [-1.11319445e+01, -2.64910622e-03, -4.65737160e+00],
       [-5.73672449e+00,  1.41073905e+00, -4.20746705e+00],
       [ 7.44521225e+00, -5.85670023e+00,  5.11676902e+00],
       [-5.28466667e+00, -5.77330568e+00,  4.80161712e+00],
       [ 4.67503852e+00,  4.53824359e+00, -5.95477824e-01],
       [ 7.03480063e+00, -1.26831145e-01,  4.09066519e+00],
       [-5.49523382e+00, -1.48561054e+00,  8.29893944e-01],
       [ 1.13892612e+00, -8.66150511e+00,  8.11887045e+00],
       [-1.12125337e+01, -1.38697592e+00, -2.48373683e+00],
       [ 8.69909870e+00, -8.01912576e+00,  5.79528224e+00],
       [-6.53599366e+00, -4.07232171e+00,  6.25652493e+00],
       [-7.66170895e+00,  2.65771474e+00, -8.25203930e+00],
       [ 6.14069163e+00, -1.51894960e+00,  6.07669704e+00],
       [-4.86369099e+00, -3.93512037e+00,  5.79946814e-01],
       [ 4.35179483e+00,  6.94950411e+00, -4.44114009e+00],
       [ 6.06072956e+00,  2.85676980e+00,  7.47042595e-01],
       [-4.97671458e+00,  2.24813217e+00, -1.88820420e+00],
       [-5.40085965e+00,  2.13309667e+00, -1.22529921e+01],
       [ 2.71575097e+00,  7.02759709e+00, -8.76251564e+00],
       [ 5.44882690e+00,  1.01649521e+01,  2.53403305e-01],
       [ 8.97680891e+00,  9.98957580e+00, -2.97481694e+00],
       [ 3.72247808e+00, -5.21471679e+00,  9.81058205e+00],
       [-3.06443575e+00, -5.03427507e+00, -1.27055828e+00],
       [-2.82072628e+00, -7.78052615e+00, -4.15610359e-01],
       [-1.02714219e+01,  2.73829321e+00, -4.38302867e+00],
       [ 6.48539085e+00, -4.33287427e+00,  9.71359571e+00],
       [-5.41702428e+00, -8.38668326e+00,  7.35914348e-01],
       [ 7.37391098e+00,  8.47164268e+00, -1.09801883e+00],
       [ 3.29746975e+00, -8.22527958e-01,  1.17721286e+00],
       [-2.59444281e+00,  2.10382969e+00,  1.97899208e+00],
       [-7.63937320e+00,  3.14760427e+00, -5.57031731e+00],
       [ 7.52253808e+00, -3.67743867e+00,  7.07103830e+00],
       [-6.20588931e+00, -5.92068720e+00,  2.03189511e+00],
       [ 6.11245334e+00,  6.19909696e+00, -2.46847124e+00],
       [ 4.92429049e+00,  1.16362259e+00,  2.57665370e+00],
       [-3.67126969e+00,  5.22410899e-01, -1.93852878e-01],
       [-1.51885474e-01, -7.39079802e+00,  7.48667327e+00],
       [ 8.74834297e-01, -1.04971125e+01,  7.64026395e+00],
       [-1.02764581e+01, -8.48493292e-01, -8.92864240e-01],
       [-1.15604601e+01, -3.25797807e+00, -2.70270989e+00],
       [ 8.57260530e+00, -8.57808659e+00,  7.63260029e+00],
       [ 8.40937798e+00, -9.46152669e+00,  4.56135136e+00],
       [-5.81315441e+00, -3.68527578e+00,  7.98831429e+00],
       [-7.56687146e+00, -2.70935385e+00,  5.36913893e+00],
       [-8.81188813e+00,  1.23295403e+00, -8.83807111e+00],
       [ 4.24488746e+00, -1.82286889e+00,  5.98407615e+00],
       [ 2.51308284e+00,  6.99695324e+00, -3.90198881e+00],
       [ 7.96747054e+00,  2.72566735e+00,  5.50748473e-01],
       [-5.76493411e+00,  3.82721811e+00, -1.15170240e+00],
       [-4.58674687e+00,  3.53967157e+00, -1.35246559e+01],
       [-4.12394696e+00,  5.04550710e-01, -1.22108910e+01],
       [-7.23788131e+00,  1.51484562e+00, -1.29664367e+01],
       [ 3.04952916e+00,  5.92839312e+00, -1.04752020e+01],
       [ 2.65835927e+00,  9.03010493e+00, -9.28890947e+00],
       [ 8.77840501e-01,  6.50696951e+00, -7.97648878e+00],
       [ 6.42536354e+00,  1.16909874e+01,  1.25383428e+00],
       [ 4.31548410e+00,  9.11928599e+00,  1.63125118e+00],
       [ 4.15501070e+00,  1.10309500e+01, -1.11108499e+00],
       [ 7.78606562e+00,  1.08145790e+01, -4.45068743e+00],
       [ 9.95095234e+00,  1.15334236e+01, -1.99836287e+00],
       [ 1.03895428e+01,  8.80084184e+00, -3.90459990e+00],
       [ 2.46413554e+00, -4.03313170e+00,  8.67043740e+00],
       [ 3.05523295e+00, -5.01376466e+00,  1.17653383e+01],
       [-3.86454709e+00, -4.83642390e+00, -3.16839101e+00],
       [-1.25735359e+00, -4.05454348e+00, -1.22324250e+00],
       [-1.36417214e+00, -7.93065822e+00,  1.03910042e+00],
       [-2.33098845e+00, -9.02885169e+00, -1.98539676e+00],
       [-1.02299907e+01,  3.28812111e+00, -2.39210628e+00],
       [-1.16346139e+01,  3.94309860e+00, -5.36134766e+00],
       [ 6.67514200e+00, -2.61563352e+00,  1.08484072e+01],
       [ 7.66078197e+00, -5.77510517e+00,  1.06115900e+01],
       [-5.34446280e+00, -9.95004967e+00,  2.07730985e+00],
       [-6.78048110e+00, -8.85147801e+00, -7.49124418e-01],
       [ 8.64352290e+00,  7.65637724e+00,  3.28849938e-01],
       [ 2.31802253e+00,  1.15036562e-01, -3.72750008e-01],
       [ 4.48428834e+00, -2.33720155e+00,  4.21622376e-01],
       [-4.12560530e+00,  3.17708600e+00,  2.86828333e+00],
       [-1.16343006e+00,  3.39426255e+00,  1.24252314e+00],
       [-7.06611840e+00,  5.11363628e+00, -5.29245283e+00],
       [ 9.50408280e+00, -3.12506991e+00,  7.26868772e+00],
       [-8.24033641e+00, -5.57543196e+00,  1.94927554e+00],
       [ 7.60990420e+00,  5.10059075e+00, -3.37732271e+00],
       [ 3.71030901e+00,  2.25913305e+00,  3.84582191e+00],
       [-2.14850446e+00, -3.46989331e-01, -1.26413910e+00]]), 'name': 'C29H48N10O10S2', 'identifiers': {'molecule_hash': 'bd5c30549dcd8cae9552be466581a0bc2ae897a9', 'molecular_formula': 'C29H48N10O10S2'}, 'molecular_charge': 0.0, 'molecular_multiplicity': 1, 'connectivity': [[0, 12, 2.0], [1, 13, 2.0], [2, 14, 2.0], [3, 15, 2.0], [4, 16, 2.0], [5, 17, 2.0], [6, 18, 2.0], [7, 19, 2.0], [8, 20, 2.0], [9, 21, 2.0], [10, 11, 1.0], [10, 43, 1.0], [11, 44, 1.0], [12, 26, 1.0], [12, 32, 1.0], [13, 29, 1.0], [13, 33, 1.0], [14, 22, 1.0], [14, 36, 1.0], [15, 23, 1.0], [15, 39, 1.0], [16, 31, 1.0], [16, 45, 1.0], [17, 24, 1.0], [17, 46, 1.0], [18, 25, 1.0], [18, 47, 1.0], [19, 30, 1.0], [19, 48, 1.0], [20, 27, 1.0], [20, 49, 1.0], [21, 28, 1.0], [21, 50, 1.0], [22, 51, 1.0], [22, 52, 1.0], [23, 53, 1.0], [23, 54, 1.0], [24, 55, 1.0], [24, 56, 1.0], [25, 57, 1.0], [25, 58, 1.0], [26, 45, 1.0], [26, 59, 1.0], [27, 46, 1.0], [27, 60, 1.0], [28, 37, 1.0], [28, 47, 1.0], [29, 48, 1.0], [29, 61, 1.0], [30, 49, 1.0], [30, 62, 1.0], [31, 50, 1.0], [31, 63, 1.0], [32, 64, 1.0], [32, 65, 1.0], [32, 66, 1.0], [33, 67, 1.0], [33, 68, 1.0], [33, 69, 1.0], [34, 42, 1.0], [34, 70, 1.0], [34, 71, 1.0], [34, 72, 1.0], [35, 42, 1.0], [35, 73, 1.0], [35, 74, 1.0], [35, 75, 1.0], [36, 40, 1.0], [36, 76, 1.0], [36, 77, 1.0], [37, 38, 1.0], [37, 78, 1.0], [37, 79, 1.0], [38, 41, 1.0], [38, 80, 1.0], [38, 81, 1.0], [39, 45, 1.0], [39, 82, 1.0], [39, 83, 1.0], [40, 46, 1.0], [40, 84, 1.0], [40, 85, 1.0], [41, 47, 1.0], [41, 86, 1.0], [41, 87, 1.0], [42, 48, 1.0], [42, 88, 1.0], [43, 49, 1.0], [43, 89, 1.0], [43, 90, 1.0], [44, 50, 1.0], [44, 91, 1.0], [44, 92, 1.0], [45, 93, 1.0], [46, 94, 1.0], [47, 95, 1.0], [48, 96, 1.0], [49, 97, 1.0], [50, 98, 1.0]], 'fix_com': True, 'fix_orientation': True, 'fix_symmetry': 'c1', 'provenance': {'creator': 'QCElemental', 'version': 'v0.17.0', 'routine': 'qcelemental.molparse.from_schema'}, 'id': '24775476', 'extras': {'canonical_isomeric_explicit_hydrogen_mapped_smiles': '[O:1]=[C:13]([N:27]([C@:46]([C:17](=[O:5])[N:32]([C@:51]([C:22](=[O:10])[N:29]1[C:38]([H:79])([H:80])[C:39]([H:81])([H:82])[C:42]([H:87])([H:88])[C@:48]1([C:19](=[O:7])[N:26]([H:58])[H:59])[H:96])([C:45]([S:12][S:11][C:44]([C@@:50]([C:21](=[O:9])[N:28]([C@:47]([C:18](=[O:6])[N:25]([H:56])[H:57])([C:41]([C:37]([C:15](=[O:3])[N:23]([H:52])[H:53])([H:77])[H:78])([H:85])[H:86])[H:95])[H:61])([N:31]([C:20](=[O:8])[C@@:49]([N:30]([C:14](=[O:2])[C:34]([H:68])([H:69])[H:70])[H:62])([C:43]([C:35]([H:71])([H:72])[H:73])([C:36]([H:74])([H:75])[H:76])[H:89])[H:97])[H:63])[H:98])([H:90])[H:91])([H:92])[H:93])[H:99])[H:64])([C:40]([C:16](=[O:4])[N:24]([H:54])[H:55])([H:83])[H:84])[H:94])[H:60])[C:33]([H:65])([H:66])[H:67]'}}, 'driver': 'gradient', 'model': {'method': 'b3lyp-d3bj', 'basis': 'dzvp'}, 'keywords': {'maxiter': 300, 'scf_properties': ['dipole', 'quadrupole', 'wiberg_lowdin_indices', 'mayer_indices'], 'scf_mem_safety_factor': 0.5}, 'protocols': {}, 'extras': {}, 'provenance': {'creator': 'QCElemental', 'version': 'v0.17.0', 'routine': 'qcelemental.models.results'}}
Computation done.
FailedOperation(error=ComputeError(error_type='unknown', error_message='geomeTRIC run_json error:\nTraceback (most recent call last):\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/run_json.py", line 214, in geometric_run_json\n    geometric.optimize.Optimize(coords, M, IC, engine, None, params)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1331, in Optimize\n    return optimizer.optimizeGeometry()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1293, in optimizeGeometry\n    self.calcEnergyForce()\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/optimize.py", line 1002, in calcEnergyForce\n    spcalc = self.engine.calc(self.X, self.dirname)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 874, in calc\n    return self.calc_new(coords, dirname)\n  File "/home/tgokey/.local/miniconda3/envs/psi4-test/lib/python3.7/site-packages/geometric/engine.py", line 866, in calc_new\n    raise QCEngineAPIEngineError("QCEngineAPI computation did not execute correctly. Message: " + ret["error"]["error_message"])\ngeometric.errors.QCEngineAPIEngineError: QCEngineAPI computation did not execute correctly. Message: QCEngine Unknown Error: Unknown error, error message is not found\n'))

The segfault is from above is

Code Block

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
libgdma.so         00007F60FDA7D0DC  for__signal_handl     Unknown  Unknown
libpthread-2.32.s  00007F61046D7830  Unknown               Unknown  Unknown
libstdc++.so.6.0.  00007F609C4B68C9  Unknown               Unknown  Unknown
libstdc++.so.6.0.  00007F609C4B6AF8  _ZSt29_Rb_tree_in     Unknown  Unknown
core.cpython-37m-  00007F61007D16AE  Unknown               Unknown  Unknown
core.cpython-37m-  00007F61007E2542  Unknown               Unknown  Unknown
libomp.so          00007F609C60E153  __kmp_invoke_micr     Unknown  Unknown
libomp.so          00007F609C5B0CED  Unknown               Unknown  Unknown
libomp.so          00007F609C5B4CF6  __kmp_fork_call       Unknown  Unknown
libomp.so          00007F609C5A39E2  __kmpc_fork_call      Unknown  Unknown
core.cpython-37m-  00007F61007E10F5  Unknown               Unknown  Unknown
core.cpython-37m-  00007F61007F1BA0  Unknown               Unknown  Unknown
core.cpython-37m-  00007F61007F0587  Unknown               Unknown  Unknown
core.cpython-37m-  00007F61009C9969  _ZN3psi5VBase10in     Unknown  Unknown
core.cpython-37m-  00007F61009C86A9  Unknown               Unknown  Unknown
core.cpython-37m-  00007F6102540A4E  Unknown               Unknown  Unknown
core.cpython-37m-  00007F610252878D  Unknown               Unknown  Unknown
core.cpython-37m-  00007F610254DC43  Unknown               Unknown  Unknown
core.cpython-37m-  00007F60FEA88E8A  Unknown               Unknown  Unknown
core.cpython-37m-  00007F60FE6CF66E  Unknown               Unknown  Unknown
python3.7          0000559523213504  _PyMethodDef_RawF     Unknown  Unknown
python3.7          0000559523215B9A  Unknown               Unknown  Unknown
python3.7          00005595231F3191  PyObject_Call         Unknown  Unknown
python3.7          00005595232525A4  Unknown               Unknown  Unknown
python3.7          00005595231F2A7A  Unknown               Unknown  Unknown
core.cpython-37m-  00007F60FE6C95BB  Unknown               Unknown  Unknown
python3.7          000055952322DF3B  _PyObject_FastCal     Unknown  Unknown
python3.7          000055952322EA79  Unknown               Unknown  Unknown
python3.7          00005595232A174A  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown
python3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown
python3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown
python3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown
python3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          00005595231E21F0  _PyFunction_FastC     Unknown  Unknown
python3.7          000055952329E869  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          000055952322C700  _PyFunction_FastC     Unknown  Unknown
python3.7          000055952322E8E8  Unknown               Unknown  Unknown
python3.7          000055952329DF95  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          000055952322C700  _PyFunction_FastC     Unknown  Unknown
python3.7          000055952322E8E8  Unknown               Unknown  Unknown
python3.7          00005595232A174A  _PyEval_EvalFrame     Unknown  Unknown
python3.7          00005595231E0932  _PyEval_EvalCodeW     Unknown  Unknown
python3.7          00005595231E1B49  PyEval_EvalCodeEx     Unknown  Unknown
python3.7          00005595232C1C2B  PyEval_EvalCode       Unknown  Unknown
python3.7          000055952332E3FF  Unknown               Unknown  Unknown
python3.7          00005595233389C7  PyRun_FileExFlags     Unknown  Unknown
python3.7          0000559523338B9C  PyRun_SimpleFileE     Unknown  Unknown
python3.7          0000559523339129  Unknown               Unknown  Unknown
python3.7          000055952333927C  _Py_UnixMain          Unknown  Unknown
libc-2.32.so       00007F61043E2EBB  __libc_start_main     Unknown  Unknown
python3.7          00005595232ADC99  Unknown               Unknown  Unknown
double free or corruption (fasttop)

Notice how this error is nowhere in the parsed data (success=True oddly enough, so deserialize seems to be the culprit for swallowing this segfault). Looking 1 nm closer, it looks like the part of the output that has this segfault is completely dropped in all cases, since deserialize is only taking the output sections. Likely this should be changed since this error will never, ever a see the light of a computer screen.

In any case, more tests are needed to see how prevalent this issue is. For now, a %1 failure rate would tell me that I have a good chance in not finishing an optimization if psi4 is called 100 times (i.e. 100 minimization steps in geomeTRIC). Since the error rate is about %1, I run 1000 iterations using the original input, which should be no issue. I wasn’t getting many issues when each process had about 1 CPU, when I reduce the number to 4 concurrent tests, I see segfaults again, at this point 2/64.

Testing with 1 core jobs, appears to be smooth sailing… so if the last paragraph read like spaghetti, I can summarize by saying using multiple threads in psi4 causes a random, instant ( < 5 seconds) crash. At this point I’d like to compile my own versions of psi4 and test, but this is getting into areas more for my bemusement than solving QCA issues, which will always use precooked binaries. It does look like an OpenMP parallel section though based on the stack trace. I wonder how kosher the conda supplied libraries are?

I have finally worked out a similar test on the UCI cluster, where the original errors manifested. I found that running the same test produced 9/10000 crashes (0.09%). Of course, I think this number depends on a lot of things, since it seems to be thread related, and therefore at the hands of the OS scheduler and other related things. Regardless, the segfault has been reproduced on the cluster.

Next is to test the memory across the UCI cluster. These jobs will run for 8+ hours, and are therefore much more expensive. In the 10k segfault test case, I kill the jobs after 60s, so throughput was high. I will run a few (100), and see what the memory profiling comes out to. Note that, because I have shown that the default settings will regularly go above the memory limit, I am going to set the safety factor to 50%. In this hypothesis, I expect all to succeed sans the random crashes we get at the start. There will also be some failures since I am running on a pre-emptable queue, and will take this into account.

All but one job was pre-empted, and finished after 152 iterations:

So as far as I am concerned, we have this fixed if we reduce the safety factor. Just to confirm, I will submit two jobs with the default safety factor using 16GB and 32GB workers, since these would be the settings used when the crashes occurred.

Running with 16GB, it dies shortly after starting. A nice thing I found, since I am running these interactively, I get this after ending the session:

Code Block
srun: error: hpc3-15-26: task 0: Out Of Memory

which is definitely telling. Here is the memory profile:

(noting here that I think the PSI memory cap is in units of GB, so I need to scale down by (1000/1024)^3 to make better sense of it)

and the memory profile when I set the Psi4 memory to 32GB:

Code Block
srun: error: hpc3-15-09: task 0: Out Of Memory

Versions Compared

Old Version 9

New Version 10

Key

Intro

Procedures/Scripts

Running

Plotting

Segfault hunting

Experiments

SCF_MEM_SAFETY_FACTOR

DIIS

DFT vs HF

DF_INTS_NUM_THREADS

Direct vs DF

Full Runs

CORE vs DISK algorithm

Segfaults