A small molecule with a modest basis set finishes in minutes on one core. The cost climbs
steeply from there: doubling the number of atoms can make a DFT job take around ten times
as long, and an IR or thermodynamics add-on can take several times longer than the
optimization before it. This page covers how to plan a job like that, and how to run it
with PySCF on a GPU, which is usually the fastest way to get it done.
Expect a long run if any of these apply:
+,aug-)A big calculation goes better as a few smaller jobs, each starting from the last one.
The cost is cores × hours the job actually ran (see
What a job costs).
A GPU has thousands of small processors working at once. That suits the integrals and
matrix algebra at the heart of Hartree–Fock and DFT very well. ChemCompute runs PySCF's
GPU version (GPU4PySCF), and for a molecule of a few dozen atoms or more a single GPU
usually finishes a DFT optimization or frequency job several times faster than 32 CPU
cores. The bigger the molecule and basis set, the bigger the gain, and on the largest
jobs it can be ten times or more. For a small molecule the GPU gains little, since start-up
and the queue take most of the time, so use CPU cores for those.
Only PySCF and NAMD use GPUs on ChemCompute. GAMESS and Psi4 run on CPU cores only.
The GPUs choice appears on the form only when your account has GPU access. That needs
a verified university email address (such as .edu) and use of ChemCompute from the U.S.
Student and researcher accounts can each use 1 GPU per job. If you think you should have
access and don't see the choice, open a support ticket.
From about the size of a cyanine dye (25 or more non-hydrogen atoms), where a GPU
clearly pays off, the guided form suggests one under the GPUs choice. When
ChemCompute has enough earlier GPU runs like yours to go on, the run-time estimate under
the time limit also says about how long the job should take on a GPU, and how many times
faster that is than the cores you chose:
Estimated time on the cluster: about 2 hours ± 30%. On a GPU with PySCF: about 12
minutes, roughly 10 times faster.
Like every run-time estimate, it is a rough guide.
| GPU Type | GPU | GPU memory | Host memory | Good for |
|---|---|---|---|---|
| a100-40 | NVIDIA A100 | 40 GB | 57.6 GB | The default and the best first choice. Delta has the most of these |
| a40-48 | NVIDIA A40 | 48 GB | 57.6 GB | NAMD. It is slow at the double-precision arithmetic quantum chemistry needs, so avoid it for PySCF |
| a100-40-bigmem | NVIDIA A100 | 40 GB | 230 GB | Jobs that need a lot of ordinary memory alongside the GPU. Few nodes, so longer waits |
| h200-141 | NVIDIA H200 | 141 GB | 230 GB | The largest molecules and basis sets. The fastest card, but few nodes, so longer waits |
Any uses the cluster's default type, which on Delta is a100-40.
Expanse and Bridges-2 have GPUs too: V100s on Expanse, and H100, L40S and V100 cards on
Bridges-2.
GPU jobs are charged 60 service units per GPU-hour, whichever GPU type you choose. A
32-core CPU job costs 32 per hour. So a GPU job costs less overall whenever it finishes in
under about half the time the 32-core job would take, which is usual for large molecules.
For small molecules CPU cores are cheaper.
While a job waits or runs, half of what it could cost at its full time limit is set
aside as Locked on the Dashboard. A GPU job with a 48-hour limit locks 1,440 units,
which can get it refused as Over allocation. Choose a shorter limit, or ask for more with
Request More CPU-Hours / Resources on the Dashboard.
Researcher accounts can upload their own PySCF script (see
Using your own input file) and set GPUs to 1 on that page too. The
script has to use the GPU code itself. Ordinary PySCF runs on the CPU cores, and the GPU
sits idle while you are charged for it. The upload is a single file, so put the
coordinates in the script. Either import the GPU classes:
from pyscf import gto
from gpu4pyscf.dft import rks
mol = gto.M(atom='''
O 0.0000 0.0000 0.1173
H 0.0000 0.7572 -0.4692
H 0.0000 -0.7572 -0.4692
''', basis='def2-tzvp')
mf = rks.RKS(mol, xc='b3lyp').density_fit()
mf.kernel()
or build the object as usual and move it to the GPU with .to_gpu():
from pyscf import gto, dft
mol = gto.M(atom='''
O 0.0000 0.0000 0.1173
H 0.0000 0.7572 -0.4692
H 0.0000 -0.7572 -0.4692
''', basis='def2-tzvp')
mf = dft.RKS(mol, xc='b3lyp').density_fit().to_gpu()
mf.kernel()
The guided form's Preview Input file shows the script ChemCompute writes for a GPU
job, which is a good starting point.