From 7007f98e68e787d77768c3a4fff4d6675fd33271 Mon Sep 17 00:00:00 2001 From: Jan Siwiec <jan.siwiec@vsb.cz> Date: Mon, 7 Oct 2024 08:54:00 +0200 Subject: [PATCH] Update dask.md --- docs.it4i/software/data-science/dask.md | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/docs.it4i/software/data-science/dask.md b/docs.it4i/software/data-science/dask.md index 592701539..eb45caadc 100644 --- a/docs.it4i/software/data-science/dask.md +++ b/docs.it4i/software/data-science/dask.md @@ -1,6 +1,3 @@ -!!!warning - This page has not been updated yet. The page does not reflect the transition from PBS to Slurm. - # Dask [Dask](https://docs.dask.org/en/latest/) is a popular open-source library that allows you to @@ -41,7 +38,7 @@ execute your tasks.  -After you start a PBS job, you should therefore first start the server and the workers on the +After you start a job, you should therefore first start the server and the workers on the available computing nodes and then run your Python program that uses Dask. There are multiple ways of deploying the cluster. A common scenario is to run a Dask server on a single computing node, run a single worker per node on all remaining computing nodes and then run your program on the node @@ -53,14 +50,14 @@ with the server. !!! note All the following deployment methods assume that you are inside a Python environment that has Dask installed. Do not forget to load Python and activate the correct virtual environment at - the beginning of your PBS job! And also do the same after connecting to any worker nodes + the beginning of your job! And also do the same after connecting to any worker nodes manually using SSH. ### Manual Deployment Both the server and the worker nodes can be started using a CLI command. If you prefer manual deployment, you can manually start the server on a selected node and then start the workers on -other nodes available inside your PBS job. +other nodes available inside your job. ```bash # Start the server on some node N @@ -73,15 +70,15 @@ $ dask-worker tcp://<hostname-of-N>:8786 ### Dask-ssh Deployment Dask actually contains [built-in support](https://docs.dask.org/en/latest/setup/ssh.html) for -automating Dask deployment using SSH. It also supports nodefiles provided by PBS, so inside of your -PBS job, you can simply run +automating Dask deployment using SSH. It also supports nodefiles provided by Slurm, +so inside of your job, you can simply run ```bash -$ dask-ssh --hostfile $PBS_NODEFILE +$ dask-ssh --hostfile $SLURM_NODELIST ``` to start the Dask cluster on all available computing nodes. This will start the server on the first -node of your PBS job and then a single worker on each node. The first node will therefore be shared +node of your job and then a single worker on each node. The first node will therefore be shared by a server and a worker, which might not be ideal from a performance point of view. > Note that for this to work, the `paramiko` Python library has to be installed inside your Python @@ -94,8 +91,8 @@ can start the scheduler and the workers on separate nodes to avoid overcrowding ### Other Deployment Options Dask has a lot of other ways of being deployed, e.g. using MPI, or using a shared file on the -network file system. It also allows you to create a PBS job directly, wait for it to be started and -then it starts the whole cluster inside the PBS job. You can find more information about Dask HPC +network file system. It also allows you to create a job directly, wait for it to be started and +then it starts the whole cluster inside the job. You can find more information about Dask HPC deployment [here](https://docs.dask.org/en/latest/setup/hpc.html). ## Connecting to the Cluster -- GitLab