HEAppE Middleware

High-End Application Execution Middleware former HPC as a Service Middleware

HPC-as-a-Service is a well known term in the area of high performance computing. It enables users to access an HPC infrastructure without a need to buy and manage their own physical servers or data center infrastructure. Through this service small and medium enterprises (SMEs) can take advantage of the technology without an upfront investment in the hardware. This approach further lowers the entry barrier for users and SMEs who are interested in utilizing massive parallel computers but often do not have the necessary level of expertise in this area.

To provide this simple and intuitive access to the supercomputing infrastructure an in-house application framework called HEAppE has been developed. This framework is utilizing a mid-layer principle, in software terminology also known as middleware. Middleware manages and provides information about submitted and running jobs and their data between the client application and the HPC infrastructure. HEAppE is able to submit required computation or simulation on HPC infrastructure, monitor the progress and notify the user should the need arise. It provides necessary functions for job management, monitoring and reporting, user authentication and authorization, file transfer, encryption, and various notification mechanisms.

References

HEAppE Middleware has already been successfully used in several public or commercial projects:

• in H2020 project LEXIS as a part of LEXIS Platform to provide the platform's job orchestrator access to a number of HPC systems in several HPC centers; https://lexis-project.eu
• in crisis decision support system Floreon+ for What-If analysis workflow utilizing HPC clusters; https://floreon.eu
• in Urban Thematic Exploitation Platform (Urban-TEP) financed by ESA as a middleware enabling sandbox execution of user-defined docker images on the cluster; https://urban-tep.eo.esa.int
• in H2020 project ExCaPE as a part of Drug Discovery Platform enabling execution of drug discovery scientific pipelines on a supercomputer; http://excape-h2020.eu
• in the area of molecular diagnostics and personalized medicine in the scope of the Moldimed project as a part of the Massive Parallel Sequencing Platform for analysis of NGS data; https://www.imtm.cz/moldimed
• in the area of bioimageinformatics as a integral part of FIJI plugin providing unified access to HPC clusters for image data processing; http://fiji.sc

Licence and Contact Information

HEAppE Middleware is licensed under the GNU General Public License v3.0. For commercial use, contact us via support.heappe@it4i.cz regarding the proprietary license information.

Next Release Information - HEAppE Middleware v2

New version of the middleware is in the pre-production state. We are fixing the last bugs, testing the new version and preparing it for the next open-source release in early 2021.

Major changes includes

• multi-platform .NET Core version
• OpenAPI REST API
• dockerized deployment and management
• updated PBS and new Slurm workload manager adapter
• SSH Agent support
• various functional and security updates

IT4Innovations national supercomputing center

The IT4Innovations national supercomputing center operates four supercomputers: Anselm (94 TFlop/s, installed in 2013), Salomon (2 PFlop/s, installed 2015), Barbora (826 TFlop/s, installed 2019) and a special system for AI computation, DGX-2 (2 PFlop/s in AI, installed in 2019). A petascale EURO IT4I system will be installed at the centre in 2020 as part of the EuroHPC project. The supercomputers are available to academic community within the Czech Republic and Europe and industrial community worldwide via HEAppE Middleware.

Salomon

The Salomon cluster consists of 1008 compute nodes, totaling 24192 compute cores with 129 TB RAM and giving over 2 Pflop/s theoretical peak performance. Each node is a powerful x86-64 computer, equipped with 24 cores, at least 128 GB RAM. Nodes are interconnected by 7D Enhanced hypercube InfiniBand network and equipped with Intel Xeon E5-2680v3 processors. The Salomon cluster consists of 576 nodes without accelerators and 432 nodes equipped with Intel Xeon Phi MIC accelerators.

https://docs.it4i.cz/salomon/hardware-overview/

Barbora

The Barbora cluster consists of 201 compute nodes, totaling 7232 compute cores with 44544 GB RAM, giving over 848 TFLOP/s theoretical peak performance. Nodes are interconnected through a fully non-blocking fat-tree InfiniBand network, and are equipped with Intel Cascade Lake processors. A few nodes are also equipped with NVIDIA Tesla V100-SXM2.

https://docs.it4i.cz/barbora/hardware-overview/

NVIDIA DGX-2

The DGX-2 is a very powerful computational node, featuring high end x86_64 processors and 16 NVIDIA V100-SXM3 GPUs. The DGX-2 introduces NVIDIA’s new NVSwitch, enabling 300 GB/s chip-to-chip communication at 12 times the speed of PCIe. With NVLink2, it enables 16x NVIDIA V100-SXM3 GPUs in a single system, for a total bandwidth going beyond 14 TB/s. Featuring pair of Xeon 8168 CPUs, 1.5 TB of memory, and 30 TB of NVMe storage, we get a system that consumes 10 kW, weighs 163.29 kg, but offers double precision performance in excess of 130TF.

https://docs.it4i.cz/dgx2/introduction/

Anselm

The Anselm cluster consists of 209 compute nodes, totaling 3344 compute cores with 15 TB RAM and giving over 94 TFLOP/s theoretical peak performance. Each node is a powerful x86-64 computer, equipped with 16 cores, at least 64 GB RAM, and 500 GB hard disk drive. Nodes are interconnected by fully non-blocking fat-tree InfiniBand network and equipped with Intel Sandy Bridge processors. A few nodes are also equipped with NVIDIA Kepler GPU or Intel Xeon Phi MIC accelerators.

https://docs.it4i.cz/anselm/hardware-overview/

Acknowledgement

This work was supported by The Ministry of Education, Youth and Sports from the National Programme of Sustainability (NPS II) project ”IT4Innovations excellence in science - LQ1602” and by the IT4Innovations infrastructure which is supported from the Large Infrastructures for Research, Experimental Development and Innovations project ”IT4Innovations National Supercomputing Center – LM2015070”.

Middleware Architecture

HEAppE's universally designed software architecture enables unified access to~different HPC systems through a simple object-oriented client-server interface using standard web services. Thus providing HPC capabilities to the users but without the necessity to manage the running jobs form the command-line interface of the HPC scheduler directly on the cluster.

Web Services

UserAndLimitationManagementWs

• AuthenticateUserDigitalSignature – the user authentication using public/private key pair
• AuthenticateUserPassword – the user authentication via username and password
• GetCurrentUsageAndLimitationsForCurrentUser – the current usage and constraints for the user

ClusterInformationWs

• GetCurrentClusterNodeUsage – information about the current node usage
• ListAvailableClusters – the list of available clusters

JobManagementWs

• GetCurrentInfoForJob – returns base information describing the current task
• ListJobsForCurrentUser – returns a list of basic information describing all user jobs
• CreateJob – creates a new job
• SubmitJob – submits a job to a HPC scheduler queue
• CancelJob – cancels a running job
• DeleteJob - deletes job files

FileTransferWs

• DownloadFileFromCluster – downloads entire files from the cluster storage
• DownloadPartsOfJobFilesFromCluster – downloads changes in the file based on the offset
• EndFileTransfer – ends the file transfer
• GetFileTransferMethod – acquires object used for data transfer
• ListChangedFilesForJob – lists newly created or edited files and folders in the base directory

JobReportingWs

• GetResourceUsageReportForJob – the used core-hours for the job
• GetUserGroupResourceUsageReport – the used core-hours for a user group
• GetUserResourceUsageReport – the used core-hours for the user
• ListAdaptorUserGroups – a list of all users in the specified group

Command Template Preparation

For security purposes HEAppE enables the users to run only pre-prepared set of so-called Command Templates. Each template defines arbitrary script or executable file that will be executed on the cluster, any dependencies or third-party software it might require and the type queue that should be used for the processing (type of computing nodes to be used on the cluster). The template also contains the set of input parameters that will be passed to the executable script during run-time. Thus, the users are only able to execute pre-prepared command templates with the pre-defined set of input parameters. The actual value of each parameter (input from the user) can be changed by the user for each job submission.

Id	Name	Description	Code	Executable File	Command Parameters	Preparation Script	Cluster Node Type
1	TestTemplate	Desc	Code	/scratch/temp/ HaasTestScript/test.sh	"%%{inputParam}"	module load Python/2.7.9-intel-2015b;	7

{: .custom-class #custom-id}

Workflow

HEAppE Integration Example (C#)

//web references for HaaS services
using WsClient.test_ClusterInformationWs;
using WsClient.test_FileTransferWs;
using WsClient.test_JobManagementWs;
using WsClient.test_JobReportingWs;
using WsClient.test_UserAndLimitationManagementWs;

namespace WsClient
{
    class Program
    {
        static ClusterInformationWs wsClusterInformation = new ClusterInformationWs();
        static FileTransferWs wsFileTransfer = new FileTransferWs();
        static JobManagementWs wsJobManagement = new JobManagementWs();
        static JobReportingWs wsJobReporting = new JobReportingWs();
        static UserAndLimitationManagementWs wsUserAndLimitationManagement = new
        UserAndLimitationManagementWs();

        private static string sessionCode;	//code acquired via authentication
        
        static void Main(string[] args)
        {
            AuthenticateUserPassword();
            CreateAndSubmitTestJob();
        }

        private static void CreateAndSubmitTestJob()
        {
            //each submitted job must contain at least one task
            TaskSpecificationExt testTask = new TaskSpecificationExt();
            testTask.name = "TestJob";
            testTask.minCores = 1;		//minimum number of cores required
            testTask.maxCores = 1;		//maximum number of cores required
            testTask.walltimeLimit = 600;	//maximum time for task to run (seconds)
            testTask.standardOutputFile = "console_Stdout";
            testTask.standardErrorFile = "console_Stderr";
            testTask.progressFile = "console_Stdprog";
            testTask.logFile = "console_Stdlog";
            testTask.commandTemplateId = 1;	//commandTemplateID
            //custom environment variables for the task
            testTask.environmentVariables = new EnvironmentVariableExt[0];
            //fill the command template parameters (see Table1 for “inputParam”)
            testTask.templateParameterValues = new CommandTemplateParameterValueExt[] { 
                new CommandTemplateParameterValueExt() { commandParameterIdentifier =
                    "inputParam", parameterValue = "someStringParam" } 
            };

            //create job specification with the task above
            JobSpecificationExt testJob = new JobSpecificationExt();
            testJob.name = "TestJob";	//job name
            testJob.minCores = 1;		//minimum number of cores required
            testJob.maxCores = 1;		//maximum number of cores required
            testJob.priority = WsClient.test_JobManagementWs.JobPriorityExt.Average;
            testJob.project = "ExpTests";	//accounting project
            testJob.waitingLimit = null;	//limit for the waiting time in cluster queue 
            testJob.walltimeLimit = 600;	//maximum time for job to run (seconds)
            testJob.clusterNodeTypeId = 7;	//Salomon express queue (1h limit)
            //custom environment variables for the job
            testJob.environmentVariables = new EnvironmentVariableExt[0];
            //assign created task to job specification
            testJob.tasks = new TaskSpecificationExt[] { testTask };

            //create job
            SubmittedJobInfoExt submittedTestJob = wsJobManagement.CreateJob(testJob, sessionCode);
            Console.WriteLine("Created job ID {0}.", submittedTestJob.id);

            //upload input files
            FileTransferMethodExt ft = wsFileTransfer.GetFileTransferMethod(
                (long)submittedTestJob.id, sessionCode);
            using (MemoryStream pKeyStream = new MemoryStream(
                Encoding.UTF8.GetBytes(ft.credentials.privateKey)))
            {
                using (ScpClient scpClient = new ScpClient(ft.serverHostname,
                    ft.credentials.username, new PrivateKeyFile(pKeyStream)))
                {
                    scpClient.Connect();
                    DirectoryInfo di = new DirectoryInfo(@"C:\InputFiles\");
                    foreach (FileInfo fi in di.GetFiles())
                    {
                        Console.WriteLine("Uploading file: " + fi.Name);
                        scpClient.Upload(fi, ft.sharedBasepath + "//" + fi.Name);
                        Console.WriteLine("File uploaded.");
                    }
                }
            }
            wsFileTransfer.EndFileTransfer((long)submittedTestJob.id, ft, sessionCode);
            
            //submit the job to a cluster queue for processing
            submittedTestJob = wsJobManagement.SubmitJob((long)submittedTestJob.id, sessionCode);
            Console.WriteLine("Submitted job ID: {0}", submittedTestJob.id);

            //check status of running job (submitted/configuring/queued/running)
            SubmittedJobInfoExt submittedJob;
            long jobId = (long)submittedTestJob.id;
            do
            {
                //wait 30s before the next status check
                System.Threading.Thread.Sleep(30000);
                //get info for the job
                submittedJob = wsJobManagement.GetCurrentInfoForJob(jobId, sessionCode);
                Console.WriteLine(submittedJob.state);
                //set offsets for the stdout, stderr, stdprog, stdlog files
                //offsets can be used for the partial download of files
                List<TaskFileOffsetExt> offsets = new List<TaskFileOffsetExt>();
                foreach (SubmittedTaskInfoExt taskInfo in submittedJob.tasks)
                {
                    TaskFileOffsetExt off = new TaskFileOffsetExt();
                    off.fileType = SynchronizableFilesExt.LogFile;
                    off.submittedTaskInfoId = taskInfo.id;
                    off.offset = 0;
                    offsets.Add(off);

                    off = new TaskFileOffsetExt();
                    off.fileType = SynchronizableFilesExt.ProgressFile;
                    off.submittedTaskInfoId = taskInfo.id;
                    off.offset = 0;
                    offsets.Add(off);

                    off = new TaskFileOffsetExt();
                    off.fileType = SynchronizableFilesExt.StandardErrorFile;
                    off.submittedTaskInfoId = taskInfo.id;
                    off.offset = 0;
                    offsets.Add(off);

                    off = new TaskFileOffsetExt();
                    off.fileType = SynchronizableFilesExt.StandardOutputFile;
                    off.submittedTaskInfoId = taskInfo.id;
                    off.offset = 0;
                    offsets.Add(off);
                }

                //donwload stdouts based on the offsets
                JobFileContentExt[] result = wsFileTransfer.DownloadPartsOfJobFilesFromCluster(
                    jobId, offsets.ToArray(), sessionCode);
                //print each file
                foreach (JobFileContentExt file in result)
                {
                    Console.WriteLine("File: " + file.fileType + ", " + file.relativePath);
                    Console.WriteLine("TaskInfoId: " + file.submittedTaskInfoId);
                    Console.WriteLine("Offset: " + file.offset);
                    Console.WriteLine("Content: " + file.content);
                }
            }
            While (submittedJob.state == WsClient.test_JobManagementWs.JobStateExt.Submitted
                || submittedJob.state == WsClient.test_JobManagementWs.JobStateExt.Configuring 
                || submittedJob.state == WsClient.test_JobManagementWs.JobStateExt.Queued 
                || submittedJob.state == WsClient.test_JobManagementWs.JobStateExt.Running);   

            //job computation is done (finished/failed/canceled)
            submittedJob = wsJobManagement.GetCurrentInfoForJob(jobId, sessionCode);
            // job finished successfully, download result files from the cluster
            if (submittedJob.state == WsClient.test_JobManagementWs.JobStateExt.Finished) 
            {
                ft = wsFileTransfer.GetFileTransferMethod(jobId, sessionCode);
                using (MemoryStream pKeyStream = new MemoryStream(Encoding.UTF8.GetBytes(
                    ft.credentials.privateKey)))
                {
                    using (ScpClient scpClient = new ScpClient(ft.serverHostname,
                        ft.credentials.username, new PrivateKeyFile(pKeyStream)))
                    {
                        scpClient.Connect();
                        // changed result files
                        string[] changedFiles = wsFileTransfer.ListChangedFilesForJob(
                            jobId, sessionCode);
                        foreach (string file in changedFiles)
                        {
                            Console.WriteLine("Downloading file: " + file);
                            FileInfo fi = new FileInfo(@"C:\OutputFiles\" + file);
                            scpClient.Download(ft.sharedBasepath + "//" + file, fi);
                        }
                    }
                }
                wsFileTransfer.EndFileTransfer(jobId, ft, sessionCode);
            }
            // job failed or was canceled
            else if (submittedJob.state == WsClient.test_JobManagementWs.JobStateExt.Failed 
                || submittedJob.state == WsClient.test_JobManagementWs.JobStateExt.Canceled)
            {
                // do nothing
            }
        }
        
        private static void AuthenticateUserPassword()
        {
            Console.WriteLine("Authenticating user [{0}]...", "[testuser]");
            PasswordCredentialsExt credentials = new PasswordCredentialsExt();
            credentials.username = "[testuser]";
            credentials.password = "[testpass]";
            sessionCode = wsUserAndLimitationManagement.AuthenticateUserPassword(credentials);
            Console.WriteLine("\tAuth OK (Session GUID: {0})", sessionCode);
        }

    }
}

Sample Template Output

Authenticating user [testuser]...
        Auth OK (Session GUID: 4a5d7017-1992-45b1-8b07-43cf5d421f50)
Created job ID 71.
Uploading file: someInputFile1.txt
File uploaded.
Uploading file: someInputFile2.txt
File uploaded.
Submitted job ID: 71
Queued
File: StandardErrorFile, console_Stderr
TaskInfoId: 71
Offset: 0
Content:
File: StandardOutputFile, console_Stdout
TaskInfoId: 71
Offset: 0
Content: Input param: someStringParam
Iteration: 01

Running
File: StandardErrorFile, console_Stderr
TaskInfoId: 71
Offset: 0
Content:
File: StandardOutputFile, console_Stdout
TaskInfoId: 71
Offset: 0
Content: Input param: someStringParam
Iteration: 01
Iteration: 02

Finished
File: StandardErrorFile, console_Stderr
TaskInfoId: 71
Offset: 0
Content:
File: StandardOutputFile, console_Stdout
TaskInfoId: 71
Offset: 0
Content: Input param: someStringParam
Iteration: 01
Iteration: 02
Iteration: 03
Iteration: 04
Iteration: 05
Iteration: 06
Iteration: 07
Iteration: 08
Iteration: 09
Iteration: 10

Downloading file: /resultFile.txt
Downloading file: /console_Stdout
Downloading file: /console_Stderr
Press any key to continue . . .