Only available on EUMETSAT side. Currently in beta testing. This service is available as best effort for testing. Be aware when trying this service.

Introduction

The EWC solution for HTC batch processing service is HTCondor.

HTCondor is a specialized batch system for managing compute-intensive jobs. HTCondor provides a queuing mechanism, scheduling policy, priority scheme, and resource classifications.

Users can submit their compute jobs to HTCondor, HTCondor puts the jobs in a queue, runs them, and then informs the user as to the result.

General

EWC HTcondor is a managed service. The central manager node is deployed in a tenancy on the EWC. Users can join the existing pool adding compute and submit nodes.

European Weather Cloud Knowledge Base > EUMETSAT - HTC batch processing > Batch Processing on EWC - HTCondor.png

Some features of the HTCondor in EWC:

Maintenance	Centrally Managed Tenancy, easy 'one click' deployment
Deployment	Multi tenancy
Resource	Join automatically the main HTCondor Pool, no need for password or any configuration, only choose the plan for the machine your want to add
Usage	Easy 'one click' deployment, simple examples for running a job with docker univers
Network	VPN
Scheduling	Single schedulers in each tenancy, no possibility to erase other tenancies jobs

Execute nodes

No access to execute host for containers
No access to other containers running on execute node
Isolated environment for containers
No autoscaling
No NFS

Submit nodes

Only docker universe allowed
Only condor_submit command allowed
Private network in the tenancy enabled to allow access to tenancy-internal resources/files
Condor transfer mechanism allowed

Deploy HTCondor nodes

Pre-requisite

Before deploying an HTcondor node, you need to create an htcondor specific security group. You can follow this page: Creating Security Groups to know how to create security groups.

htcondor security group with the following rules:

Rule name	Direction	Rule Type	Protocol	Port Range	Source Type	Source	Destination Type
	egress	Custom Rule	TCP		All		Instance
	egress	Custom Rule	UDP		All		Instance
9618-tcp	ingress	Custom Rule	TCP	9618	Network	100.64.0.0/10	Instance

Deploy execute or submit node

Go to Provisioning → Instances and click on Add+ to add a new instance
Select Htcondor Submit/Execute node
Fill data required:
- plan: choose your plan
- network: private
- security group: htcondor, ssh
4. Finalize provisioning steps.

Once submit node is up: ssh into the machine and try to submit a job.

Once execute node is up: you can check from a submit node if the node appears in the list, running condor_status