Condor/Firewall

Long Winded Version

From Condor manual: http://www.cs.wisc.edu/condor/manual/v6.6/3_10Setting_Up.html#16640

3.10.8.2 Firewalls

If a Condor pool is completely behind a firewall, then no special consideration is needed. However, if there is a firewall between the machines within a Condor pool, then configuration variables may be set to force the usage of specific ports and to utilize a specific range of ports.

By default, Condor uses port 9618 for the condor_ collector daemon, 9614 for the condor_ negotiator daemon, and dynamic (apparently random) ports for everything else. See section 3.10.8, if dynamic ports are desired for the condor_ collector and condor_ negotiator daemons.

The configuration variables HIGHPORT and LOWPORT facilitate setting a restricted range of ports that Condor will use. This may be useful when some machines are behind a firewall. The configuration macros HIGHPORT and LOWPORT will restrict dynamic ports to the range specified. The configuration variables are fully defined in section 3.3.2. Note that both HIGHPORT and LOWPORT must be at least 1024 for Condor version 6.6.8.

The total number of ports needed depends on the size of the pool, the usage of the machines within the pool (which machines run which daemons), and the number of jobs that may execute at one time. Here we discuss how many ports are used by each participant in the system.

The central manager of the pool needs 5 + NEGOTIATOR_SOCKET_CACHE_SIZE ports for daemon communication, where NEGOTIATOR_SOCKET_CACHE_SIZE is specified in the configuration or defaults to the value 16.

Each execute machine (those machines running a condor_ startd daemon) requires 5 + (5 * number of virtual machines advertised by that machine) ports. By default, the number of virtual machines advertised will equal the number of physical CPUs in that machine.

Submit machines (those machines running a condor_ schedd daemon) require 5 + (5 * MAX_JOBS_RUNNING) ports. The configuration variable MAX_JOBS_RUNNING limits (on a per-machine basis, if desired) the maximum number of jobs. Without this configuration macro, the maximum number of jobs that could be simultaneously executing at one time is a function of the number of reachable execute machines.

Also be aware that HIGHPORT and LOWPORT only impact dynamic port selection used by the Condor system, and they do not impact port selection used by jobs submitted to Condor. Thus, jobs submitted to Condor that may create network connections may not work in a port restricted environment. For this reason, specifying HIGHPORT and LOWPORT is not going to produce the expected results if a user submit jobs to be executed under the PVM or MPI job universes.

Where desired, a local configuration for machines not behind a firewall can override the usage of HIGHPORT and LOWPORT, such that the ports used for these machines are not restricted. This can be accomplished by adding the following to the local configuration file of those machines not behind a firewall:

HIGHPORT = UNDEFINED LOWPORT = UNDEFINED

If the maximum number of ports allocated using HIGHPORT and LOWPORT is too few, socket binding errors of the form

failed to bind any port within <$LOWPORT> - <$HIGHPORT>

are like to appear repeatedly in log files.

Condor Head

Use the following calculation to find the number of necessary ports to open:

Daemon Communication 5 + NEGOTIATOR_SOCKET_CACHE_SIZE = 5 + 16 = 21

Add the following to condor_config.local

NEGOTIATOR_SOCKET_CACHE_SIZE = 16
HIGHPORT = 9621 # 9600 + 21 = 9621
LOWPORT = 9600

Open the following ports:

condor_collector     9618
condor_negotiator    9614
Daemon Communication 9600-9621

Submit Only

Use the following calculation to find the number of necessary ports to open:

condor_schedd Dynamic, 5 + (5 * MAX_JOBS_RUNNING) = 5+(5*100) = 505

Add the following to condor_config.local

MAX_JOBS_RUNNING = 100
HIGHPORT = 10005 # 9600 + 505 = 10005
LOWPORT = 9600

Open the following ports:

condor_collector 9618
condor_schedd    9600-10005

Run Only

Use the following calculation to find the number of necessary ports to open:

condor_startd Dynamic, 5 + (5 * NUM VMs) = 5+(5*2) = 15

Add the following to condor_config.local

MAX_JOBS_RUNNING = 100
HIGHPORT = 9615 # 9600 + 15 = 9615
LOWPORT = 9600

Open the following ports:

condor_startd 9600-9615

Run and Submit

Use the following calculation to find the number of necessary ports to open:

condor_startd Dynamic, 5 + (5 * NUM VMs) = 5+(5*2) = 15
condor_schedd Dynamic, 5 + (5 * MAX_JOBS_RUNNING) = 5+(5*100) = 505

Add the following to condor_config.local

MAX_JOBS_RUNNING = 100
HIGHPORT = 9615 # 9600 + 15 + 505= 10020
LOWPORT = 9600

Open the following ports:

condor_collector 9618
condor_startd    9600-10020
condor_schedd