NCEP FTP Server Usage Guidelines

Background
----------
Our environment consists of five load-balanced servers. Each
server allows up to 300 connections, with a per-client limit of
five per server. Theoretically, one client can have up to twenty
five connections, but this is affected by how the load balancer
distributes the connections. 

Recommendations for Use
-----------------------
1) Some users connect from two or three systems to get past
   the connection limits. We currently allow this to happen,
   because it hasn't affected network bandwidth nor taken
   us near the maximum number of connections.  We reserve
   the right to disallow this behaviour, without notice.

2) Do not establish open IDLE connections.  It is not
   considered polite.  We have a 2 minute timeout on idle
   connections for this purpose.  We reserve the right to
   reduce the idle timeout, without notice.
   
3) For multiple connections, insert a few seconds delay before
   spawning each client. This increases the likelyhood that the
   load balancer will distribute the clients across all servers.
   Multiple parallel connections in one second result in all the
   clients connecting to the same server.

4) Refrain from using the FTP "DIR" or "LS" commands. A better
   approach would be to have a lookup table of the filenames you
   want to retrieve. Do a "GET" of each file in the table. If the
   file is not on the server yet, FTP will return an error. If
   you get an error, place the filename in a separate table, for
   retries. Perform the retries at a friendly interval, like
   every five minutes, until the file is retrieved, or a retry
   limit is reached.

5) If a file retrieval fails, don't immediately attempt to
   retrieve the file again.  Wait a few minutes between retries.
   We recommend a minimum five(5) minute interval.
   
6) Please use a valid email address on login. If you have a
   runaway script, we can at least attempt to notify you. Most
   times, we are forced to deny access to the abusive system
   until the user gets in touch with us.

7) We have found the performance to be similar, whether using FTP
   or HTTP.  Switch to using HTTP.  This is friendlier to our
   servers because it does not have the processing overhead of
   creating, then tearing down a client session.  The URL for the
   HTTP interface is http://www.ftp.ncep.noaa.gov/.

------------------------------------------------------------------
Author:  Ricardo Romero (ncep.helpdesk@noaa.gov)
Organization: NCEP Central Operations
Created: 14-Feb-2006  Last Modified: 16-Feb-2006