stella.util
Class TransferListener

java.lang.Object
  util.PropertyContainer
      util.PropertyResources
          util.PropertyBundles
              util.rmi.RmiPropertyContainer
                  util.rmi.AbstractRmiListener
                      stella.util.TransferListener

All Implemented Interfaces:

Cloneable, Remote, EventListener, RmiErrorListener, ExitCleaning, Initializable, LocalizedSupplying, Preserving, PropertySupplying, Refreshing, ResourceSupplying, RmiEventListener, RmiExiting, RmiListener, RmiPropertyBearing, RmiStatusProviding

public class TransferListener
extends AbstractRmiListener
implements RmiErrorListener, Refreshing, RmiExiting, RmiStatusProviding, Preserving

A transfer listener is responsible for tranferring fits files from the local host to a remote archive. It is used mainly to transfer the science fits files from Tenerife to the AIP archive at pina or pera. It uses external programs, in particular scp, imcopy.

Nested Class Summary

static class	TransferListener.DirCheck Checks all .fitz files in this directory and move them to another location, if the are erronously put here.
static class	TransferListener.Inspect We inspect a saved transfer serial file and print out some info.
static class	TransferListener.Squeeze
private class	TransferListener.TransferThread This is the main transfer thread.
static class	TransferListener.UpdateLight We update the light-loss fields on all files since 20110528B-0006.

Nested classes/interfaces inherited from class util.PropertyResources

PropertyResources.URLResource

Field Summary

private List<File>	dbenter The list of files that have been correctly entered into the db.
private List<String>	dbheads For convenience, we store the db-relevant heads.
private static long	DEFCOMPRESSTIMEOUT Various timeouts in ms.
private static long	DEFCOPYDELAY Various timeouts in ms.
private static boolean	DEFDEBUG Default is debug mode.
static String	DEFER Files starting with this are preprocessed.
private static String	DEFIMCOPY Default path to imcopy.
private static String	DEFIMCOPYFLAG Default extension for compress.
private static String	DEFIMMEDIATE List of immediately to move to transferred file parts.
private static String	DEFLOCALTMP Directory where the compressed files are stored for transfer.
private static long	DEFMKDIRTIMEOUT Various timeouts in ms.
private static long	DEFPOLLTIME Various timeouts in ms.
private static String	DEFPRIVILEGE Default copy program.
private static String	DEFRSYNC Default copy program.
private static String	DEFRSYNCARGS Default copy program.
private static String	DEFTRANSFERDIR Default transferred (sub)dir.
private static long	DEFTRANSFERTIMEOUT Various timeouts in ms.
private static long	DEFWAKEUP Default wake-up intervall.
static DateFormat	DIRFORMAT How we format a directory on the remote machine.
private List<String>	dirlist The list of all remotely created directories.
private boolean	doexit Gets true on exit.
private List<File>	ignore All the files that are in scanned dir, but should be ignored.
private List<String>	immediate For convenience, we store the parts immediately removed.
static String	KEY_BIND If true we are available in the registry.
static String	KEY_CASTERS The list of error caster that notify me.
static String	KEY_COMPRESSTIMEOUT Various timeouts in ms.
static String	KEY_COPYDELAY The pure copy files have a delay built in.
static String	KEY_COPYFILES Files that should be copied only, ;-list of header,extension.
static String	KEY_DATADIRS The list of directories I scan for ongoing traffic.
static String	KEY_DATEKEYS If we have this, we scan non-regular date keys for the date-obs.
static String	KEY_DEBUG If this is, as in the default, true, no transfer is actually done.
static String	KEY_FITSDB The sub-list of above that also need a db-entry.
static String	KEY_FITSHEADS The list of file-name headers that are treated as fits.
static String	KEY_FITSMETHOD Fits files that need separate treatment, header to method.
static String	KEY_IMCOPY The path to the imcopy external program.
static String	KEY_IMCOPYFLAG The thing to append to the target fits for imcopy.
static String	KEY_IMMEDIATE Files that have one of these before a requested extension are del.
static String	KEY_LISTENHOST If we signal to a host that a file has been transferred.
static String	KEY_LISTENPORT If we signal to a host that a file has been transferred.
static String	KEY_LOCALTMP Directory where the compressed files are stored for transfer.
static String	KEY_MKDIRTIMEOUT Various timeouts in ms.
static String	KEY_MODELDRIFT Properties of the pinhole modelling.
static String	KEY_OBSERVEDB The obs database I update.
static String	KEY_POLLTIME Various timeouts in ms.
static String	KEY_PRESERVEFILE Path to the file where the state is preserved in case of exits.
static String	KEY_PRIVILEGE The fits files that are tranfered first.
static String	KEY_RSYNC The path to the scp external program.
static String	KEY_RSYNCARGS The command line arguments to the scp program.
static String	KEY_TARGETDIR The target host top-level directory.
static String	KEY_TARGETHOST The target host.
static String	KEY_TELESCOPE The telescope name we work on.
static String	KEY_TRANSFERDIR The sub-dir in data dir(s) where transfered files are moved to.
static String	KEY_TRANSFERTIMEOUT Various timeouts in ms.
static String	KEY_WAKEUP The wake-up intervall.
private Map<String,String>	letter The mapping of telescope name to letter.
private Map<String,Method>	methods Header keys to methods to call for pre-transfer of fits.
private ObserveSql	observesql The obs-db reference.
private List<String>	secunique All secure uniques, i.e.
private List<File>	secure This is the list of files that can be transferred for sure.
private boolean	signalexit Gets true on exit request.
private long	tenter Enter time of last refresh call.
private long	texit Exit time stamp of last refresh call.
private static TransferListener.TransferThread	transfer The thread that does the actual remote copy.

Fields inherited from class util.rmi.AbstractRmiListener

KEY_ALIVE, KEY_EXPORTPORT, KEY_INIT, KEY_LISTENERNAME, KEY_RETRY, KEY_RETRYSLEEP, KEY_SHUTDOWNDELAY

Fields inherited from class util.PropertyBundles

KEY_LOCALECOUNTRY, KEY_LOCALELANGUAGE, KEY_RESOURCEBUNDLES

Fields inherited from class util.PropertyResources

KEY_NOINITONCREATE, localurl, locate, POSTFIX_DIR, POSTFIX_EXT, POSTFIX_FILE, POSTFIX_LIST, POSTFIX_URL, urlset

Fields inherited from class util.PropertyContainer

KEY_LISTSEPARATOR, KEY_MAPKEYVALUECHAR, KEY_MAPSEPARATOR

Fields inherited from interface util.ResourceSupplying

KEY_URLRESOURCES, KEY_URLUSECONFIG, KEY_URLUSECURRENT, KEY_URLUSEHOME

Fields inherited from interface util.PropertySupplying

CONFIG, KEY_CLASS

Constructor Summary

TransferListener(Map<String,String> prop)
Creates a new transfer listener.

Method Summary

protected boolean	deregisterAtCaster(RmiCaster rmi, Class service) Deregisters at caster.
private int	execute(String exe, long timeout) Uses the given executable command line string and executes that with a specified timeout.
void	exit() Exits the receiver torso.
long	getRefreshInterval() This is our wakup interval.
void	init() Initializes the transfer listener and binds it to the registry.
private void	initObsSql()
boolean	isFixedRate() We want fixed delay, thus return false.
private boolean	lightLoss(String uniq, File median) This method takes a median-averaged guider file and does light-loss analysis on the pinhole using fixed pinhole diameter, but letting the pinhole position vary.
private void	loadObs(String unique, List<File> fits) From a list of fits file, bundle to a single unique, we update the obs table to reflect the fits headers.
static void	main(String[] arg) We start the transfer listener with its properties on the command line.
private List<File>	median(String head, String uniq, List<File> guide) This method takes a list of fits files and calculates the median of it.
private boolean	mkdir(String subdir, double jd) Creates a directory on the remote machine using ssh.
private int	preProcess(String unique, List<File> fits) This method is the master method for pre-processing completed unique-sorted fits files.
boolean	preserve() Preserves the current state of this transfer listener by writing all information to the KEY_PRESERVEFILE property.
void	refresh() On refresh, we scan all directories.
private boolean	register(List<String> casters, Class casterclass) Registers at a caster list.
protected boolean	registerAtCaster(RmiCaster rmi, Class service) Registers at the specified caster if the caster is one of the four possible caster instances this class listens to.
private void	relayFitsFile(File writtn) We receive a message that a fits file has been completely writtn.
boolean	restore() Restores the last state of this transfer listener by reading all information from the KEY_PRESERVEFILE property.
void	rmiErrorOccured(ErrorEvent ee) We pass the error received to our listeners.
Object	rmiProvideStatus() We return a long string with a multi-line status message.
private void	scanAll() On timed wakeups, we scan all our directories and see if new files have arrived.
private boolean	scanDir(File dir, List<String> heads, String ext, SortedMap<String,List<File>> treat) We scan a local directory for all transfer-relevant files.
private boolean	signal(String filename) Signals an external server that a file has been transferred.
long	signalForExit() Retunrs our shecduled time as the exit time.
private List<File>	spillover(String head, String uniq, List<File> guide) This method takes a list of spilled-over guiding frames, calculates the average and uses this fits to model the light losses.
private File	transferred(File f) Transfers a file into the KEY_TRANSFERDIR subfolder to the current directory.
private boolean	updateObs(File fits, String remotedir) After the file has been successfully copied into the archive, we update the transfered flag and the new path.
private void	updateUnique(Object uniq) We receive a message that a target with this specified unique has been completed, thus pre-processing of the guider files may commence.

Methods inherited from class util.rmi.AbstractRmiListener

allBound, blockBinding, blockBinding, disconnect, getIP, getListenerName, isConnected, listenerName, releaseBinding, releaseBinding, rescanned, startBinding, startBinding

Methods inherited from class util.rmi.RmiPropertyContainer

containsRemoteProperty, getRemoteProperties, getRemoteProperty, setRemoteProperty

Methods inherited from class util.PropertyBundles

clone, getLocalized, getLocalized, getLocalizedString, getLocalizedString, loadResource

Methods inherited from class util.PropertyResources

createFrom, createFrom, createFrom, getApplet, getAsResources, getLocalClassLoader, getPropertiesToKey, getPropertiesToKey, getResource, getResourceAsStream, getResourceFromKey, getResources, keyCreate, keyCreate, reload, setApplet

Methods inherited from class util.PropertyContainer

augment, augment, augment, defaultBoolean, defaultChar, defaultDouble, defaultFloat, defaultInt, defaultLong, defaultObject, defaultObject, defaultProperties, defaultProperty, getAsBoolean, getAsChar, getAsDouble, getAsEnums, getAsFloat, getAsInt, getAsList, getAsLong, getAsMap, getAsMap, getAsObject, getAsObject, getProperties, getProperty, has, isNew, parseObject, reload, removeProperty, setObject, setProperties, setProperty, stringProperties, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface util.rmi.RmiEventListener

getListenerName

Methods inherited from interface util.rmi.RmiListener

disconnect, isConnected

Methods inherited from interface util.ResourceSupplying

getResource, getResourceAsStream, getResources

Methods inherited from interface util.PropertySupplying

defaultBoolean, defaultChar, defaultDouble, defaultFloat, defaultInt, defaultLong, defaultObject, defaultObject, defaultProperties, defaultProperty, getAsBoolean, getAsChar, getAsDouble, getAsFloat, getAsInt, getAsList, getAsLong, getAsMap, getAsObject, getAsObject, getProperties, getProperty, has, parseObject, removeProperty, setObject, setProperty, stringProperties

Field Detail

DIRFORMAT

public static final DateFormat DIRFORMAT

How we format a directory on the remote machine.

DEFER

public static final String DEFER

Files starting with this are preprocessed. On transfer renamed.

See Also:

Constant Field Values

KEY_BIND

public static final String KEY_BIND

If true we are available in the registry.

See Also:

Constant Field Values

KEY_DEBUG

public static final String KEY_DEBUG

If this is, as in the default, true, no transfer is actually done.

See Also:

Constant Field Values

KEY_TELESCOPE

public static final String KEY_TELESCOPE

The telescope name we work on.

See Also:

Constant Field Values

KEY_CASTERS

public static final String KEY_CASTERS

The list of error caster that notify me.

See Also:

Constant Field Values

KEY_WAKEUP

public static final String KEY_WAKEUP

The wake-up intervall.

See Also:

Constant Field Values

KEY_DATADIRS

public static final String KEY_DATADIRS

The list of directories I scan for ongoing traffic.

See Also:

Constant Field Values

KEY_TRANSFERDIR

public static final String KEY_TRANSFERDIR

The sub-dir in data dir(s) where transfered files are moved to.

See Also:

Constant Field Values

KEY_OBSERVEDB

public static final String KEY_OBSERVEDB

The obs database I update. Necessarry only for non-default.

See Also:

Constant Field Values

KEY_TARGETHOST

public static final String KEY_TARGETHOST

The target host.

See Also:

Constant Field Values

KEY_TARGETDIR

public static final String KEY_TARGETDIR

The target host top-level directory.

See Also:

Constant Field Values

KEY_IMCOPY

public static final String KEY_IMCOPY

The path to the imcopy external program.

See Also:

Constant Field Values

KEY_IMCOPYFLAG

public static final String KEY_IMCOPYFLAG

The thing to append to the target fits for imcopy.

See Also:

Constant Field Values

KEY_RSYNC

public static final String KEY_RSYNC

The path to the scp external program.

See Also:

Constant Field Values

KEY_RSYNCARGS

public static final String KEY_RSYNCARGS

The command line arguments to the scp program.

See Also:

Constant Field Values

KEY_FITSHEADS

public static final String KEY_FITSHEADS

The list of file-name headers that are treated as fits.

See Also:

Constant Field Values

KEY_FITSDB

public static final String KEY_FITSDB

The sub-list of above that also need a db-entry.

See Also:

Constant Field Values

KEY_FITSMETHOD

public static final String KEY_FITSMETHOD

Fits files that need separate treatment, header to method.

See Also:

Constant Field Values

KEY_COPYFILES

public static final String KEY_COPYFILES

Files that should be copied only, ;-list of header,extension.

See Also:

Constant Field Values

KEY_IMMEDIATE

public static final String KEY_IMMEDIATE

Files that have one of these before a requested extension are del.

See Also:

Constant Field Values

KEY_LOCALTMP

public static final String KEY_LOCALTMP

Directory where the compressed files are stored for transfer.

See Also:

Constant Field Values

KEY_PRESERVEFILE

public static final String KEY_PRESERVEFILE

Path to the file where the state is preserved in case of exits.

See Also:

Constant Field Values

KEY_COMPRESSTIMEOUT

public static final String KEY_COMPRESSTIMEOUT

Various timeouts in ms.

See Also:

Constant Field Values

KEY_TRANSFERTIMEOUT

public static final String KEY_TRANSFERTIMEOUT

Various timeouts in ms.

See Also:

Constant Field Values

KEY_MKDIRTIMEOUT

public static final String KEY_MKDIRTIMEOUT

Various timeouts in ms.

See Also:

Constant Field Values

KEY_POLLTIME

public static final String KEY_POLLTIME

Various timeouts in ms.

See Also:

Constant Field Values

KEY_DATEKEYS

public static final String KEY_DATEKEYS

If we have this, we scan non-regular date keys for the date-obs.

See Also:

Constant Field Values

KEY_LISTENHOST

public static final String KEY_LISTENHOST

If we signal to a host that a file has been transferred.

See Also:

Constant Field Values

KEY_LISTENPORT

public static final String KEY_LISTENPORT

If we signal to a host that a file has been transferred.

See Also:

Constant Field Values

KEY_MODELDRIFT

public static final String KEY_MODELDRIFT

Properties of the pinhole modelling.

See Also:

Constant Field Values

KEY_COPYDELAY

public static final String KEY_COPYDELAY

The pure copy files have a delay built in.

See Also:

Constant Field Values

KEY_PRIVILEGE

public static final String KEY_PRIVILEGE

The fits files that are tranfered first.

See Also:

Constant Field Values

DEFDEBUG

private static final boolean DEFDEBUG

Default is debug mode.

See Also:

Constant Field Values

DEFWAKEUP

private static final long DEFWAKEUP

Default wake-up intervall.

See Also:

Constant Field Values

DEFTRANSFERDIR

private static final String DEFTRANSFERDIR

Default transferred (sub)dir.

See Also:

Constant Field Values

DEFIMCOPY

private static final String DEFIMCOPY

Default path to imcopy.

See Also:

Constant Field Values

DEFIMCOPYFLAG

private static final String DEFIMCOPYFLAG

Default extension for compress.

See Also:

Constant Field Values

DEFRSYNC

private static final String DEFRSYNC

Default copy program.

See Also:

Constant Field Values

DEFRSYNCARGS

private static final String DEFRSYNCARGS

Default copy program.

See Also:

Constant Field Values

DEFIMMEDIATE

private static final String DEFIMMEDIATE

List of immediately to move to transferred file parts.

See Also:

Constant Field Values

DEFLOCALTMP

private static final String DEFLOCALTMP

Directory where the compressed files are stored for transfer.

See Also:

Constant Field Values

DEFCOMPRESSTIMEOUT

private static final long DEFCOMPRESSTIMEOUT

Various timeouts in ms.

See Also:

Constant Field Values

DEFTRANSFERTIMEOUT

private static final long DEFTRANSFERTIMEOUT

Various timeouts in ms.

See Also:

Constant Field Values

DEFMKDIRTIMEOUT

private static final long DEFMKDIRTIMEOUT

Various timeouts in ms.

See Also:

Constant Field Values

DEFPOLLTIME

private static final long DEFPOLLTIME

Various timeouts in ms.

See Also:

Constant Field Values

DEFCOPYDELAY

private static final long DEFCOPYDELAY

Various timeouts in ms.

See Also:

Constant Field Values

DEFPRIVILEGE

private static final String DEFPRIVILEGE

Default copy program.

See Also:

Constant Field Values

observesql

private ObserveSql observesql

The obs-db reference.

dbheads

private List<String> dbheads

For convenience, we store the db-relevant heads.

immediate

private List<String> immediate

For convenience, we store the parts immediately removed.

ignore

private List<File> ignore

All the files that are in scanned dir, but should be ignored.

dbenter

private List<File> dbenter

The list of files that have been correctly entered into the db.

secure

private List<File> secure

This is the list of files that can be transferred for sure.

secunique

private List<String> secunique

All secure uniques, i.e. earlier-than-last uniques or two-scanned.

dirlist

private List<String> dirlist

The list of all remotely created directories.

letter

private Map<String,String> letter

The mapping of telescope name to letter.

methods

private Map<String,Method> methods

Header keys to methods to call for pre-transfer of fits.

transfer

private static TransferListener.TransferThread transfer

The thread that does the actual remote copy.

tenter

private long tenter

Enter time of last refresh call.

texit

private long texit

Exit time stamp of last refresh call.

doexit

private boolean doexit

Gets true on exit.

signalexit

private boolean signalexit

Gets true on exit request.

Constructor Detail

TransferListener

public TransferListener(Map<String,String> prop)
                 throws RemoteException

Creates a new transfer listener.

Throws:

RemoteException

Method Detail

init

public void init()

Initializes the transfer listener and binds it to the registry.

Specified by:

init in interface Initializable

Overrides:

init in class PropertyResources

initObsSql

private void initObsSql()

exit

public void exit()

Exits the receiver torso. Unregisters from the command caster and removes its registry binding.

Specified by:

exit in interface ExitCleaning

Overrides:

exit in class AbstractRmiListener

rmiErrorOccured

public void rmiErrorOccured(ErrorEvent ee)
                     throws RemoteException

We pass the error received to our listeners.

Specified by:

rmiErrorOccured in interface RmiErrorListener

Throws:

RemoteException

signalForExit

public long signalForExit()
                   throws RemoteException

Retunrs our shecduled time as the exit time.

Specified by:

signalForExit in interface RmiExiting

Throws:

RemoteException

rmiProvideStatus

public Object rmiProvideStatus()
                        throws RemoteException

We return a long string with a multi-line status message.

Specified by:

rmiProvideStatus in interface RmiStatusProviding

Throws:

RemoteException

refresh

public void refresh()

On refresh, we scan all directories.

Specified by:

refresh in interface Refreshing

isFixedRate

public boolean isFixedRate()

We want fixed delay, thus return false.

Specified by:

isFixedRate in interface Refreshing

See Also:

Timer

getRefreshInterval

public long getRefreshInterval()

This is our wakup interval.

Specified by:

getRefreshInterval in interface Refreshing

preserve

public boolean preserve()

Preserves the current state of this transfer listener by writing all information to the KEY_PRESERVEFILE property.

Specified by:

preserve in interface Preserving

Returns:

false, if preserving the current state was not possible

restore

public boolean restore()

Restores the last state of this transfer listener by reading all information from the KEY_PRESERVEFILE property.

Specified by:

restore in interface Preserving

Returns:

False, if restoring the value was not possible.

registerAtCaster

protected boolean registerAtCaster(RmiCaster rmi,
                                   Class service)

Registers at the specified caster if the caster is one of the four possible caster instances this class listens to.

Specified by:

registerAtCaster in class AbstractRmiListener

Returns:

True, if this listener can register at the specified caster

deregisterAtCaster

protected boolean deregisterAtCaster(RmiCaster rmi,
                                     Class service)

Deregisters at caster. Simply checks if the caster handed over is of one of the four supported types and removes itself from its listener list. Note that this can also mean de-registering from casters we never registered to because this info is lost.

Specified by:

deregisterAtCaster in class AbstractRmiListener

Returns:

True, if this listener can deregister at the specified caster

register

private boolean register(List<String> casters,
                         Class casterclass)

Registers at a caster list.

relayFitsFile

private void relayFitsFile(File writtn)

We receive a message that a fits file has been completely writtn. We put this file on the list of secure-to-transfer list immediately.

updateUnique

private void updateUnique(Object uniq)

We receive a message that a target with this specified unique has been completed, thus pre-processing of the guider files may commence.

scanDir

private boolean scanDir(File dir,
                        List<String> heads,
                        String ext,
                        SortedMap<String,List<File>> treat)

We scan a local directory for all transfer-relevant files. Additionally, we check for completed uniques, which are simply all uniques found except the highest.

scanAll

private void scanAll()

On timed wakeups, we scan all our directories and see if new files have arrived.

preProcess

private int preProcess(String unique,
                       List<File> fits)

This method is the master method for pre-processing completed unique-sorted fits files. It depends on the header-to-method mapping in KEY_FITSMETHOD. For each entry there, all fits files in this unique are collected and processed via the appropriate method. The collection is done by augmenting the header with the provided unique and grabbing all files from the list, whose names start accordingly. Grabbing is here a true transfer into the new list. The list returned by the method called is in turn re-entered into the transfer list. This method is also responsible for moving these collected fits file out of the way.

See Also:

median

median

private final List<File> median(String head,
                                String uniq,
                                List<File> guide)

This method takes a list of fits files and calculates the median of it.

spillover

private final List<File> spillover(String head,
                                   String uniq,
                                   List<File> guide)

This method takes a list of spilled-over guiding frames, calculates the average and uses this fits to model the light losses.

lightLoss

private boolean lightLoss(String uniq,
                          File median)

This method takes a median-averaged guider file and does light-loss analysis on the pinhole using fixed pinhole diameter, but letting the pinhole position vary.

loadObs

private void loadObs(String unique,
                     List<File> fits)

From a list of fits file, bundle to a single unique, we update the obs table to reflect the fits headers. Additionally we set the transferred flag to false, but enter the currently correct path to the fits file. After the file has been transfered, this path and the transferred flag should change.

updateObs

private boolean updateObs(File fits,
                          String remotedir)

After the file has been successfully copied into the archive, we update the transfered flag and the new path.

transferred

private File transferred(File f)

Transfers a file into the KEY_TRANSFERDIR subfolder to the current directory. Returns true on success.

signal

private boolean signal(String filename)

Signals an external server that a file has been transferred.

mkdir

private boolean mkdir(String subdir,
                      double jd)

Creates a directory on the remote machine using ssh.

execute

private int execute(String exe,
                    long timeout)

Uses the given executable command line string and executes that with a specified timeout.

main

public static void main(String[] arg)
                 throws RemoteException

We start the transfer listener with its properties on the command line.

Throws:

RemoteException