astro.fits
Class FlatMaster

java.lang.Object
  util.PropertyContainer
      util.PropertyResources
          util.PropertyBundles
              astro.fits.ReductionMaster
                  astro.fits.FlatMaster

All Implemented Interfaces:

Cloneable, ExitCleaning, Initializable, LocalizedSupplying, Preserving, PropertySupplying, ResourceSupplying

public class FlatMaster
extends ReductionMaster
implements Preserving

We generate master flats out of a flat blocks, normally done straight after the bias blocks (before on dawn). After overscan correction, the closest master bias is subtracted from all flats. No dark correction is applied. Flats are grouped by derotator position, so normally we generate 0 and 180 deg flats, each from five exposures. The exposures are scaled by the mode and than combined using FitsStatistic.avSigClipCombine(java.util.List, astro.fits.FitsStatistic.Scale, boolean, double, double, int, boolean, int). Combining flats is normally a median operation, see KEY_FLATCOMBINEAV. The combined flat is then normalized using the same scaling possibilities as for individual combine. Normally, an average normalization would conserve the gain, thus average normalization is the choice for well-behaved CCD. If the flat shows many dark feature, but little bright features, a mode normalization may yield better results. Those five-exp normalized flats are stored as norm flats with the extension 000 and 180 to mark their individual gradient direction. Only if both flats could be generated, a master flat is constructed as the average of the two. If the #KEY_DIVFLAT points to true, the division of both masters is also generated. flatminima_U=5000,5000,5000,5000 flatmaxima_U=55000,55000,55000,55000 flatminima_B=5000,5000,5000,5000 flatmaxima_B=55000,55000,55000,55000 flatminima_V=5000,5000,5000,5000 flatmaxima_V=55000,55000,55000,55000 flatminima_R=5000,5000,5000,5000 flatmaxima_R=55000,55000,55000,55000 flatminima_I=5000,5000,5000,5000 flatmaxima_I=55000,55000,55000,55000 flatminima_u=5000,5000,5000,5000 flatmaxima_u=55000,55000,55000,55000 flatminima_v=5000,5000,5000,5000 flatmaxima_v=55000,55000,55000,55000 flatminima_b=5000,5000,5000,5000 flatmaxima_b=55000,55000,55000,55000 flatminima_y=5000,5000,5000,5000 flatmaxima_y=55000,55000,55000,55000 flatminima_up=5000,5000,5000,5000 flatmaxima_up=55000,55000,55000,55000 flatminima_gp=5000,5000,5000,5000 flatmaxima_gp=55000,55000,55000,55000 flatminima_rp=5000,5000,5000,5000 flatmaxima_rp=55000,55000,55000,55000 flatminima_ip=5000,5000,5000,5000 flatmaxima_ip=55000,55000,55000,55000 flatminima_zp=5000,5000,5000,5000 flatmaxima_zp=55000,55000,55000,55000 flatminima_han=5000,5000,5000,5000 flatmaxima_han=55000,55000,55000,55000 flatminima_haw=5000,5000,5000,5000 flatmaxima_haw=55000,55000,55000,55000 flatminima_hbn=5000,5000,5000,5000 flatmaxima_hbn=55000,55000,55000,55000 flatminima_hbw=5000,5000,5000,5000 flatmaxima_hbw=55000,55000,55000,55000

Nested Class Summary

Nested classes/interfaces inherited from class astro.fits.ReductionMaster

ReductionMaster.ImageBlock, ReductionMaster.ImageInfo, ReductionMaster.Interpolate, ReductionMaster.Listing, ReductionMaster.MasterFits

Nested classes/interfaces inherited from class util.PropertyResources

PropertyResources.URLResource

Field Summary

static String	CROSSTALK A file name extension for temporarily saved crosstalk-corrected fits.
private static boolean	DEBUG
private static double	DEFALLOWSPANAMBTEMP The maximum spread in det temps.
private static double	DEFALLOWSPANCCDTEMP The maximum spread in det temps.
private static long	DEFALLOWSPANTIME The maximum spread in flat observations.
private static String	DEFBESTRULES How we interpolate.
private static String	DEFBESTSPANS For each rule, we must exceed this value in the master difference.
private static String	DEFFLATBIASPURGE The cards to remove from the subtracted bias.
private static int	DEFFLATCOUNTMIN The default minimum number of useable bias .
private static double	DEFFLATTREND A trend may only be called a trend, if ADU difference exceeds this.
private static String	DEFFLATWEIGHT The cards used for weighed averages.
private static String	DEFOVERSCANCARDS The cards used from the overscan.
private static double	DEFROTATION The minimum derotator difference for 180 deg.
private static long	DEFROTBLOCK The maximum time difference between a zero and an 180 deg block.
private static String	DEFSERIALIZE The default serialization file.
private Map<ReductionMaster.ImageBlock,ReductionMaster.MasterFits>	done The list of image blocks already reduced.
private FlatCheck	flatcheck We need a flat checker to allow in-depth check of flat quality.
private Map<String,NavigableMap<Date,ReductionMaster.MasterFits>>	flatlookup A doubly-folded mapping of filter names and dates to master flats.
static String	KEY_FLATBIASPURGE The cards to be purged from the subtracted bias.
static String	KEY_FLATCOMBINEAV
static String	KEY_FLATCOUNTMIN The minimum number of useable darks to remain in a block.
static String	KEY_FLATHISIGMA
static String	KEY_FLATKEEP
static String	KEY_FLATLOSIGMA
static String	KEY_FLATNORMALIZE
static String	KEY_FLATPURGEAMP The cards to be purged from the constructed extensions.
static String	KEY_FLATREJECTLOOPS
static String	KEY_FLATSCALING How to scale the different flat to same luminosity.
static String	KEY_FLATSPEEDCLIP
static String	KEY_FLATTREND A drift in the bias is present, if this ADU difference is exceeded.
static String	KEY_FLATWEIGHT In constructing a intermediate master, keep this keys as weighed av.
static String	KEY_NORMHEAD The intermediate flats, normalized but not gradient corrected.
static String	KEY_OVERSCANCARDS The cards used from the overscan.
static String	KEY_ROTATION The minimum derotator difference for 180 deg.
static String	KEY_ROTBLOCK The maximum time difference between a zero and an 180 deg block.
static String	KEY_SERIALIZE On restarts, these are the image blocks we have already checked.
private BiasMaster	masterbias We need a master bias to allow the extraction of the bias.
static String	NORMFAC Header key denoting how normalization was done.
static String	NORMFACREM Header key denoting how normalization was done.
static String	NORMHOW Header key denoting how normalization was done.
static String	NORMHOWREM Header key denoting how normalization was done.

Fields inherited from class astro.fits.ReductionMaster

ADUSEPARATOR, DEFLOCALNFS, FIRSTNIGHT, gain, INPOL, INPOLREM, IPOLANTE, IPOLANTEREM, IPOLBASE, IPOLBASEREM, IPOLDRFT, IPOLDRFTREM, IPOLPOST, IPOLPOSTREM, IPOLTIME, IPOLTIMEREM, KEY_ALLOWSPANAMBTEMP, KEY_ALLOWSPANCCDTEMP, KEY_ALLOWSPANTIME, KEY_AMBTEMP, KEY_AMPLIFIERS, KEY_AVCARDS, KEY_BESTRULES, KEY_BESTSPANS, KEY_CCDTEMP, KEY_COPYCARDS, KEY_DEROTSEPARATE, KEY_EQUALCARDS, KEY_EXTCARDORDER, KEY_EXTRAPOLATE, KEY_FILTERAWAREADU, KEY_FILTERSEPARATE, KEY_FIRSTCARDS, KEY_FITSSETBRIDGE, KEY_GAIN, KEY_GENERATECARDS, KEY_IMCOPY, KEY_IMCOPYFLAG, KEY_INTRINSIC, KEY_LASTCARDS, KEY_LOCALFITS, KEY_LOCALNFS, KEY_LOCALTMP, KEY_MASTERHEAD, KEY_MAXCARDS, KEY_MAXEXPOSURETIME, KEY_MEDIANCARDS, KEY_MINCARDS, KEY_OBSDB, KEY_OVERSCAN, KEY_PRIMCARDORDER, KEY_QUADRANTS, KEY_QUADRANTSIZE, KEY_READNOISE, KEY_REDUCED, KEY_ROTTOLERANCE, KEY_SET, KEY_SITE, KEY_STRICTADU, KEY_TELESCOPE, PREFIX_ADUMAX, PREFIX_ADUMIN, PREFIX_AMPCARDS, readnoise

Fields inherited from class util.PropertyBundles

KEY_LOCALECOUNTRY, KEY_LOCALELANGUAGE, KEY_RESOURCEBUNDLES

Fields inherited from class util.PropertyResources

KEY_NOINITONCREATE, localurl, locate, POSTFIX_DIR, POSTFIX_EXT, POSTFIX_FILE, POSTFIX_LIST, POSTFIX_URL, urlset

Fields inherited from class util.PropertyContainer

KEY_LISTSEPARATOR, KEY_MAPKEYVALUECHAR, KEY_MAPSEPARATOR

Fields inherited from interface util.ResourceSupplying

KEY_URLRESOURCES, KEY_URLUSECONFIG, KEY_URLUSECURRENT, KEY_URLUSEHOME

Fields inherited from interface util.PropertySupplying

CONFIG, KEY_CLASS

Constructor Summary

FlatMaster(Map<String,String> prop)

Method Summary

private boolean	adjacentFlat(ReductionMaster.ImageBlock first, ReductionMaster.ImageBlock second) We check two image blocks if they have opposing rotation, identical filter and not too much time differrence.
private ReductionMaster.MasterFits	averageRotated(ReductionMaster.MasterFits rot180, ReductionMaster.MasterFits rot000)
private Map<String,List<FitsCheck.Failure>>	checkBlock(ReductionMaster.ImageBlock ib, Map<String,File> u2fits) Does a simple ADU-statistic check on all file in the image block, followed by the more sophisticated checks in FlatCheck
private ReductionMaster.MasterFits	createMasterFlat(ReductionMaster.ImageBlock flat, nom.tam.fits.Fits masterbias, SettableBoolean recover) We create a master fits from a fits observing block, identical rotation.
private Map<String,File>	crosstalkCorrectAll(Map<String,File> raw) We cross-talk correct all fits in the look-up list and store them temporarily.
private File	crosstalkSingle(File fits) We cross-talk correct an uncompressed fits file.
void	exit() On exit, we save all treated bias blocks.
nom.tam.fits.Fits	getBestFlat(ReductionMaster.ImageInfo ii) We return the best dark that should be used for the image in this image info.
File	getClosestFlatFile(String filter, Date d) We return the master dark file that is closest to the given date in question.
private ReductionMaster.MasterFits	getClosestFlatMaster(String filter, Date d) Returns the master fits that is pointing closest to the given date in question.
File[]	getEnclosingFlatFile(String filter, Date d) We return the to master dark that are closest before and after this date.
private ReductionMaster.MasterFits[]	getEnclosingFlatMaster(String filter, Date d) We return the to master fits that are closest before and after this date.
void	init() On init, we restore the list of all treated bias blocks.
boolean	preserve() On preserving, we try to store the already treated image blocks.
private int	reduceNight(Date now) We scan the given night, first get all observing blocks, secondly check if they are present in our done list.
boolean	restore() On restoring, we try to read the already treated image blocks.
protected void	setBiasMaster(BiasMaster bm) We set the helper application that deals with bias fits.
static ObserveSql.Types	typeOf()

Methods inherited from class astro.fits.ReductionMaster

addAverageCards, addEqualCards, addGeneratorCards, addMaximumCards, addMedianCards, addMinimumCards, addSingleCards, aduFailure, aduStatistic, allTransferred, alreadyTested, checkAmbTempSpan, checkDetTempSpan, compress, crosstalkCorrect, existing, existingFits, getAduConstraints, getHeaderValueStatistic, getMasterCards, getMasterCardsFromPrimary, getMidnight, getReducedParentDir, getTemporalFits, getUncompressed, intermediate, markTested, merge, merge, overscanCorrect, primaryHDU, readPrimaryCards, removeUncompressed, restoreMasterFits, scanNight, secondaryHDU, skipToHDU, uncompress, uncompressEventually

Methods inherited from class util.PropertyBundles

clone, getLocalized, getLocalized, getLocalizedString, getLocalizedString, loadResource

Methods inherited from class util.PropertyResources

createFrom, createFrom, createFrom, getApplet, getAsResources, getLocalClassLoader, getPropertiesToKey, getPropertiesToKey, getResource, getResourceAsStream, getResourceFromKey, getResources, keyCreate, keyCreate, reload, setApplet

Methods inherited from class util.PropertyContainer

augment, augment, augment, defaultBoolean, defaultChar, defaultDouble, defaultFloat, defaultInt, defaultLong, defaultObject, defaultObject, defaultProperties, defaultProperty, getAsBoolean, getAsChar, getAsDouble, getAsEnums, getAsFloat, getAsInt, getAsList, getAsLong, getAsMap, getAsMap, getAsObject, getAsObject, getProperties, getProperty, has, isNew, parseObject, reload, removeProperty, rescanned, setObject, setProperties, setProperty, stringProperties, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface util.ResourceSupplying

getResource, getResourceAsStream, getResources

Methods inherited from interface util.PropertySupplying

defaultBoolean, defaultChar, defaultDouble, defaultFloat, defaultInt, defaultLong, defaultObject, defaultObject, defaultProperties, defaultProperty, getAsBoolean, getAsChar, getAsDouble, getAsFloat, getAsInt, getAsList, getAsLong, getAsMap, getAsObject, getAsObject, getProperties, getProperty, has, parseObject, removeProperty, setObject, setProperty, stringProperties

Field Detail

DEBUG

private static final boolean DEBUG

See Also:

Constant Field Values

NORMHOW

public static final String NORMHOW

Header key denoting how normalization was done.

See Also:

Constant Field Values

NORMHOWREM

public static final String NORMHOWREM

Header key denoting how normalization was done.

See Also:

Constant Field Values

NORMFAC

public static final String NORMFAC

Header key denoting how normalization was done.

See Also:

Constant Field Values

NORMFACREM

public static final String NORMFACREM

Header key denoting how normalization was done.

See Also:

Constant Field Values

CROSSTALK

public static final String CROSSTALK

A file name extension for temporarily saved crosstalk-corrected fits.

See Also:

Constant Field Values

KEY_NORMHEAD

public static final String KEY_NORMHEAD

The intermediate flats, normalized but not gradient corrected.

See Also:

Constant Field Values

KEY_SERIALIZE

public static final String KEY_SERIALIZE

On restarts, these are the image blocks we have already checked.

See Also:

Constant Field Values

KEY_FLATTREND

public static final String KEY_FLATTREND

A drift in the bias is present, if this ADU difference is exceeded.

See Also:

Constant Field Values

KEY_FLATWEIGHT

public static final String KEY_FLATWEIGHT

In constructing a intermediate master, keep this keys as weighed av.

See Also:

Constant Field Values

KEY_FLATCOUNTMIN

public static final String KEY_FLATCOUNTMIN

The minimum number of useable darks to remain in a block.

See Also:

Constant Field Values

KEY_OVERSCANCARDS

public static final String KEY_OVERSCANCARDS

The cards used from the overscan.

See Also:

Constant Field Values

KEY_FLATPURGEAMP

public static final String KEY_FLATPURGEAMP

The cards to be purged from the constructed extensions.

See Also:

Constant Field Values

KEY_FLATBIASPURGE

public static final String KEY_FLATBIASPURGE

The cards to be purged from the subtracted bias.

See Also:

Constant Field Values

KEY_ROTATION

public static final String KEY_ROTATION

The minimum derotator difference for 180 deg.

See Also:

Constant Field Values

KEY_ROTBLOCK

public static final String KEY_ROTBLOCK

The maximum time difference between a zero and an 180 deg block.

See Also:

Constant Field Values

KEY_FLATSCALING

public static final String KEY_FLATSCALING

How to scale the different flat to same luminosity.

See Also:

Constant Field Values

KEY_FLATCOMBINEAV

public static final String KEY_FLATCOMBINEAV

See Also:

Constant Field Values

KEY_FLATLOSIGMA

public static final String KEY_FLATLOSIGMA

See Also:

Constant Field Values

KEY_FLATHISIGMA

public static final String KEY_FLATHISIGMA

See Also:

Constant Field Values

KEY_FLATKEEP

public static final String KEY_FLATKEEP

See Also:

Constant Field Values

KEY_FLATSPEEDCLIP

public static final String KEY_FLATSPEEDCLIP

See Also:

Constant Field Values

KEY_FLATREJECTLOOPS

public static final String KEY_FLATREJECTLOOPS

See Also:

Constant Field Values

KEY_FLATNORMALIZE

public static final String KEY_FLATNORMALIZE

See Also:

Constant Field Values

DEFSERIALIZE

private static final String DEFSERIALIZE

The default serialization file.

See Also:

Constant Field Values

DEFFLATCOUNTMIN

private static final int DEFFLATCOUNTMIN

The default minimum number of useable bias .

See Also:

Constant Field Values

DEFALLOWSPANTIME

private static final long DEFALLOWSPANTIME

The maximum spread in flat observations.

See Also:

Constant Field Values

DEFALLOWSPANCCDTEMP

private static final double DEFALLOWSPANCCDTEMP

The maximum spread in det temps.

See Also:

Constant Field Values

DEFALLOWSPANAMBTEMP

private static final double DEFALLOWSPANAMBTEMP

The maximum spread in det temps.

See Also:

Constant Field Values

DEFBESTRULES

private static final String DEFBESTRULES

How we interpolate.

See Also:

Constant Field Values

DEFBESTSPANS

private static final String DEFBESTSPANS

For each rule, we must exceed this value in the master difference.

See Also:

Constant Field Values

DEFFLATTREND

private static final double DEFFLATTREND

A trend may only be called a trend, if ADU difference exceeds this.

See Also:

Constant Field Values

DEFROTATION

private static final double DEFROTATION

The minimum derotator difference for 180 deg.

See Also:

Constant Field Values

DEFROTBLOCK

private static final long DEFROTBLOCK

The maximum time difference between a zero and an 180 deg block.

See Also:

Constant Field Values

DEFOVERSCANCARDS

private static final String DEFOVERSCANCARDS

The cards used from the overscan.

See Also:

Constant Field Values

DEFFLATWEIGHT

private static final String DEFFLATWEIGHT

The cards used for weighed averages.

See Also:

Constant Field Values

DEFFLATBIASPURGE

private static final String DEFFLATBIASPURGE

The cards to remove from the subtracted bias.

See Also:

Constant Field Values

masterbias

private BiasMaster masterbias

We need a master bias to allow the extraction of the bias.

flatcheck

private FlatCheck flatcheck

We need a flat checker to allow in-depth check of flat quality.

done

private Map<ReductionMaster.ImageBlock,ReductionMaster.MasterFits> done

The list of image blocks already reduced.

flatlookup

private Map<String,NavigableMap<Date,ReductionMaster.MasterFits>> flatlookup

A doubly-folded mapping of filter names and dates to master flats.

Constructor Detail

FlatMaster

public FlatMaster(Map<String,String> prop)

Method Detail

typeOf

public static final ObserveSql.Types typeOf()

init

public void init()

On init, we restore the list of all treated bias blocks.

Specified by:

init in interface Initializable

Overrides:

init in class ReductionMaster

exit

public void exit()

On exit, we save all treated bias blocks.

Specified by:

exit in interface ExitCleaning

preserve

public boolean preserve()

On preserving, we try to store the already treated image blocks.

Specified by:

preserve in interface Preserving

Returns:

false, if preserving the current state was not possible

restore

public boolean restore()

On restoring, we try to read the already treated image blocks.

Specified by:

restore in interface Preserving

Returns:

False, if restoring the value was not possible.

setBiasMaster

protected void setBiasMaster(BiasMaster bm)

We set the helper application that deals with bias fits.

getClosestFlatMaster

private ReductionMaster.MasterFits getClosestFlatMaster(String filter,
                                                        Date d)

Returns the master fits that is pointing closest to the given date in question.

getEnclosingFlatMaster

private ReductionMaster.MasterFits[] getEnclosingFlatMaster(String filter,
                                                            Date d)

We return the to master fits that are closest before and after this date. These two dark could then be used to interpolate to the 'best' dark.

Returns:

Null array if none found, otherwise always an array of size two, may have zero on 0/1.

getClosestFlatFile

public File getClosestFlatFile(String filter,
                               Date d)

We return the master dark file that is closest to the given date in question.

getEnclosingFlatFile

public File[] getEnclosingFlatFile(String filter,
                                   Date d)

We return the to master dark that are closest before and after this date. These two dark could then be used to interpolate to the 'best' dark.

Returns:

Null array if none found, otherwise always an array of size two, may have zero on 0/1.

getBestFlat

public nom.tam.fits.Fits getBestFlat(ReductionMaster.ImageInfo ii)
                              throws IOException,
                                     nom.tam.fits.FitsException

We return the best dark that should be used for the image in this image info. The Fits object is constructed from the two enclosing dark, sacling according to the rules specified in ReductionMaster.KEY_BESTRULES.

Throws:

IOException

nom.tam.fits.FitsException

reduceNight

private int reduceNight(Date now)
                 throws IOException,
                        nom.tam.fits.FitsException,
                        SQLException

We scan the given night, first get all observing blocks, secondly check if they are present in our done list. If not, we process them, mark them as done in the list and store the final master in the lookup table.

Returns:

Number of new master bias generated.

Throws:

IOException

nom.tam.fits.FitsException

SQLException

adjacentFlat

private boolean adjacentFlat(ReductionMaster.ImageBlock first,
                             ReductionMaster.ImageBlock second)

We check two image blocks if they have opposing rotation, identical filter and not too much time differrence. Both are guaranteed not null.

averageRotated

private ReductionMaster.MasterFits averageRotated(ReductionMaster.MasterFits rot180,
                                                  ReductionMaster.MasterFits rot000)
                                           throws IOException,
                                                  nom.tam.fits.FitsException

Throws:

IOException

nom.tam.fits.FitsException

createMasterFlat

private ReductionMaster.MasterFits createMasterFlat(ReductionMaster.ImageBlock flat,
                                                    nom.tam.fits.Fits masterbias,
                                                    SettableBoolean recover)
                                             throws IOException,
                                                    nom.tam.fits.FitsException

We create a master fits from a fits observing block, identical rotation.

Throws:

IOException

nom.tam.fits.FitsException

checkBlock

private Map<String,List<FitsCheck.Failure>> checkBlock(ReductionMaster.ImageBlock ib,
                                                       Map<String,File> u2fits)
                                                throws IOException,
                                                       nom.tam.fits.FitsException

Does a simple ADU-statistic check on all file in the image block, followed by the more sophisticated checks in FlatCheck

Throws:

IOException

nom.tam.fits.FitsException

See Also:

FlatCheck

crosstalkCorrectAll

private Map<String,File> crosstalkCorrectAll(Map<String,File> raw)
                                      throws IOException,
                                             nom.tam.fits.FitsException

We cross-talk correct all fits in the look-up list and store them temporarily.

Throws:

IOException

nom.tam.fits.FitsException

crosstalkSingle

private File crosstalkSingle(File fits)
                      throws IOException,
                             nom.tam.fits.FitsException

We cross-talk correct an uncompressed fits file. If the input file is a temporal file, we remove it.

Throws:

IOException

nom.tam.fits.FitsException