gavo.utils.fitstools module

Some utility functions to deal with FITS files.

It’s unclear if pyfits coming from astropy is now threadsafe, but I’m hoping it might be, so for now I’m making fitslock a no-op.

exception gavo.utils.fitstools.ESODescriptorsError(msg: str, offending: Optional[str] = None, location: str = 'unspecified location', source: str = '<unspecified source>', hint: Optional[str] = None)

Bases: SourceParseError

is raised when something goes wrong while parsing ESO descriptors.

exception gavo.utils.fitstools.FITSError

Bases: Exception

class gavo.utils.fitstools.WCSAxis(name: str, crval: float, crpix: float, cdelt: float, ctype: str = 'UNKNOWN', cunit: Optional[str] = '', axisLength: int = 1)

Bases: object

represents a single 1D WCS axis and allows easy metadata discovery.

You’ll usually use the fromHeader constructor.

The idea of using this rather than astropy.wcs or similar is that this is simple and robust. It doesn’t know many of the finer points of WCS, though, and in particular it’s 1D only.

However, for the purposes of cutouts it probably should do for the overwhelming majority of non-spatial FITS axes.

The default pixel coordinates are handled in the FITS sense here, i.e., the first pixel has the index 1. Three are methods that have pix0 in their names; these assume 0-based arrays. All the transforms return Nones unchanged.

To retrieve the metadata shoved in, use the name, crval, crpix, cdelt, ctype, cunit, and axisLength attributes.

classmethod fromHeader(header: Header, axisIndex: int, forceSeparable: bool = False) → WCSAxis

returns a WCSAxis for the specified axis in header.

If the axis is mentioned in a transformation matrix (CD or PC), a ValueError is raised; this is strictly for 1D coordinates.

The axisIndex is 1-based; to get a transform for the axis described by CTYPE1, pass 1 here.

getLimits() → Tuple[float, float]

returns the minimal and maximal physical values this axis takes within the image.

physToPix(physCoo: float) → float

returns a 1-based pixel coordinate for a physical value.

physToPix0(physCoo: float) → float

returns a 0-based pixel coordinate for a physical value.

pix0ToPhys(pix0Coo: float) → float

returns the physical value for a 0-based pixel coordinate.

pixToPhys(pixCoo: float) → float

returns the physical value for a 1-based pixel coordinate.

gavo.utils.fitstools.cutoutFITS(hdu: ImageHDU, *cuts: Tuple[Optional[float], ...]) → ImageHDU

returns a cutout of hdu restricted to cuts.

hdu is a primary FITS hdu. cuts is a list of cut specs, each of which is a triple (axis, lower, upper). axis is between 1 and naxis, lower and upper a 1-based pixel coordinates of the limits, and “border” pixels are included. Specifications outside of the image are legal and will be cropped back. Open limits are supported via a specification of None.

If an axis would vanish (i.e. length 0 or less), the function fudges things such that the axis gets a length of 1.

axis is counted here in the FORTRAN/FITS sense, not in the C sense, i.e., axis=1 cuts along NAXIS1, which is the last index in a numpy array.

WCS CRPIXes in hdu’s header will be updated. Axes and specified will not be touched. It is an error to specify cuts for an axis twice (behaviour is undefined).

Note that this will lose all extensions the original FITS file might have had.

gavo.utils.fitstools.cutoutHeader(header: Header, *cuts: Tuple[Optional[float], ...]) → Tuple[List, Header]

returns a array slices and a new pyfits header reflecting cuts.

(see cutoutFits for what cuts is).

gavo.utils.fitstools.fitsLock()

gavo.utils.fitstools.fixImageExtind(hdus: List, extInd: int) → int

returns the index of a compressed image HDU if it immediately follows extInd.

If hdus[extInd] is a CompImageHDU itself, this will return extInd unchanged.

gavo.utils.fitstools.getAxisLengths(hdr: Header) → List[int]

returns a sequence of the lengths of the axes of a FITS image described by hdr.

gavo.utils.fitstools.getWCSAxis(header: Header, axisIndex: int, forceSeparable: bool = False) → WCSAxis

returns a WCSAxis instance from an axis index and a FITS header.

If the axis is mentioned in a transformation matrix (CD or PC), a ValueError is raised (use forceSeparable to override).

The axisIndex is 1-based; to get a transform for the axis described by CTYPE1, pass 1 here.

The object returned has methods like pixToPhys, physToPix (and their pix0 brethren), and getLimits.

Note that at this point WCSAxis only supports linear transforms (it’s a DaCHS-specific implementation). We’ll extend it on request.

gavo.utils.fitstools.headerFromDict(d: Dict[str, str]) → Header

returns a primary header sporting the key/value pairs given in the dictionary d.

In all likelihood, this header will not work properly as a primary header because, e.g., there are certain rules on the sequence of header items. fitstricks.copyFields can make a valid header out of what’s returned here.

keys mapped to None are skipped, i.e., you have to do nullvalue handling yourself.

gavo.utils.fitstools.iterFITSRows(hdr: Header, f: BinaryIO) → Generator[ndarray[Any, dtype[ScalarType]], None, None]

iterates over the rows of a FITS (primary) image.

You pass in a FITS header and a file positioned to the start of the image data.

What’s returned are 1d numpy arrays of the datatype implied by bitpix. The function understands only very basic FITSes (BSCALE and BZERO are known, though, and lead to floats arrays).

We do this ourselves since pyfits may pull in the whole thing or at least mmaps it; both are not attractive when I want to stream-process large images.

gavo.utils.fitstools.iterScaledBytes(inFileName: str, factor: float, extraCards: Dict[str, str] = {}) → Generator[bytes, None, None]

iterates over the bytes for a simple FITS file generated by scaling down the 2D image inFileName by factor.

factor must be an integer between 1 and something reasonable.

This function acquires the FITS lock itself.

gavo.utils.fitstools.iterScaledRows(inFile: BinaryIO, factor: Optional[float] = None, destSize: Optional[int] = None, hdr: Optional[Header] = None, slow: bool = False, extInd: int = 0) → Generator[ndarray[Any, dtype[ScalarType]], None, None]

iterates over numpy arrays of pixel rows within the open FITS stream inFile scaled by it integer in factor.

The arrays are always float32, regardless of the input. When the image size is not a multiple of factor, border pixels are discarded.

A FITS header for this data can be obtained using shrinkWCSHeader.

You can pass in either a factor the file is to be scaled down, or an approximate size desired. If both are given, factor takes precedence, if none is given, it’s an error.

If you pass in hdr, it will not be read from inFile; the function then expects the file pointer to point to the start of the first data block. Use this if you’ve already read the header of a non-seekable FITS.

extInd lets you select a different extension. extInd=0 means the first image HDU, which may be in extension 1 for compressed images.

iterScaledRows will try to use a hand-hacked interface guaranteed to stream. This only works for plain, 2D-FITSes from real files. iterScaledRows normally notices when it should fall back to pyfits code, but if it doesn’t you can pass slow=True.

gavo.utils.fitstools.openFits(fitsName: str) → HDUList

returns the hdus for fName.

This is simulating legacy pyfits behaviour in that it will use memmap I/O if it thinks that works. You’ll probably want to use astropy directly these days if this gives you a headache.

gavo.utils.fitstools.padCard(input: Union[str, bytes], length: int = 80) → bytes

pads input (a string or bytes) with blanks until len(result)%80=0

The length keyword argument lets you adjust the “card size”. Use this to pad headers with length=FITS_BLOCK_SIZE

This always returns bytes.

>>> padCard("")
b''
>>> len(padCard(b"end"))
80
>>> len(padCard(b"whacko"*20))
160
>>> len(padCard("junkodumnk"*17, 27))%27
0

gavo.utils.fitstools.parseCards(aString: Union[str, bytes]) → Header

returns a list of pyfits Cards parsed from aString.

This will raise a ValueError if aString’s length is not divisible by 80. It may also raise pyfits errors for malformed cards.

Empty (i.e., all-whitespace) cards are ignored. If an END card is encountered processing is aborted.

gavo.utils.fitstools.parseESODescriptors(hdr: Header) → Dict[str, List[float]]

returns parsed ESO descriptors from a pyfits header hdr.

ESO descriptors are data columns stuck into FITS history lines. They were produced by MIDAS. This implementation was made without actual documentation, is largely based on conjecture, and is certainly incomplete.

What’s returned is a dictionary mapping column keywords to lists of column values.

gavo.utils.fitstools.readHeaderBytes(f: BinaryIO, maxHeaderBlocks: int = 80) → bytes

returns the bytes belonging to a FITS header starting at the current position within the file f.

If the header is not complete after reading maxHeaderBlocks blocks, a FITSError is raised.

gavo.utils.fitstools.readPrimaryHeaderQuick(f: BinaryIO, maxHeaderBlocks: int = 80) → Header

returns a pyfits header for the primary hdu of the opened file f.

f must be opened in binary mode.

This is mostly code lifted from pyfits._File._readHDU. The way that class is made, it’s hard to use it with stuff from a gzipped source, and that’s why this function is here. It is used in the quick mode of fits grammars.

This function is adapted from pyfits.

gavo.utils.fitstools.replacePrimaryHeader(inputFile: BinaryIO, newHeader: Header, targetFile: BinaryIO, bufSize: int = 100000) → None

writes a FITS to targetFile having newHeader as the primary header, where the rest of the data is taken from inputFile.

inputFile must be a file freshly opened for binary reading, targetFile one freshly opened for binary writing.

This function is (among other things) a workaround for pyfits’ misfeature of unscaling scaled data in images when extending a header.

gavo.utils.fitstools.replacePrimaryHeaderInPlace(fitsName: str, newHeader: Header) → None

replaces the primary header of fitsName with newHeader.

Doing this, it tries to minimize the amount of writing necessary; if fitsName has enough space for newHeader, just the header is written, and newHeader is extended if necessary. Only if newHeader is longer than the existing header is fitsName actually copied. We try to be safe in this case, only overwriting the old entry when the new data is safely on disk.

gzipped inputs used to be supported here, but they aren’t any more.

gavo.utils.fitstools.serializeHeader(hdr: Header) → bytes

returns the FITS serialization of a FITS header hdr.

This is always bytes.

gavo.utils.fitstools.shrinkWCSHeader(oldHeader: Header, factor: float) → Header

returns a FITS header suitable for a shrunken version of the image described by oldHeader.

This only works for 2d images, scale must be an integer>1. The function assumes no “fractional” pixels are handled, i.e, remainders in divisions with factors are discarded. It is thus a companion for iterScaledRows.

Note that oldHeader must be an actual pyfits header instance; a dictionary will not do.

This is a pretty straight port of wcstools’s imutil.c#ShrinkFITSHeader, except we clear BZERO and BSCALE and set BITPIX to -32 (float array) under the assumption that in the returned image, 32-bit floats are used.

gavo.utils.fitstools.sortHeaders(header: Header, commentFilter: Optional[Callable[[str], bool]] = None, historyFilter: Optional[Callable[[str], bool]] = None, cardSequence: Optional[List[Tuple[str, bool]]] = None) → Header

returns a pyfits header with “real” cards first, then history, then comment cards.

Blanks in the input are discarded, and one blank each is added in between the sections of real cards, history and comments.

Header can be an iterable yielding Cards or a pyfits header.

Duplicate history or comment entries will be swallowed.

cardSequence, if present, is a sequence of (item, mandatory) pairs. There item can be a card name, in which case the corresponding card will be inserted at this point in the sequence. If mandatory is True, a missing card is an error. Keywords already in fitstools.STANDARD_CARD_SEQUENCE are ignored.

Item can also a pyfits.Card instance; it will be put into the header as-is.

As a shortcut, a sequence item may be something else then a tuple; it will then be combined with a False to make one.

These days, if you think you need this, have a look at fitstricks.makeHeaderFromTemplate first.