Digestor API¶

digestor¶

Scripts and metadata for loading survey data into the Data Lab database.

digestor.base¶

Base class containing common functionality.

class digestor.base.Digestor(schema, table, description=None, merge=None, pixels=True, random=True, ecliptic=True, galactic=True)[source]¶

Base class for FITS+SQL to FITS+SQL conversion.

Parameters

schema (str) – Name of the PostgreSQL schema containing table.
table (str) – Name of the PostgreSQL table.
description (str, optional) – A short description of schema.
merge (str, optional) – Name of a JSON file containing existing TapSchema metadata.
pixels (bool, optional) – If False, don’t HTM and HEALPix columns.
random (bool, optional) – If False, don’t add a random_id column.
ecliptic (bool, optional) – If False, don’t add ecliptic coordinates (probably because they already exist).
galactic (bool, optional) – If False, don’t add galactic coordinates (probably because they already exist).

_dlColumns()[source]¶

Add SQL column definitions of Data Lab-added columns.

Returns: A list suitable for appending to an existing list of columns.
Return type: list

_getYAML(filename)[source]¶

Cache reads of YAML configuration files.

Parameters: filename (str) – Name of the YAML configuration file.
Returns: The contents of filename, or None if there is no such file.
Return type: dict

_initTapSchema(description='', merge=None)[source]¶

Create a dictionary compatible with TapSchema.

Parameters

description (str, optional) – A short description of schema.
merge (str, optional) – Name of a JSON file containing existing TapSchema metadata.

Returns

A dictionary compatible with TapSchema.

Return type

dict

Raises

ValueError – When merging, if the schema names don’t match, or if the table is already loaded.

addDLColumns(filename, ra='ra', overwrite=False)[source]¶

Add DL columns to FITS file prior to column reorganization.

Parameters

filename (str) – Name of the FITS file.
ra (str, optional) – Look for Right Ascension in this column (default ‘ra’).
overwrite (bool, optional) – If True, remove any existing file.

Returns

The name of the processed file.

Return type

str

Raises

ValueError – If a problem with stilts is detected.

property colNames¶: List of columns in the table.

columnIndex(column)[source]¶

Find the index of the column in the list of columns.

Raises: ValueError – If the column is not found.

classmethod configureLog(filename, debug=False)[source]¶

Set up logging for the module.

Parameters

filename (str) – Name of the log file.
debug (bool, optional) – If True, set log level to DEBUG.

createSQL()[source]¶

Construct a CREATE TABLE statement from the TapSchema metadata.

Returns: A SQL table definition.
Return type: str

customSTILTS(filename)[source]¶

Add (prepend) custom STILTS commands to the default command.

Parameters: filename (str) – Name of the YAML configuration file.

fixColumns(filename)[source]¶

Fix any table definition oddities “by hand”.

Parameters: filename (str) – Name of the YAML configuration file.
Raises: ValueError – If the configuration file contains an unknown column.

logName(method)[source]¶

Get a logger with name method.

Parameters: method (str) – Name of the log object. Will be appended to the root name.
Returns: A configured log object.
Return type: logging.Logger

mapColumns()[source]¶

Complete mapping of FITS table columns to SQL columns.

This method may need to be overridden by a subclass.

Raises: KeyError – If an expected mapping cannot be found.

property nColumns¶: Number of columns in the table.

parseFITS(filename, hdu=1)[source]¶

Read FITS metadata from filename.

Parameters

filename (str) – Name of the FITS file.
hdu (int, optional) – Read data from this HDU (default 1).

processFITS(hdu=1, overwrite=False)[source]¶

Convert a pre-processed FITS file into one ready for database loading.

This method may be overridden in subclasses with survey-specific requirements.

Parameters

hdu (int, optional) – Read data from this HDU (default 1).
overwrite (bool, optional) – If True, remove any existing file.

Returns

The name of the file written.

Return type

str

Raises

ValueError – If the FITS data type cannot be converted to SQL.

sortColumns()[source]¶: Sort the SQL columns for best performance.

property stable¶: Schema-qualified table name.

tableIndex()[source]¶

Find the index of the table in the list of tables.

Raises: ValueError – If the table is not found.

tapColumn(column, **kwargs)[source]¶

Create a TapSchema-compatible column definition.

Parameters: column (str) – Name of the column.
Returns: A column definition in TapSchema format.
Return type: dict

writeSQL(filename)[source]¶

Write the CREATE TABLE statement to filename.

Parameters: filename (str) – Name of the SQL file.

writeTapSchema(filename)[source]¶

Write the TapSchema metadata to a JSON file.

Parameters: filename (str) – Name of the JSON file.

digestor.sdss¶

Convert SDSS SQL (MS SQL Server) table definitions to Data Lab SQL (PostgreSQL).

class digestor.sdss.SDSS(*args, **kwargs)[source]¶

Convert SDSS FITS+SQL files into Data Lab-compatible forms.

_photoFlag(column, table)[source]¶

Handle photometric flags in SDSS data.

Parameters

column (dict) – A TapSchema column definition.
table (astropy.table.Table) – Table containing the input data.

Returns

The combined flags and flags2 data, or None if the column did not match.

Return type

numpy.ndarray

Raises

AssertionError – If the required columns are not present in the FITS file.

fixMapping(filename)[source]¶

Fix any FITS to SQL mapping problems using the YAML configuration file filename.

Parameters: filename (str) – Name of the YAML configuration file.

fixNOFITS(filename)[source]¶

Fix any missing data designated by --/F NOFITS using the YAML configuration file filename.

Parameters: filename (str) – Name of the YAML configuration file.

mapColumns()[source]¶

Complete mapping of FITS table columns to SQL columns.

Raises: KeyError – If an expected mapping cannot be found.

parseColumnMetadata(column, data)[source]¶

Parse the metadata for an individual column.

Parameters

column (str) – Name of the column.
data (str) – Metadata string extracted from the SQL file.

Returns

A tuple containing a dictionary containing the parsed metadata in TapSchema format and a FITS column name, if found.

Return type

tuple

parseLine(line)[source]¶

Parse a single line from a SQL file.

Parameters: line (str) – A single line from a SQL file.

Notes

Currently, the long description (--/T) is thrown out.

parseSQL(filename)[source]¶

Parse an entire SQL file.

Parameters: filename (str) – Name of the SQL file.

processFITS(hdu=1, overwrite=False)[source]¶

Convert a pre-processed FITS file into one ready for database loading.

Parameters

hdu (int, optional) – Read data from this HDU (default 1).
overwrite (bool, optional) – If True, remove any existing file.

Returns

The name of the file written.

Return type

str

Raises

ValueError – If the FITS data type cannot be converted to SQL.

writePOSTSQL(filename, pkey='objid')[source]¶

Write additional SQL commands needed after loading the table itself.

Parameters

filename (str) – Name of the SQL file.
pkey (str, optional) – Name of the PRIMARY KEY column (default ‘objid’).

writeSQL(filename)[source]¶

Write the CREATE TABLE statement to filename, along with any SQL commands to pre-load.

Parameters: filename (str) – Name of the SQL file.

digestor.sdss.get_options()[source]¶

Parse command-line options.

Returns: The parsed options.
Return type: argparse.Namespace

digestor.sdss.main()[source]¶

Entry-point for command-line script.

Returns: An integer suitable for passing to sys.exit().
Return type: int

digestor.view¶

Handle schema metadata for views.

digestor.view.get_options()[source]¶

Parse command-line options.

Returns: The parsed options.
Return type: argparse.Namespace

digestor.view.main()[source]¶

Entry-point for command-line script.

Returns: An integer suitable for passing to sys.exit().
Return type: int