gavo.user.upgrade module

Stuff dealing with the upgrade of the database schema.

From software version 0.8.2 on, there is a dc.metastore table with a key schemaversion. Each change in the central schema increases the value (interpreted as an integer) by one, and this module will contain a corresponding upgrader.

An upgrader inherits form the Upgrader class. See there for more details.

This module contains the current schemaversion expected by the software; gavo upgrade does everything required to bring the what’s in the database in sync with the code (or so I hope).

class gavo.user.upgrade.AnnotatedString(content, annotation)

Bases: str

a string with an annotation.

This is (optionally) used to hold SQL statements here; the annotation is shown to the user instead of the naked statement when present.

class gavo.user.upgrade.To0Upgrader

Bases: Upgrader

This is executed when there’s no schema version defined in the database.

The assumption is that the database reflects the status of 0.8, so it adds the author column in dc.services if necessary (which it’s not if the software has been updated to 0.8.1).

classmethod s_000_addauthor(connection)

add an author column to dc.services if necessary

classmethod u_010_makeMetastore(connection)

create the meta store

version = -1

class gavo.user.upgrade.To10Upgrader

Bases: Upgrader

classmethod u_000_dropADQLExamples(connection)

drop old TAP examples tables (gone to _examples meta)

classmethod u_010_createDLAsyncTable(connection)

import job table for async datalink

version = 9

class gavo.user.upgrade.To11Upgrader

Bases: Upgrader

classmethod u_000_findMixedinTables(connection)

inform about tables with non-trivial mixins

version = 10

class gavo.user.upgrade.To12Upgrader

Bases: Upgrader

classmethod u_010_updateTAPRecord(connection)

prettify the TAP record’s IVOID

version = 11

class gavo.user.upgrade.To13Upgrader

Bases: Upgrader

classmethod u_010_updateObscore(connection)

upgrade ivoa.obscore to obscore 1.1.

version = 12

class gavo.user.upgrade.To14Upgrader

Bases: Upgrader

classmethod s_010_addColIndex(connection)

Adding column_index column to TAP_SCHEMA.columns”

version = 13

class gavo.user.upgrade.To15Upgrader

Bases: Upgrader

classmethod s_10_add_uws_creationTime(connection)

adding creationTime to UWS tables for UWS 1.1 support

version = 14

class gavo.user.upgrade.To16Upgrader

Bases: Upgrader

classmethod s_10_upgrade_tap_schema_tables(connection)

Adding 1.1 columns to TAP_SCHEMA.tables.

classmethod s_20_upgrade_tap_schema_columns(connection)

Adding 1.1 columns to TAP_SCHEMA.columns.

u_010_delete_obscore_1_0_model = "DELETE FROM tap_schema.supportedmodels  WHERE dmivorn='ivo://ivoa.net/std/ObsCore/v1.0'"

classmethod u_50_update_tap_schema(connection)

Filling new TAP_SCHEMA columns

classmethod u_60_update_obscore_spectra(connection)

Updating obscore-published spectral tables

version = 15

class gavo.user.upgrade.To17Upgrader

Bases: Upgrader

classmethod u_10_import_adql(connection)

loading new ADQL UDFs

classmethod u_20_reload_tap_schema(connection)

updating tap_schema.schemas conflict rule

classmethod u_30_update_obscore_meta(connection)

updating obscore metadata

version = 16

class gavo.user.upgrade.To18Upgrader

Bases: Upgrader

classmethod u_10_restoreTAP_SCHEMA(connection)

recreating a possibly damaged TAP_SCHEMA

version = 17

class gavo.user.upgrade.To19Upgrader

Bases: Upgrader

classmethod s_10_add_schema_index(connection)

adding schema_index to tap_schema.schemas

classmethod u_20_add_tap_schema_keys(connection)

re-importing TAP_SCHEMA to update foreign key declarations

version = 18

class gavo.user.upgrade.To1Upgrader

Bases: Upgrader

classmethod u_000_update_funcs(connection)

update GAVO server-side functions

version = 0

class gavo.user.upgrade.To20Upgrader

Bases: Upgrader

classmethod u_10_update_obscore_meta(connection)

updating obscore metadata

classmethod u_20_import_adql(connection)

updating ADQL UDFs

version = 19

class gavo.user.upgrade.To21Upgrader

Bases: Upgrader

classmethod s_10_update_obscoresources(connection)

adding sourcerd column to obscore sources table

classmethod u_20_fill_obscoresources_sourcerd(connection)

filling sourcerd column in obscore sources

version = 20

class gavo.user.upgrade.To22Upgrader

Bases: Upgrader

classmethod s_10_add_nrows_to_tablemeta(connection)

adding nrows to dc.tablemeta

version = 21

class gavo.user.upgrade.To23Upgrader

Bases: Upgrader

classmethod u_10_add_and_fill_rds(connection)

creating and filling dc.rdmeta

version = 22

class gavo.user.upgrade.To24Upgrader

Bases: Upgrader

classmethod u_000_update_funcs(connection)

update GAVO server-side functions

version = 23

class gavo.user.upgrade.To25Upgrader

Bases: Upgrader

classmethod u_000_hash_passwords(connection)

scrypt-hashing existing passwords

version = 24

class gavo.user.upgrade.To26Upgrader

Bases: Upgrader

classmethod u_000_give_extra_float_digits(connection)

ensuring postgres doesn’t lose bits in floats

version = 25

class gavo.user.upgrade.To27Upgrader

Bases: Upgrader

classmethod s_010_create_col_stats_table(connection)

Creating the dc.simple_col_stats table

classmethod s_020_add_coverage_to_tablemeta(connection)

Adding a coverage column to rdmeta

classmethod u_30_update_obscore_meta(connection)

Re-import obscore metadata to update for UCD erratum

version = 26

class gavo.user.upgrade.To28Upgrader

Bases: Upgrader

classmethod u_10_update_TAP_SCHEMA_META(connection)

Declaring TAP_SCHEMA descriptions as unicode

version = 27

class gavo.user.upgrade.To29Upgrade

Bases: Upgrader

classmethod u_10_update_EPNTAP_meta(connection)

Updating EPNTAP2 metadata where necessary and possible

version = 28

class gavo.user.upgrade.To2Upgrader

Bases: Upgrader

classmethod u_000_siapDateObsToMJD(connection)

change SIAP and SSAP dateObs columns to MJD

version = 1

class gavo.user.upgrade.To30Upgrader

Bases: Upgrader

classmethod u_10_make_stats_index(connection)

Updating indices on dc.simple_col_stats

classmethod u_20_create_discrete_text_values(connection)

Creating dc.discrete_string_values

version = 29

class gavo.user.upgrade.To31Upgrader

Bases: Upgrader

classmethod u_10_update_ssa_fov_ucd(connection)

Updating SSA published metadata for SSA 1.1 erratum 2 (instr.fov)

version = 30

class gavo.user.upgrade.To32Upgrader

Bases: Upgrader

classmethod u_10_update_obscore_defs(connection)

updating obscore definitions to use reals for spectral

version = 31

class gavo.user.upgrade.To33Upgrader

Bases: Upgrader

classmethod s_010_add_nrows_in_tap_schema(connection)

adding nrows to tap_schema.tables

classmethod u_020_fill_tap_schema(connection)

updating tap_schema to include nrows

version = 32

class gavo.user.upgrade.To34Upgrader

Bases: Upgrader

classmethod u_000_update_funcs(connection)

update GAVO server-side functions

version = 33

class gavo.user.upgrade.To3Upgrader

Bases: Upgrader

classmethod u_000_tapSchema(connection)

add supportedmodels table to tap_schema

classmethod u_010_declareObscoreModel(connection)

declare obscore data model if the obscore table is present

version = 2

class gavo.user.upgrade.To4Upgrader

Bases: Upgrader

classmethod u_000_adqlfunctions(connection)

update ADQL GAVO-defined functions for the postgres planner’s benefit

version = 3

class gavo.user.upgrade.To5Upgrader

Bases: Upgrader

classmethod u_000_updateObscore(connection)

update obscore to work even when the table is empty

version = 4

class gavo.user.upgrade.To6Upgrader

Bases: Upgrader

classmethod s_010_addProductColumns(connection)

add preview and datalink columns to dc.products

classmethod u_020_remetaObscore(connection)

update obscore metadata to fix the erroneous id

version = 5

class gavo.user.upgrade.To7Upgrader

Bases: Upgrader

classmethod s_010_addPreviewMIMEColumn(connection)

add preview_mime column to dc.products

version = 6

class gavo.user.upgrade.To8Upgrader

Bases: Upgrader

classmethod s_010_removeColumnsMeta(connection)

version = 7

class gavo.user.upgrade.To9Upgrader

Bases: Upgrader

u_010_chuckADQLPrefix = "UPDATE TAP_SCHEMA.columns SET datatype=substring(datatype from 6) WHERE datatype LIKE 'adql:%%'"

u_020_setSize1OnAtoms = 'UPDATE tap_schema.columns SET "size"=1 WHERE NOT datatype LIKE \'%%(*)\''

u_030_removeArrayMarkInText = "UPDATE tap_schema.columns SET datatype=replace(datatype, '(*)', '') WHERE datatype LIKE '%%(*)'"

version = 8

class gavo.user.upgrade.Upgrader

Bases: object

A specification to upgrade from some schema version to another schema version.

Upgraders live as uninstanciated classes. Their version attribute gives the version their instructions update from; their destination version therefore is version+1.

An upgrader can have attributes s_<secno>_<something>, which are classmethods(connection) -> None performing schema changes. They always need to see if the manipulation already has been performed. Use relationExists and/or getColumnNamesFor as appropriate.

An upgrader can have attributes named u_<seqno>_<something>.

These can be

• strings, which are then directly executed in the database (but you should only do that for idempotent operations)

• classmethods, which will be executed in sequence (version, seqno) after all schemachange methods are executed.

The methods will be called with a connection argument. You must not commit this connection. You must not swallow exceptions that have left the connection unready (i.e., require a rollback).

Note that if you run rsc.makeData, you MUST pass both connection=connection and runCommit=False in order to avoid messing people’s lives up.

The updaters should have 1-line docstrings explaining what they do.

The update of the schemaversion is done automatically, you don’t need to worry about it.

Note that whenever you change the schema in a system RD, the corresponding upgrader needs to be a schemachange; otherwise, it’s very likely that any use of the system RD will fail, which usually makes an installation not upgradeable without manual intervention.

classmethod iterNormalActions()

returns strings and classmethods that run non-schemachanging actions.

classmethod iterSchemaChanges()

returns classmethods changing the schema as necessary.

These always run before the normal actions.

classmethod updateSchemaversion(connection)

version = None

gavo.user.upgrade.clearThirdPartyCache()

gavo.user.upgrade.dumpExtensionUpdater(conn)

prints SQL to bring [db]managedExtensions up to date.

gavo.user.upgrade.getColumnNamesFor(tableId, connection)

returns (normalised) column names for tableId.

(tableId is something like //tap#tables)

gavo.user.upgrade.getDBSchemaVersion()

returns the schemaversion given in the database.

This will return -1 if no schemaversion is declared.

gavo.user.upgrade.iterEPNTAPTables(connection)

yields table descriptors for all tables that claim to conform to EPN-TAP.

gavo.user.upgrade.iterStatements(startVersion, endVersion=34, upgraders=None)

yields all upgraders from startVersion to endVersion in sequence.

gavo.user.upgrade.main()

gavo.user.upgrade.parseCommandLine()

gavo.user.upgrade.relationExists(tableName, connection)

returns True if tableName (as schema.name) exists, False otherwise.

gavo.user.upgrade.showProgress(msg)

outputs msg to stdout without lf “immediately”.

gavo.user.upgrade.upgrade(forceDBVersion=None)

runs all updates necessary to bring a database to the CURRENT_SCHEMAVERSION.

Unless catastrophic things go on, each upgrade is a transaction of its own; the first failed transaction stops the upgrade at the version last successfully upgraded to.