gavo.rscdef.rmkdef module

Definition of rowmakers.

rowmakers are objects that take a dictionary of some kind and emit a row suitable for inclusion into a table.

class gavo.rscdef.rmkdef.ApplyDef(parent, **kwargs)

Bases: ProcApp

A code fragment to manipulate the result row (and possibly more).

Apply elements allow embedding python code in rowmakers.

The current input fields from the grammar (including the rowmaker’s vars) are available in the vars dictionary and can be changed there. You can also add new keys.

You can add new keys for shipping out in the result dictionary.

The active rowmaker is available as parent. It is also used to expand macros.

The table that the rowmaker feeds to can be accessed as targetTable. You probably only want to change meta information here (e.g., warnings or infos).

As always in procApps, you can get the embedding RD as rd; this is useful to, e.g., resolve references using rd.getByRD, and specify resdir-relative file names using rd.getAbsPath.

attrSeq = [<gavo.base.complexattrs.StructListAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.parsecontext.OriginalAttribute object>, <gavo.base.parsecontext.ReferenceAttribute object>, <gavo.base.complexattrs.StructListAttribute object>, <gavo.base.attrdef.EnumeratedUnicodeAttribute object>]

completedCallbacks = []

formalArgs = 'vars, result, targetTable, _self'

getFuncCode()

returns a function definition for this proc application.

This includes bindings of late parameters.

Locally defined code overrides code defined in a procDef.

managedAttrs = {'bind': <gavo.base.complexattrs.StructListAttribute object>, 'bindings': <gavo.base.complexattrs.StructListAttribute object>, 'code': <gavo.base.attrdef.UnicodeAttribute object>, 'deprecated': <gavo.base.attrdef.UnicodeAttribute object>, 'doc': <gavo.base.attrdef.UnicodeAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'name': <gavo.base.attrdef.UnicodeAttribute object>, 'original': <gavo.base.parsecontext.OriginalAttribute object>, 'procDef': <gavo.base.parsecontext.ReferenceAttribute object>, 'setup': <gavo.base.complexattrs.StructListAttribute object>, 'setups': <gavo.base.complexattrs.StructListAttribute object>, 'type': <gavo.base.attrdef.EnumeratedUnicodeAttribute object>}

name_ = 'apply'

requiredType = 'apply'

exception gavo.rscdef.rmkdef.Error(msg: str = '', hint: Optional[str] = None)

Bases: Error

class gavo.rscdef.rmkdef.MapRule(parent, **kwargs)

Bases: MappedExpression

A mapping rule.

To specify the source of a mapping, you can either

• grab a value from what’s emitted by the grammar or defined using var via

  the source attribute. The value given for source is converted to a python value and stored.

• or give a python expression in the body. In that case, no further type conversion will be attempted.

If neither source or a body is given, map uses the key attribute as its source attribute.

The map rule generates a key/value pair in the result record.

attrSeq = [<gavo.rscdef.column.ColumnNameAttribute object>, <gavo.base.structure.DataContent object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>]

completedCallbacks = []

destDict = 'result'

getAutoMapper(columns)

returns an expression to automatically map self.source to a column in the destination table.

managedAttrs = {'content_': <gavo.base.structure.DataContent object>, 'dest': <gavo.rscdef.column.ColumnNameAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'key': <gavo.rscdef.column.ColumnNameAttribute object>, 'name': <gavo.rscdef.column.ColumnNameAttribute object>, 'nullExcs': <gavo.base.attrdef.UnicodeAttribute object>, 'nullExpr': <gavo.base.attrdef.UnicodeAttribute object>, 'source': <gavo.base.attrdef.UnicodeAttribute object>, 'src': <gavo.base.attrdef.UnicodeAttribute object>}

name_ = 'map'

class gavo.rscdef.rmkdef.MappedExpression(parent, **kwargs)

Bases: Structure

a base class for map and var.

You must give a destDict class attribute to make these work.

attrSeq = [<gavo.rscdef.column.ColumnNameAttribute object>, <gavo.base.structure.DataContent object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>]

completeElement(ctx)

completedCallbacks = []

destDict = None

getAutoMapper()

returns an expression literal for turning what is self.source to a value suitable for self.key.

This must be defined for concrete classes derived from this.

getCode(columns)

returns python source code for this map.

managedAttrs = {'content_': <gavo.base.structure.DataContent object>, 'dest': <gavo.rscdef.column.ColumnNameAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'key': <gavo.rscdef.column.ColumnNameAttribute object>, 'name': <gavo.rscdef.column.ColumnNameAttribute object>, 'nullExcs': <gavo.base.attrdef.UnicodeAttribute object>, 'nullExpr': <gavo.base.attrdef.UnicodeAttribute object>, 'source': <gavo.base.attrdef.UnicodeAttribute object>, 'src': <gavo.base.attrdef.UnicodeAttribute object>}

restrictedMode = False

validate()

checks that code content is a parseable python expression and that the destination exists in the tableDef

class gavo.rscdef.rmkdef.ParmakerDef(parent, **kwargs)

Bases: RowmakerDef

attrSeq = [<gavo.base.complexattrs.StructListAttribute object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.attrdef.StringListAttribute object>, <gavo.base.complexattrs.StructAttribute object>, <gavo.base.complexattrs.StructListAttribute object>, <gavo.base.parsecontext.OriginalAttribute object>, <gavo.rscdef.common.RDAttribute object>, <gavo.base.attrdef.IdMapAttribute object>, <gavo.base.complexattrs.StructListAttribute object>]

completedCallbacks = []

getFullId()

managedAttrs = {'apply': <gavo.base.complexattrs.StructListAttribute object>, 'apps': <gavo.base.complexattrs.StructListAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'idmaps': <gavo.base.attrdef.StringListAttribute object>, 'ignoreOn': <gavo.base.complexattrs.StructAttribute object>, 'map': <gavo.base.complexattrs.StructListAttribute object>, 'maps': <gavo.base.complexattrs.StructListAttribute object>, 'original': <gavo.base.parsecontext.OriginalAttribute object>, 'rd': <gavo.rscdef.common.RDAttribute object>, 'simplemaps': <gavo.base.attrdef.IdMapAttribute object>, 'var': <gavo.base.complexattrs.StructListAttribute object>, 'vars': <gavo.base.complexattrs.StructListAttribute object>}

name_ = 'parmaker'

property rd

class gavo.rscdef.rmkdef.Rowmaker(source, name, globals, defaults, lineMap)

Bases: object

A callable that arranges for the mapping of key/value pairs to other key/value pairs.

Within DaCHS, Rowmakers generate database rows (and parameter dictionaries) from the output of grammars.

They are constructed with the source of the mapping function, a dictionary of globals the function should see, a dictionary of defaults, giving keys to be inserted into the incoming rowdict before the mapping function is called, and a map of line numbers to names handled in that line.

It is called with a dictionary of locals for the functions (i.e., usually the result of a grammar iterRows).

class gavo.rscdef.rmkdef.RowmakerDef(parent, **kwargs)

Bases: Structure, RowmakerMacroMixin

A definition of the mapping between grammar input and finished rows ready for shipout.

Rowmakers consist of variables, procedures and mappings. They result in a python callable doing the mapping. In python code within rowmaker elements, you can use a large number of functions. See `Functions available for row makers`_ in the reference documentation.

RowmakerDefs double as macro packages for the expansion of various macros. The standard macros will need to be quoted, the rowmaker macros above yield python expressions.

Within map and var bodies as well as late apply pars and apply bodies, you can refer to the grammar input as vars[“name”] or, shorter @name.

To add output keys, use map or, in apply bodies, add keys to the result dictionary.

attrSeq = [<gavo.base.complexattrs.StructListAttribute object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.attrdef.StringListAttribute object>, <gavo.base.complexattrs.StructAttribute object>, <gavo.base.complexattrs.StructListAttribute object>, <gavo.base.parsecontext.OriginalAttribute object>, <gavo.rscdef.common.RDAttribute object>, <gavo.base.attrdef.IdMapAttribute object>, <gavo.base.complexattrs.StructListAttribute object>]

compileForTableDef(tableDef)

returns a function receiving a dictionary of raw values and returning a row ready for adding to a tableDef’d table.

To do this, we first make a rowmaker instance with idmaps resolved and then check if the rowmaker result and the table structure are compatible.

completeElement(ctx)

completedCallbacks = []

copyShallowly()

getFullId()

classmethod makeIdentityFromTable(table, **kwargs)

returns a rowmaker that just maps input names to column names.

classmethod makeTransparentFromTable(table, **kwargs)

returns a rowmaker that maps input names to column names without touching them.

This is for crazy cases in which the source actually provides pre-parsed data that any treatment would actually ruin.

managedAttrs = {'apply': <gavo.base.complexattrs.StructListAttribute object>, 'apps': <gavo.base.complexattrs.StructListAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'idmaps': <gavo.base.attrdef.StringListAttribute object>, 'ignoreOn': <gavo.base.complexattrs.StructAttribute object>, 'map': <gavo.base.complexattrs.StructListAttribute object>, 'maps': <gavo.base.complexattrs.StructListAttribute object>, 'original': <gavo.base.parsecontext.OriginalAttribute object>, 'rd': <gavo.rscdef.common.RDAttribute object>, 'simplemaps': <gavo.base.attrdef.IdMapAttribute object>, 'var': <gavo.base.complexattrs.StructListAttribute object>, 'vars': <gavo.base.complexattrs.StructListAttribute object>}

name_ = 'rowmaker'

property rd

class gavo.rscdef.rmkdef.RowmakerMacroMixin

Bases: StandardMacroMixin

A collection of macros available to rowmakers.

NOTE: All macros should return only one single physical python line, or they will mess up the calculation of what constructs caused errors.

macro_dlMetaURI(dlId)

returns a link to the datalink document for the current product.

This assumes you’re assigning standard pubDIDs (see also standardPubDID, which is used by this).

dlId is the XML id of the datalink service, which is supposed to be in the sameRD as the rowmaker.

macro_docField(name)

returns an expression giving the value of the column name in the document row.

macro_fullPath()

returns an expression expanding to the full path of the current input file.

macro_inputRelativePath(liberalChars='True')

see grammars.common.GrammarMacroMixin

macro_inputSize()

returns an expression giving the size of the current source.

macro_lastSourceElements(numElements)

returns an expression calling rmkfuncs.lastSourceElements on the current input path.

macro_property(propName)

returns an expression giving the value of the property propName on the current DD.

macro_qName()

returns the qName of the table we are currently parsing into.

macro_rootlessPath()

returns an expression giving the current source’s path with the resource descriptor’s root removed.

macro_rowsMade()

returns an expression giving the number of records already returned by this row maker.

This number excludes failed and skipped rows.

macro_rowsProcessed()

returns an expression giving the number of records already delivered by the grammar.

macro_sourceCDate()

returns an expression giving the timestamp for the create date of the current source.

Use dateTimeToJdn or dateTimeToMJD to turn this into JD or MJD (which is usually preferred in database tables). See also the sourceDate macro.

macro_sourceDate()

returns an expression giving the timestamp of the current source.

This is a timestamp of the modification date; use dateTimeToJdn or dateTimeToMJD to turn this into JD or MJD (which is usually preferred in database tables). See also the sourceCDate macro.

macro_srcstem()

returns python code for the stem of the source file currently parsed in a rowmaker.

Example: if you’re currently parsing /tmp/foo.bar.gz, the stem is foo.

macro_standardPubDID()

returns the “standard publisher DID” for the current product.

The publisher dataset identifier (PubDID) is important in protocols like SSAP and obscore. If you use this macro, the PubDID will be your authority, the path component ~, and the current value of @prodtblAccref. It thus will only work where products#define (or a replacement) is in action. If it isn’t, a normal function call getStandardPubDID(\inputRelativePath) would be an obvious alternative.

You can of course define your PubDIDs in a different way.

class gavo.rscdef.rmkdef.VarDef(parent, **kwargs)

Bases: MappedExpression

A definition of a rowmaker variable.

It consists of a name and a python expression, including function calls. The variables are entered into the input row coming from the grammar.

var elements are evaluated before apply elements, in the sequence they are in the RD. You can refer to keys defined by vars already evaluated in the usual @key manner.

attrSeq = [<gavo.rscdef.column.ColumnNameAttribute object>, <gavo.base.structure.DataContent object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>]

completedCallbacks = []

destDict = 'vars'

getAutoMapper(columns)

returns var[self.source].

Having source with var doesn’t make a lot of sense, but it’s a nifty way to introduce None-s for missing keys if one wants. And it should do something.

managedAttrs = {'content_': <gavo.base.structure.DataContent object>, 'dest': <gavo.rscdef.column.ColumnNameAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'key': <gavo.rscdef.column.ColumnNameAttribute object>, 'name': <gavo.rscdef.column.ColumnNameAttribute object>, 'nullExcs': <gavo.base.attrdef.UnicodeAttribute object>, 'nullExpr': <gavo.base.attrdef.UnicodeAttribute object>, 'source': <gavo.base.attrdef.UnicodeAttribute object>, 'src': <gavo.base.attrdef.UnicodeAttribute object>}

name_ = 'var'