Global Metadata

The root element of an RD always is resource, and you should immediately define the database schema anything you import will end up in:

<resource schema="arihip">

RDs are normal XML files (meaning that you could, e.g., add an XML declaration if you want an encoding other than utf-8).

DaCHS assumes the schema name to be identical to the resource directory, which again is a child of the inputs directory. If you have a strong reason to deviate from that convention, you must give the subdirectory name in the resource element's resdir attribute – either way, this is then used to build absolute paths within the RD, e.g., for the sources element discussed below.

In general, you should have exactly one RD per database schema. This is not enforced, but sharing schema names between RDs will cause some undesirable behaviour. An example is permissions: When importing a table, the schema access rights are adapted. If you have one RD A defining an ADQL-queriable table in schema X and another RD B that has no ADQL-queriable table, importing A will make schema X readable to untrusted queries, whereas importing B will make it unreadable again; this would lead to query failures.

<resource>
  <schema>arihip</schema>

The RD goes on to give metadata applying to everything within the RD:

<meta name="title">ARIHIP astrometric catalogue</meta>
<meta name="creationDate">2010-11-03T10:13:00</meta>
<meta name="description">
  The catalogue ARIHIP has been constructed by
  selecting the 'best data' for a given star from combinations of HIPPARCOS
  data with Boss' GC and/or the Tycho-2 catalogue as well as the FK6.  It
  provides 'best data' for 90 842 stars with a typical mean error of
  0.89 mas/year (about a factor of 1.3 better than Hipparcos for this
  sample of stars).
</meta>
<meta name="creator">Wielen, R.; Schwan, H.; Dettbarn, C.; et al</meta>

<meta name="subject">Catalogs</meta>
<meta name="subject">Astrometry</meta>
<meta name="subject">Stars: Proper Motions</meta>
<meta name="type">Catalog</meta>

<meta name="coverage.waveband">Optical</meta>

<coverage>
  <spatial>0/0-11</spatial>
  <spectral>2.721e-19 4.138e-19</spectral>
  <temporal>1989-09-01 1993-08-15</temporal>
</coverage>

<FEED source="//procs#license-cc-by" what="ARIHIP"/>

<meta name="_longdoc" format="rst">
  The ARIHIP Catalogue is a suitable combination of the results of the
  HIPPARCOS astrometry satellite with ground-based data.

  [...]
</meta>

<meta name="source">2001VeARI..40....1W</meta>

<meta name="_intro" format="rst"> <![CDATA[
  For advanced queries on this catalogue use ADQL_
  possibly via TAP_

  .. _ADQL: /adql
  .. _TAP: /tap
]]> </meta>

This metadata is crucial for later registration of the service, some of it turns up in service responses, and quite a bit is used in making web pages, in the metadata sidebars and elsewhere.

Metadata elements have a name attribute that gives the “kind” of metadata contained, and sometimes also determine a specific type. The names should usually give a good indication of what information is given. We will later revisit individual items and what should and should not be put there.

If there is more than one meta element for a name (like for subject in the example), the key is multi-valued. For quite a few meta keys, DaCHS will at some point raise errors you if you have multiple values where you can't have them, as for instance, with titles, descriptions, or creation dates. Because DaCHS itself tries to treat all metadata as equally as possible, the failures will not happen during metadata definition but when generating things like registry records, which sometimes makes it a bit hard to figure out the error's origin actually lies in the metadata definition. The dachs val subcommand should catch such problems, though (and if it doesn't, please report that as a bug).

Metadata can come in various formats as determined by the format attribute. If you give nothing there (which makes it plain), DaCHS will apply some whitespace normalisation, and it will interpret empty lines as paragraphs if the target format supports it.

With format="rst" (as used in _longdoc and _intro in the example), the content will be interpreted as reStructuredText, which lets you do rather fancy things when the metadata is being rendered into HTML. Be careful to use consistent indendation in this case. There are also raw and literal as formats; don't worry about them now.

Metadata with names starting with underscores – like _intro in the example – is for DaCHS-internal use, very typically when building informational pages; where there's no underscore, there is generally some external standard, and in all likelihood the information will end up in the VO Registry in some way. Many of those we discuss in data_checklist.html. This checklist was actually written to hand out to people wanting to publish in the Heidelberg deata center, but you are welcome to re-use it.

In the above RD snippet, there are two elements that are not simple meta elements. One is coverage; this gives the extent of the data contained in the service in space, time, and spectrum. You will usually only define this manually if you know things the computer can't easily find out itself. This is typically “data consists of several well-defined campaigns” (so you have clear time intervals) or “data is taken in a few of rather small bands” (so you have clear spectral intervals).

The other non-meta element is FEED. In DaCHS uppercase elements signify active tags, elements that are replaced by something else (or nothing) in the document tree. FEED, as used here, simply inserts a stream (essentially, a sequence of elements) defined elsewhere and puts it in. The reference manual has list of predefined streams with an indication of what is inserted. Here, this is used to insert a few more formalised meta items declaring the licence under which your data can be used and distributed. Please be explicit on your licences, see Licensing below for why we ask that (and why not).

The last major concept you should now in connection with DaCHS metadata is metadata inheritance. For instance, if you define a service within the RD, and there is no title meta within the service, DaCHS will use the title meta on the RD. The root of the metadata inheritance hierarchy is the file etc/defaultmeta.txt. If you defined a meta item title there, this would be the default title for everything in your data center (which is something you probably don't want). This default metadata, however, is handy for items that plausibly are essentially constant across a data center, like a publisher or a contact. By the inheritance rules you can still override it in individual RDs or services if necessary.

Defining Tables

A major part of the metadata DaCHS deals with is the table structure. It is defined in table elements, which usually are direct children of the resoource element. A resource element may contain multiple table definitions.

Skip over the macDef elements for now to where the table element begins. What you see is something like:

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
  primary="hipno">

The id attribute of the table doubles as the name of the database table; make sure you use something that works as a valid simple SQL identifier (i.e., [A-Za-z_][A-Za-z0-9_]*) – DaCHS does not support for delimited identifiers as table names (but over standard SQL it will let you have leading underscores).

Be sure to always specify onDisk="True" unless you're going for special effects – without it, the table will end up only in memory and be gone after the import. The adql attribute says that TAP queries should be allowed on the table; leave it out for tables not suitable for “raw” consumption by your clients.

For the mixin attribute is one of the deeper concepts in DaCHS. We defer a closer look at that to the Mixin Introduction.

Finally, using the primary attribute you can specify an explicit primary key of the table (if it is made up of several columns, concatenate their names with commas). This is made into a primary key for postgres straightforwardly, which means that the database makes sure there are no two rows with the same value for the primary key. Also, the database creates an index for efficient queries using the primary key.

While this table doesn't have metadata (because this is a one-table resource, and the global resource metadata is the one of the table itself), table elements may override RD metadata, and in multi-table RDs it's usually a good idea to override title and description in the table element itself.

What follows is a definition of the structure of the space-time coordinates:

<stc>
  Position ICRS Epoch J2000.0 "raj2000" "dej2000" Error "err_ra" "err_de"
    Velocity "pmra" "pmde" Error "err_pmra" "err_pmde"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raHIP" "deHIP" Error "err_raHIP" "err_deHIP"
    Velocity "pmraHIP" "pmdeHIP" Error "err_pmraHIP" "err_pmdeHIP"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raSTP" "deSTP" Error "err_raSTP" "err_deSTP"
    Velocity "pmraSTP" "pmdeSTP" Error "err_pmraSTP" "err_pmdeSTP"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raLTP" "deLTP" Error "err_raLTP" "err_deLTP"
    Velocity "pmraLTP" "pmdeLTP" Error "err_pmraLTP" "err_pmdeLTP"
</stc>

What this says is that there's a number of coordinate structures in the table, grouping together positions, errors, and proper motions in a specific reference frame. See Space-Time Metadata for more information.

Column definitions

The RD continues with the definitions of the columns that make up the table:

<column name="hipno" type="integer" ucd="meta.id;meta.main"
  tablehead="HIP id" verbLevel="1"
  description="Number of the star in the HIPPARCOS Catalogue (ESA 1997)."
  required="True"/>
<column name="srcSel" type="text" ucd="meta.flag"
  tablehead="Source" verbLevel="25"
  description="Source of the astrometric solution"
  note="src"/>

For every column in a table, there is one column element with a host of attributes. The name attribute is central in that it will be the column name in the database, the key for the column's value in record dictionaries that the software uses internally, and it is usually used to reference the column from the outside (see Referencing in DaCHS).

Column names must be legal identifiers for both python and SQL in DaCHS. SQL delimited identifiers thus are not allowed (this is not the whole truth, but it's true enough, and you're saving yourself a lot of headache if you simply believe it).

The type attribute defaults to real, and can otherwise take values in valid SQL datatypes. Among the more common types are:

• text – a string. You could also use types with explicit length like char(7), but this is discouraged. It does not help postgres (or much anything else within the DC). The only advantage would be if you later want to handle FITS Binary serialisations, because when all strings and arrays in a record have fixed lengths, libraries can directly seek to records. Also note that text-valued columns can only keep ASCII. Use the unicode type if you need non-ASCII.
• real – a floating point number, pretty much like float in C.
• double precision – a floating point number. You should use such doubles if you need to keep more than about 7 digits of mantissa.
• integer – typically a 32-bit integer
• bigint – typically a 64-bit integer
• smallint – typically a 16-bit integer
• timestamp – a combination of date and time. While postgres can process a very large range of dates, the DC stores timestamps in datetime.datetime objects, which means that for “astronomical” times (like 10000 B.C. or 10000 A.D. you may need to use custom representations. Also, the DC assumes all times to be without time zones. Further time metadata (like distinguishing TT from UT) is given through STC specifications. See Some Words on Times below for why you probably do not want timestamps.
• date – a date. See timestamp.
• time – a time. See timestamp
• spoint, scircle, sbox, spoly – objects of spherical geometry, taken from pgSphere. See Creating pgSphere Geometries below on how to ingest them.

Some more types (like raw and file) are available to tables in service definitions, but they should, in general, not appear in database tables.

Futher metadata on columns includes:

• unit – the unit the column values are in. The syntax is that of VOUnits. Unit is left out for unitless values. Because it's hard to rigorously define, DaCHS does not distinguish between dimensionless (as, for instance, width/height, which in principle has a unit that just happens to come out as 1), things that can't have units to begin with (e.g., an observer name), and things that should have units but are written in a way that they can't (e.g., sexagesimal coordinates or ISO datetime strings).

• tablehead – a very short string designating the content. This string is typically used for display purposes, e.g., as table headings or labels on input fields and defaults to the capitalized column name.

• description – a longer string characterizing the content. This may end up in help bubbles or VOTable descriptions. Since these could be longer, you may want to put them in a child element rather than an attribute. In both cases, whitespace is normalized, so you can enter line breaks and similar for readability in the source, and they will always be rendered as a single blank. For even longer, note-like material, see Table Notes. An example for a descripton:

  <column name="aperture">
    <description>The aperture is the full-width-half-max of the
      response function of our sage 3000 hyper-detector.</description>
  </column>
  

• ucd – a Unified Content Descriptor as defined by IVOA. To figure out “good” UCDs, the GAVO UCD resolver can help. An easy way to come up with them is also to leave them out initially and then run dachs admin suggestucds (it's what we do these days).

• required – True if value must be set in order for the record to be valid. By default, NULL (which in python is None) is a valid value for any column and will silently be inserted if you don't assign a value in the rowmaker (see below). For required columns, an error will be raised when a value is missing. In HTTP service interfaces, missing required parameters will lead to a 400 invalid parameters HTTP response.

• verbLevel – A measure for the “importance” of the column. Various protocols have the notion of “verbosity”, where higher verbosity means you get to see more columns with more esoteric content. Within DaCHS, verbLevel is a number between (usefully) 1 and 30, with columns with verbLevel 1 always given and those with verbLevel 30 only given if someone really wants to see all columns. Technically, in SCS, a column is part of the output table if its verbLevel is smaller or equal to ten times the query's VERB parameter.

Column elements may have a child element values. This lets you specify metadata like maximum or minimum, or enumerate possible values. The most common use is the definition of null literals, though. This is not necessary for floats, and usually not even strings, because these have useful (and actually non-overridable) null values in the VOTable representation (where this sort of thing counts most). It is, however, highly recommended to give null literals when defining integral types (including chars) in columns that may have NULLs. DaCHS will try to pick useful null values for those automatically when possible, but when streaming tables, this is impossible, and errors will be raised during VOTable rendering when NULLs are encountered in such a situation.

So, just define null values whenever you define a non-required integral column, like this:

<column name="n_obs" type="integer"
    description="Number of
    observations, NULL if interpolated data">
  <values nullLiteral="-1"/>
</column>

The output of dachs info can help you to choose suitable NULL values. To help people spot them when metadata is missing, it's usually wise to choose “conspicuous” null values (like -1, 9999, or similar).

A child element of table we have not mentioned above is this:

<index columns="mv"/>

This is the simplest, but mostly sufficient, form of telling DaCHS to index a column, where the columns attribute just lists the names of the columns to index – and most of the time, that will be just one. More complicated index patterns are possible – see element index in the reference.

Indices are usually generated after a full import. If you add or change indices later, run dachs imp -I, which will re-make all indices. For large tables, you may want to just create one particular index; in that case, see the indexStatements subcommand to dachs admin.

Scrolling a bit further down in the arihip RD, you will notice some LOOP constructs. These are discussed below under active tags.

After the column definitions, there are meta elements called “note”; for those, see Table Notes.

Parsing Input Data

Going further down in the RD, you will find a data element. Their main purpose is to describe how certain input files fill the table(s) defined above.

It starts like this:

<data id="import">
  <sources>data/data.txt.gz</sources>

Data elements must have ids which can be used to individually reference them from a dachs imp command line; this is useful if you just want to import one part of a multi-table data collection. The default of dachs imp default is to build all data elements except those having an auto="False" attribute.

It is recommended that the id is a short verb phrase, as data basically contains instructions for an action. You might rightly argue that we have not chosen the element name too aptly (recipe might have been more appropriate), but we feel it's too late to change it now.

<sources>
  <pattern>inp2/*.txt</pattern>
  <pattern>inp1/*.txt</pattern>
</sources>

<columnGrammar topIgnoredLines="9" preFilter="zcat">
  <colDefs>
    hipno:     3-8
    srcSel:    47-49
    alphaHMS:  59-73
    [...]
  </colDefs>

dachs imp -M 100 --dumpRows q.rd | less

<make table="main">
  <rowmaker idmaps="*">
    <var name="raj2000">hmsToDeg(@alphaHMS, None)</var>
    <var name="dej2000">dmsToDeg(@deltaDMS, None)</var>
    ...
    <map dest="kbin">parseWithNull(@kbin, str, "9")</map>
    <map dest="vrad">parseWithNull(
      @vrad, lambda a:float(killBlanks(a)), "")</map>

<rowmaker>
  <map dest="evi" src="evi"/>
  <map dest="av" src="av"/>
  <map dest="ai" src="ai"/>
</rowmaker>

<rowmaker simplemaps="evi:evi,av:av,ai:ai"/>

<rowmaker idmaps="evi,av,ai"/>

<rowmaker idmaps="*"/>

<map dest="dateIngested">datetime.datetime.now()</map>

<map dest="Vmag">parseWithNull(@VmagSrc, float, "99.99")</map>

<map dest="Vmag" nullExcs="TypeError"
  >parseWithNull(@VmagSrc, float, "99999")/1000.</map>

<map dest="magic" nullExcs="ValueError,TypeError"
  >@magicOffset+float(@magicLit)</map>

Service Definitions

The last but one part of the RD deals with how to get the data out of the database again, i.e., the services exposing the data. This part is fairly simple for arihip:

<service id="cone" allowed="scs.xml,form">
  <meta name="shortName">arihip cone</meta>
  <meta name="testQuery">
    <meta name="ra">9.4076</meta>
    <meta name="dec">9.6414</meta>
    <meta name="sr">1.0</meta>
  </meta>

  <dbCore queriedTable="main">
    <FEED source="//scs#coreDescs"/>
    <condDesc buildFrom="mv"/>
    <condDesc>
      <inputKey original="hipno" required="False"/>
    </condDesc>
  </dbCore>

  <publish render="scs.xml" sets="ivo_managed"/>
  <publish render="form" sets="ivo_managed,local"/>
  <outputTable verbLevel="20"/>
</service>

To understand what's going on here, some basic understanding of DaCHS' service architecture is required; it consists of:

• cores; they do the actual computation or database query.
• renderers; these digest the bits coming in from the client and (in general) format the result in some way requested by the client again. There are renderers for web forms and several VO protocols, ones producing HTML documentation for RDs, and so on. In the ideal case – as in the example –, you can use the same core for both a VO protocol and a form-based service by just allowing different renderers.
• services; they hold together the core and the renderer, can reformat core results, and they are what is registered, which means that they hold the metadata that ends up in the registry (most of it is typically inherited from their RDs).

The renderers are referenced by name in the service's allowed attribute. What can be given there (concatenated by commas) is listed in the reference documentation's section on Renderers Available.

Which renderer is used when a service is run is selected by the client: it's (usually) the last segment of the path part of the access URL. If a client tries to retrieve a URL with a renderer that is not in the service's allowed list, DaCHS will respond with a 403 forbidden HTTP code (excepting certain “unchecked” renderers like info that typically expose service metadata). Note that some of the more exotic renderers may pose special requirements on cores. Don't be too disappointed of a given renderer won't work with your core.

The most common core for catalogue services (and the one you'll typically use for SCS services) is the Element scsCore, as used here; you might also use a generic Element dbCore. Other cores you should know about include the Element nullCore (for services that don't do any server-side computation at all), Element datalinkCore (which can define all sorts of manipulations on and linking of datasets), the python and custom cores (cf. writing-custom-cores), and several protocol-specific cores discussed below.

The dbCore generates a (single-table) query from condition descriptors and returns a table that you describe through an output table. Cores are defined as direct children of the resource (as with grammars and rowmakers, you can also have them in resource and then write core="id-of-element", which makes sense when a single core is shared by several services).

dbCores need a queriedTable attribute, the value of which must be the id of a table element. This is the table the query will run against.

The condition descriptors within dbCores (defined in Element condDesc) define, in essence, input fields: in the form renderer, these will be rendered as form items people can fill in. Most commonly, you will either define them using the original attribute (when inheriting from predefined condDescs) or using buildFrom. The first case is typically used in connection with protocols and on tables having mixins; such condDescs result in zero or more input fields, and they typically inspect the queried table. For example, the //scs#humanScs condDesc (which is part of the //scs#coreDescs stream) locates the “main” positions as identified by UCDs and generates queries against them using two input fields. One somehow specifies center position of the region of interest, the other gives the search radius.

When you define a condDesc using buildFrom, the result is usually one or more input field(s) constraining values in the column named in the buildFrom attribute. The software tries to make some useful input definition from that column, depending on the current renderer's parameter style (which is given in renderers available).

For instance, if you build a condDesc from a real- or double precision-typed column, and the service is accessed with a form-styled renderer, users can constrain the column's values with Vizier-like float expressions, where, for instance, an interval would be written as range_min .. range_max. When the request comes in through renderers with a parameter style of pql (e.g., SSAP), the condDesc would expect SSAP-style intervals (e.g., range_min/range_max). In the parameter style dali, that interval would be written range_min range_max. Yes, it is deplorable that something like the parameter style exists, but that particular uglyness is not DaCHS' fault; in practice, it's a lot less horrifying than it sounds, because clients very typically just use one renderer, and they hopefully agree with it on the parameter syntax.

The service element must have an id attribute that is used to select the service run in the access URL's last but one segment. Furthermore, there should be certain pieces of metadata useful for the Registry. First, there's shortName, which is typically used by clients in space-restricted displays. It must not be longer than 16 characters, so something like an acronym and a very terse role identifier is the best you can do (hence the “arihip cone” here). Frequently, a title meta is also useful, in particular when an RD contains multiple services.

Many standard VO protocols require additional, protocol-specific metadata. In the case of arihip, we have a Simple Cone Search service, which, as laid down in the reference for the the scs.xml renderer, requires the parameters of a test query returning a non-empty result.

Regression Tests

While it's admittedly boring, writing regression tests while developing your RD will save you a lot of fear, uncertainty, and doubt later. In the arihip RD, there are two regression tests:

<regSuite title="ARIHIP regression">
  <regTest title="ARIHIP form service appears to work.">
    <url parSet="form" hscs_sr="1.0" hscs_pos="0.00 1.08"
      >cone/form</url>
    <code>
      self.assertHasStrings("PM RA (STP)",
        "1.0890086361", "-5.00", "Note bin")
    </code>
  </regTest>

  <regTest title="ARIHIP SCS response looks reasonable">
    <url RA="0.0273384667" DEC="25.8864593500" SR="0.001"
      VERB="3">cone/scs.xml</url>
    <code>
      row = self.getFirstVOTableRow()
      self.assertEqual(row["flags"], 's 00 .. .... ... ...')
      self.assertAlmostEqual(row["err_parallaxSI"], 5.25e-07)
      self.assertEqual(row["hipno"], '8')
    </code>
  </regTest>
</regSuite>

As you can see, the tests are grouped into suites, where for most RDs you will just have one suite.

Each test has a title, which is used in diagnostics. The body of a test is a pair of an element url – which, really, is just a convenient way to write a URL with potentially complex parameters – and of an element code, which mostly gives a bunch of assertions about what a running server should be returning for the query encoded by url.

We have tried to write the reference documentation's section on regression testing in a tutorial-like fashion and therefore point you there for further details.

SCS Tables

SCS can expose any table that has exactly one column each with the UCDs pos.eq.ra;meta.main, pos.eq.dec;meta.main, and meta.id;meta.main, where the coordinates must be real or double precision, and the id must be either some integral type or text; the standard requires the id to be text, but the renderer will automatically convert integral types. The main query is then ran against the position specified in this way.

You almost always want to have a spatial index on these columns. To do that, use the //scs#pgs-pos-index mixin on the tables, like this:

<table id="forSCS" onDisk="true" mixin="//scs#pgs-pos-index"> ...

The “pgs” in pgs-pos-index refers to pgSphere, a postgres database extension for spherical geometry. You may see RDs around that use the //scs#q3cindex mixin instead here. It does the same thing (dramatically speed up spatial queries) but uses a different scheme. It's faster and takes up less space, but it's also less general, which is why we are trying to phase it out. Only use it when you are sure you cannot afford the (reasonable, i.e., mostly within a factor of two) cost of pgSphere.

Note that to have a valid SCS service, you must make sure the output table always contains the three required columns (as defined by the UCDs) discussed above. To ensure that, these columns' verbLevel attribute must be 10 or less (we advise to have it at 1).

SCS Cores

SCS could work with a dbCore, but friendly cone search services include a field with the distance between the object found and the position passed in; this is added by the special element scsCore.

You (in effect) must include the some pre-defined condDescs that make up the SCS protocol, like this:

<scsCore queriedTable="main">
  <FEED source="//scs#coreDescs"/>
</scsCore>

This will provide the RA, DEC, and SR parameters for most renderers. The form renderer, however will show a nice input box that lets humans enter object names or, if they cannot live without them, sexagesimal positions (if you are curious: this works by setting the onlyForRenderer and notForRenderer attributes on Element inputKey).

In addition, //scs#coreDescs gives you a paramter MAXREC to limit or raise the number of matches returned. This parameter is not required by SCS, but it is useful if people with sufficient technical skills (they'll need those because common SCS clients don't support MAXREC yet) want to raise or lower DaCHS' default match limit (which is configured in [ivoa]dalDefaultLimit and can be raised up to [ivoa]dalHardLimit).

SCS allows more query parameters; you can usually use condDesc's buildFrom attribute to directly make one from an input column. If you want to add a larger number of them, you might want to use active tags:

<dbCore id="xlcore" queriedTable="main">
  <FEED source="//scs#coreDescs"/>
  <LOOP listItems="ipix bmag rmag jmag pmra pmde">
    <events>
      <condDesc buildFrom="\item"/>
    </events>
  </LOOP>
</dbCore>

Note that most current SCS clients are not good at discovering such additional parameters, since for SCS this requires going through the Registry. In TOPCAT, for example, users would have to manually edit the cone search URL.

Also note that SCS does not really define the syntax of these parameters, which is relevant because most of the time they will be float-valued, and hence you will generally need to use intervals as constraints. The interval syntax used by the SCS renderer is DALI, so a bounded interval would be 22.3 30e5, and you'd build half-bounded intervals with IEEE infinity literals, like -Inf -1. Of course, when accessed through a form, the usual VizieR parameter syntax applies.

SCS Services

To expose that core through a service, just allow the scs.xml renderer on it. With the extra human-oriented positional constraint and mainly builtFrom condDescs, you can usually have a web-based form interface for free:

<service id="cone" allowed="scs.xml,form">
  <meta name="title">Nice Catalogue Cone Search</meta>
  <meta name="shortName">NC Cone</meta>
  <meta name="testQuery.ra">10</meta>
  <meta name="testQuery.dec">10</meta>
  <meta name="testQuery.sr">0.01</meta>
  <scsCore queriedTable="main">
    <FEED source="//scs#coreDescs"/>
    <LOOP listItems="ipix bmag rmag jmag pmra pmde">
      <events>
        <condDesc buildFrom="\item"/>
      </events>
    </LOOP>
  </scsCore>
</service>

The meta information given is used when generating registry records; the idea is that a query with the ra, dec, and sr you give actually returns some data.

Quick Sample Image Service

Check out a sample resource directory:

cd `dachs config inputsDir`
svn co http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/emi
cd emi
mkdir data

Now fetch some files to populate the data directory so you have something to import:

cd data
SRCURL=http://dc.g-vo.org/emi/q/s/siap.xml
curl -s $SRCURL"?POS=163.3,57.8&SIZE=20,20&MAXREC=5&weighting=uniform&RESPONSEFORMAT=votabletd" \
  | tr '<TD>' '\n' \
  | grep "^http://" \
  | sed -e 's/<[^>]*>//g' \
  | xargs -n1 curl -sO

(no, this is not in general the way to operate SIAP services; use a proper client for real work, and we didn't show you this).

This RD also publishes to obscore, so make sure you have the obscore table:

dachs imp //obscore

If you do not plan to publish via obscore yourself (which is reasonably unlikely) and you try this on a box that the Registry will see later (you shouldn't), be sure to dachs drop obscore again when done.

Run the import:

cd ..
dachs imp q

Now start the server as necessary (see above), and start TOPCAT and Aladin. In TOPCAT, open VO/SIA Query, enter your new service's access URL (it's http://localhost:8080/emi/q/s/siap.xml unless you did something cunning and should know better yourself) under “SIA URL” pretty far down in the dialog.

Then have “Lockman Hole” as Object Name and resolve it, or manually enter 161.25 and 58.0 as RA and Dec, respectively, and have 2 as Angular Size. Send off the request. You'll get back a table that you can send to Aladin (Interop/Send to/Aladin), which will respond by presenting a load dialog. Doubleclick and load as you like. Yes, the images look a bit like static noise. That's all right here – but do combine these images with, say, DSS colored optical imagery and marvel at the wonders of modern VLBI interferometry.

Indicentally, we made the detour through TOPCAT since there's no nice UI to query non-registred SIAP services in Aladin.

Defining SIAP Tables

SIAP-capable tables should mix in //siap#pgs. This mixin provides all the columns necessary for valid SIAP responses, and it will prepare the table so that spatial queries (which are the most common) will use a pgSphere index.

So, in the simplest case, a table published through a SIAP service would be declared like this:

<table id="images" onDisk="True" mixin="//siap#pgs"/>

This only has the minimal SIAP metadata. You will usually want to add additional columns with extra metadata from your images.

The //siap#pgs mixin also takes care that anything added to the table also ends up in the products table. This means that the grammar filling the table needs a //products#define rowfilter.

Filling SIAP Tables

When filling SIAP tables, you will almost always use:

• Element fitsProdGrammar as the grammar; see also fitsProdGrammars further down.
• the //siap#computePGS apply to interpret the FITS WCS to fill the coverage-related columns in the table.
• the //siap#setMeta apply to fill the non-spatial SIAP metadata.

In practice, this might look like this:

<data id="import_main">
  <sources recurse="True">
    <pattern>data/*.fits</pattern>
  </sources>
  <fitsProdGrammar qnd="True">
    <maxHeaderBlocks>80</maxHeaderBlocks>
    <mapKeys>
      <map key="object">OBJECT</map>
      <map key="obsdec">OBSDEC</map>
      <map key="obsra">OBSRA</map>
    </mapKeys>
    <rowfilter procDef="//products#define">
      <bind key="table">"emi.main"</bind>
    </rowfilter>
  </fitsProdGrammar>

  <make table="main" >
    <rowmaker id="gen_rmk" idmaps="object, obsra, obsdec">
      <apply procDef="//siap#computePGS"/>
      <apply procDef="//siap#setMeta">
        <bind name="bandpassLo">0.207</bind>
        <bind name="bandpassHi">0.228</bind>
        <bind name="bandpassId">"1.4 GHz"</bind>
        <bind name="bandpassRefval">0.214</bind>
        <bind name="bandpassUnit">"m"</bind>
        <!-- since the images have a fairly complex provenance,
        there's no way we can have sensible dates; this one here
        ought to be reasonably representative -->
        <bind name="dateObs"
          >dateTimeToMJD(datetime.datetime(2010, 7, 4))</bind>
        <bind name="instrument">@INSTRUME</bind>
        <bind name="title">"VLBA 1.4 GHz "+@object</bind>
      </apply>

      <apply name="fixObjectName">
        <setup>
          <code>
            import csv
            with open(rd.getAbsPath(
                "res/namemap.csv")) as f:
              nameMap = dict(csv.reader(f))
          </code>
        </setup>
        <code>
          @object = nameMap[@object]
        </code>
      </apply>

      <map key="weighting">\inputRelativePath.split("_")[-1][:-5]</map>
    </rowmaker>
  </make>
</data>

This does, step by step:

• The sources element is as always – with image collections, the recurse attribute often comes in particularly handy.
• When ingesting images, you will very typically read from FITS primary headers. That is what element fitsProdGrammar does unless told otherwise: Its rawdicts simply are the (astropy.io.fits) headers turned into plain python dictionaries.
• The qnd attribute of the grammar is recommended as long as you get away with it. It makes some (weak) assumptions to yield significant speedups, but it limits you to the primary header. You cannot use qnd with compressed FITS images. Also note the hdusField attribute when you have more complex FITSes to process.
• The fitsProdGrammar will map keys with hyphens to names with underscores, which allows for smoother action with them in rowmakers. The mapKeys element can produce additional mappings; in this case, we abuse it a bit to let us have idmaps (rather than simplemaps) in the rowmaker. And, actually, to illustrate the feature, as this data does not need key mapping, really.
• Since we are defining a table serving data products here, the grammar needs the //products#define rowfilter discussed in the products table.
• We have mentioned the //siap#computePGS mixin above; as long as astropy can deal with your WCS, it's really automatic (though you may want to pass parameters when you have cubes with suboptimal WCS or want to keep products without WCS in your table). And if you don't have proper WCS: see above on checking with dachs-supoort.
• The second apply you want when feeding SIAP tables is //siap#setMeta. Try to give all its keys somewhat sensible values, you will make your users' lives much easier.
• The exception to the “just fill all of setMeta's parameters” is the various bandpass* parameters. If the bandpass used is known to DaCHS, just set bandpassId and then use the //siap#getBandFromFilter apply. Whether a band is known you can find out by running dachs admin dumpDF data/filters.txt – and we are grateful for further contributions to that file.
• Typically, many values you find in the FITS headers will be messy and fouled up. You'll spend some quality time fixing these values in the typical case. Here, we translate somewhat broken object names using a simple mapping file that was provided by the author. In many other situations, the //procs#mapValue or //procs#dictMap applys let you do fixes with less code.
• As is usual in DaCHS procdures, you can access the embedding RD as rd. In our object name fixer, we use that to let DaCHS find the input file independently of where the programme was started.

Somewhat regrettably, //siap#setMeta cannot be used with idmaps="*"; as settings from setMeta would be overwritten then. This is because setMeta was written before the idmaps attribute existed and hence it wrote directly into the rowdict. Changing setMeta now would break existing RDs, which we try hard to avoid.

Cores

There are two cores you may want for SIAP services:

• Element dbCore, to which you add the necessary condDescs manually as below, for “normal” SIAP services.
• siapCutoutCore, which speaks SIAP but returns cutouts rather than full images; the size of these cutouts is determined by the SIZE argument (i.e., the region of interest).

To furnish these cores with the parameters required by the standard, use the //siap#protoInput condDesc. If you want to re-use the core for a form-based service, simply add the //siap#humanInput condDesc; as for SCS, the respective renderers will use parsers adapted to where the inputs come from.

So, a basic core with a couple of additional fields would look like this:

<dbCore id="query_images" queriedTable="main">
  <condDesc original="//siap#protoInput"/>
  <condDesc original="//siap#humanInput"/>
  <condDesc buildFrom="dateObs"/>
  <condDesc buildFrom="bandpassId" />
  <condDesc>
    <inputKey name="object" type="text"
        tablehead="Target Object"
        description="Object being observed, Simbad-resolvable form"
        ucd="meta.name" verbLevel="5" required="True">
        <values fromdb="object FROM lensunion.main"/>
    </inputKey>
  </condDesc>
</dbCore>

Service

If you wrote the core to work for both SIAP and form as described above, there's little more to say except you'll want to use the siap.xml renderer, and you need some additional metadata for VO registration. The latter is described with the siap.xml renderer.

With this, the service definition would look like this:

<service id="im" allowed="form,siap.xml" core="query_image">
  <meta name="shortName">sample images</meta>
  <meta name="title">Sample Image Archive</meta>
  <meta name="sia.type">Pointed</meta>

  <meta name="testQuery.pos.ra">230.444</meta>
  <meta name="testQuery.pos.dec">52.929</meta>
  <meta name="testQuery.size.ra">0.1</meta>
  <meta name="testQuery.size.dec">0.1</meta>

  <publish render="siap.xml" sets="ivo_managed"/>
  <publish render="form" sets="local,ivo_managed"/>
</service>

– where again you can just write the above core inline rather than referencing it; that's the style we usually recommend, and what the dachs start template has.

SIAP and Obscore

SIAP's metadata is quite a bit poorer than Obscore's (cf. publishing anything through Obscore). This means that you will usually have to fill a few extra parameters to arrive at a full Obscore record. But even a simple use of the //obscore#publishSIAP mixin will let people find your SIAP data in the obscore table.

The defaults given in the reference documentation show how we are mapping SIAP to Obscore. At least whatever is NULL by default you should critically examine and fill as appropriate.

DaCHS' SIAP template gives the most important extra fields:

<mixin
   calibLevel="2"
   collectionName="'%a few letters identifying this data%'"
   targetName="%column name of an object designation%"
   expTime="%column name of an exposure time%"
   targetClass="'%simbad taget class%'"
 >//obscore#publishSIAP</mixin>

You can delete any parameter you cannot fill or do not want to fill, though you really should put in a useful collectionName.

SIAP and Datalink

If you have larger images or cubes and serve them through SIAP, consider offering datalinks or perhaps even have Datalinks as Products. The latter case is particularly attractive if your images are so large that people just clicking on some row in Aladin might not expect a download of that size (in 2020, I'd set that limit at, perhaps, 100 MB). In both cases, people can select and download only parts of the image or request scaled versions of it (a less powerful and transparent alternative is the siapCutoutCore discussed above; the two options don't mix very well).

Defining a datalink service for normal FITS images is not hard. In the simplest case, you just give a bit of metadata, use the //soda#fits_genDesc descriptor generator (you don't need to understand exactly what that is; if you are curious: Datalink and SODA has the full story) and FEED //soda#fits_standardDLFuncs. Done.

The following example, taken from lswscans/res/positions, adds a fixed link to a scaled version, which might work a bit smoother with unsophisticated Datalink clients, using a meta maker:

<service id="dl" allowed="dlget,dlmeta">
  <meta name="title">HDAP Datalink</meta>
  <meta name="description">This service lets you access cutouts
    from HDAP plates and retrieve scaled versions.</meta>
  <datalinkCore>
    <descriptorGenerator procDef="//soda#fits_genDesc">
      <bind key="accrefPrefix">lswscans</bind>
    </descriptorGenerator>
     <FEED source="//soda#fits_standardDLFuncs"/>

    <metaMaker semantics="#science">
      <code>
        yield descriptor.makeLink(
          makeProductLink(descriptor.accref+"?scale=4"),
          contentType="image/fits",
          description="FITS, scaled by 1/4",
          contentLength=descriptor.estimateSize()/16.)
      </code>
    </metaMaker>
  </datalinkCore>
</service>

See Meta Makers for more information on what is going on inside the meta maker. The remaining material is either stereotypical or pure metadata: title and description are as for any other serivce, and the accrefPrefix should in general reflect your resource directory name. DaCHS' datalink machinery will reject any publisher DID asking for an accref not starting with that string. The idea here is to avoid applying code to datasets it is not written for.

When you attach the datalink functionality (rather than having datalink links as access URLs), the following is the recommended pattern:

1. Add a pub_did column to the SIAP table. Make sure you're giving it a UCD of meta.id;meta.main and that you have no other such column in your table:

   <table id="data" onDisk="true" mixin="//siap#pgs">
     <index columns="pub_did"/>
     <column name="pub_did" type="text"
       ucd="meta.id;meta.main"
       tablehead="P. DID"
       description="Dataset identifier assigned by the publisher"
       verbLevel="15"/>
   

2. Populate the pub_did column in the row maker:

   <map key="pub_did">\standardPubDID</map>
   

3. Declare your table as having datalink support; both SIAP and TAP will pick that up and add the necessary declarations so datalink-aware clients will know they can run datalink queries against your service:

   <meta name="_associatedDatalinkService">
     <meta name="serviceId">dl</meta>
     <meta name="idColumn">pub_did</meta>
   </meta>
   

   This block needs to sit in the table element. The serviceId meta contains the id of the datalink service.

4. If you also produce HTML forms and tables, see datalinks in columns.

Quick Sample SSAP Service

Check out the feros resource directory into your inputs directory:

cd `dachs config inputsDir`
svn co http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/feros
cd feros
mkdir data

As recommended, the checkout does not contain actual data, so let's fetch a file:

cd data
curl -O http://dc.g-vo.org/getproduct/feros/data/f04031.fits
cd ..

This RD also publishes to obscore, so make sure you have the obscore table:

dachs imp //obscore

If you do not plan to publish via obscore yourself (which is reasonably unlikely) and you try this on a box that the Registry will see later (you shouldn't), be sure to dachs drop obscore again when done.

Run the input and the regression tests:

dachs imp q
dachs test q

One regression test should fail since you've not yet pre-generated the previews (which are optional but recommended for your datasets, too):

python3 bin/makepreviews.py
dachs test q

If the regressions tests don't pass now, please complain to the authors of this tutorial.

From here on, you can point your favourite spectral client (fallback: TOPCAT's SSA client; note that TOPCAT itself cannot open this service's native format and you'll have to go through datalink, which TOPCAT knows how to do since about version 4.6) to http://localhost:8080/feros/q/ssa/ssap.xml and do your queries (if you don't know anything else, do a positional query for 149.734, 2.28216).

Please drop the dataset again when you're done playing with it:

dachs drop feros/q

Since the dataset is in the obscore table, it would otherwise be globally discoverable, and that'd be bad.

SSAP Tables

Contrary to what DaCHS does with the relatively small SCS, SIAP, and SLAP models, due to the size of the SSAP model, spectral services are always based on a database view on top of a table the structure of which is controlled by you; you're saving quite a bit of work if you keep your own table's columns as close to their SSA form as possible, though.

Another special situation is that most spectra are delivered in fairly crazy formats, which means that it's usually a good idea to have a datalink service that serves halfway standard files – in DaCHS, these comply to the VO's spectral data model, which is a VOTable with a bit of standard metadata. It's certainly not beautiful or very sensible, but it sure beats the IRAF-style 1D images people still routinely push around.

So, to start a spectral service, use the ssap+datalink template:

$ mkdir myspectra; cd myspectra
$ dachs start ssap+datalink

This will result in the usual q.rd file as per starting from scratch; see there for how to efficiently edit this and for explanations on the common metadata items.

SSAP-specific material starts at the meta definitions for ssap.dataSource and ssap.creationType. These are actually used in service discovery, so you should be careful to select the right words. Realistically, using survey/archival for observational data and theory/archival for theoretical spectra should be the right thing most of the time.

Next, you define the raw_data table; this should contain all metadata “unpredictably” varying between datasets (or, for large data collections, anything that needs to be indexed for good performance). For instance, for observational data, the observed location is going to change from row to row. The start and the end of the spectrum is probably going to be fixed for a given instrument, and so if you have a homogeneous data collection you probably will not have columns for them and rather provide constant values when defining the view.

To conveniently define the table, it is recommended to pull the SSA columns for raw_data by name from DaCHS' SSAP definitions and use SSAP conventions (e.g., units). The generated RD is set up for this by giving namePath="//ssap#instance", which essentially means “if someone requests an element by id, first look in the instance table of the RD”. This is then used in the following LOOP (cf. Active Tags). As generated, this will look somewhat like:

<LOOP listItems="ssa_dateObs ssa_dstitle ssa_targname ssa_length
  ssa_specres ssa_timeExt">

– this will pull in the enumerated columns as if you had defined them literally in the table. Depending on the nature of your data, you may want to pull in more columns if they vary for your datasets (or throw out ones you don't need, as ssa_dateObs for theoretical data).

To see what is available, refer to the reference documentation of the the //ssap#view mixin. Any parameter that starts with ssa_ can be pulled in as a column.

The template RD then mixes in //products#table (which you pretty certainly want; see The Products Table for an explanation), //ssap#plainlocation (which you want if you have positions on observational data) and //ssap#simpleCoverage (which you want if you want to publish your observational spectra through obscore). The template then defines:

<FEED source="//scs#splitPosIndex"
  long="degrees(long(ssa_location))"
  lat="degrees(lat(ssa_location))"/>

This again is mainly useful for obscore as long as DaCHS' ADQL engine may turn queries into q3c statements; just leave it if you have positions, and remove it if you don't.

You can, of course, define further columns here, both for later use in the view and for local management. SSAP lets you return arbitrary local columns, and in particular for theory services, you will have to (to define the physics of your model). As a DaCHS convention, please don't use the ssa_ prefix on your custom columns. See theossa/q for an example of a table with many extra columns.

The SSA template then goes on with a data item filling the raw_data table. The template assumes you're parsing from IRAF-style 1D images. You will have to use a different grammar if that is not what you have, and in that case you in particular you cannot use the specAx var defined in the rowmaker.

The data item has <recreateAfter>make_view</recreateAfter> quite early on; this simply makes sure that the SSA view will be regenerated after you import the table itself.

The rowfilter in the grammar is fairly complex here because we will completely hide the original; if you simply want to serve your upstream format, just cut it down to just giving table, mime, preview and preview_mime. If you do that, use the following strings in mime:

• image/fits for IRAF-style 1D image spectra
• application/fits for spectra in FITS tables
• application/x-votable+xml for spectra in VOTables

Please do not put anything else into SSA tables, because you will most certainly overstrain most SSA clients; if you have a different upstream format and you want to make it available, turn it into SDM VOTables and use datalink to link to the original source.

Hence, for most cases (including also ASCII spectra), here's what we recommend as the product definition rowfilter (it's roughly what's in the template) to isolate your clients from the odd upstream formats:

<rowfilter procDef="//products#define">
  <bind name="table">"\\schema.main"</bind>
  <bind name="path">\\fullDLURL{"sdl"}</bind>
  <bind name="fsize">%typical size of an SDM VOTable%</bind>
  <bind name="datalink">"\\rdId#sdl"</bind>
  <bind name="mime">"application/x-votable+xml"</bind>
  <bind name="preview">\\standardPreviewPath</bind>
  <bind name="preview_mime">"image/png"</bind>
</rowfilter>

This is pointing the path accessed to a datalink URL using the fullDLURL macro, which expands to a URL retrieving the full dataset; the “sdl” argument to the macro references the datalink service defined further down. Since the data returned is generated on the fly, you will have to give an estimate of how large the VOTable will be (overriding DaCHS' default of the size of the source file). Don't sweat this too much, just don't claim something is 1e9 bytes when you're really just returning a few kilobytes. The rowfilter expects the size in bytes.

The bindings already prepare for making and serving previews, which is discussed in more detail in Product Previews in the DaCHS reference; see there for everything mentioning “preview“.

SSAP has a feature that lets users request certain formats, and for clients that don't know Datalink, this may be a good idea. In that scheme, you use a rowfilter to return a description of your native data and the processed SDM-compliant dataset as used here. See theossa/q for an example how that would look like. Our recommendation: don't bother, it's a misfeature that will most likely just confuse your users.

The rowmaker is fairly standard; we should perhaps mention the elements:

<var name="specAx">%use getWCSAxis(@header_, 1) for IRAF FITSes
  (delete otherwise)%</var>
<map key="ssa_specstart">%typically, @specAx.pixToPhys(1)*1e-10%</map>
<map name="ssa_length">%typically, @specAx.axisLength%</map>

getWCSAxis is a function that looks at a FITS image's WCS information to let you transform pixel to physical coordinates. This currently uses a simplified DaCHS implemenation that only does a small part of WCS (but we may change that, keeping the interface stable). The var, anyway, binds the resulting object to specAx. You can use that later to find out the limits of the spectrum. The way it is written here, you will still have convert the value to metres manually. But as said above, if you're publishing a homogeneous collection of spectra, both values are probably constant, and you'll want to remove both maps from the template.

The SSA View

The template goes on defining the data table that will serve as the basis of the service. It starts with the declaration:

<meta name="_associatedDatalinkService">
  <meta name="serviceId">sdl</meta>
  <meta name="idColumn">ssa_pubDID</meta>
</meta>

This is lets DaCHS add a link to the datalink service to results generated from this (both via SSAP and TAP). There's nothing you need to change here (unless you chuck datalink); see SSAP and Datalink and Datalink for details.

The main body of the table definition is the //ssap#view mixin. In it, you need to write the SSA parameters as SQL literals (i.e., strings need single quotes around them) or expressions involving column references. To keep things fast, you should have SSA-ready columns in the source table, so you will usually have column references only here. Most of these items default to NULL, so if you do not have a piece of metadata, it is reasonably safe to just remove the attribute.

A few of the mixin's parameters deserve extra discussion:

• sourcetable – this is a reference, i.e., this must resolve to the id of some table element. It can be cross-RD if really necessary. It is not the SQL table reference (that would include a schema reference).
• copiedcolumns – this lets you copy over columns from the source table, i.e., the one you just defined using (comma-separated) shell patterns of their names (yes, that's just like idmaps in rowmakers). The * given in the template should work in most cases, but if you have private columns in the source table, you can suppress them in the view; a useful convention might be to start all private columns with a p; you'd then say copiedcolumns="[^p]*". Note that copied columns are automatically added in the view as 1:1 maps, and you cannot use view arguments to override them. Use different column names in the source table and the view if you (think you) have to do view-level processing of your values.
• customcode – use this if you have extra material to enter in the view definition generated by the mixin. We hope you won't need that and would be interested in your use case if you find yourself using this.
• ssa_spectralunit, ssa_fluxunit – these are the only mandatory parameters starting with ssa_ (but their values are still overwritten if they are in copiedcolumns). There really is no point in having them vary from row to row because their values are metadata for the corresponding error columns (which is one of the many spec bugs in SSAP).
• ssa_spectralucd, ssa_fluxucd – these are like the unit parameters in that they contain data collection-level metadata. The only reason they are not mandatory is that there are defaults that seem sensible for a large number of cases. Check them, and again, you cannot really let them vary from row to row.
• ssa_fluxSI, ssa_spectralSI, ssa_timeSI – these were an attempt to work around a missing specification for unit strings in the VO. Since we now have VOUnit, just ignore them.

The data item making this table is trivial. You should set it to auto="False" (i.e., don't build this on an unadorned dachs imp). The building of this data will normally be triggered by the recreateAfter of the source table import.

SSAP Services

Use the element ssapCore for SSAP services. You must feed in the condition descriptors for the SSAP parameters you want to support (some are mandatory). The simplest way to do that is to FEED the //ssap#hcd_condDescs stream. It includes condition descriptors for all mandatory and optional parameters that we can remotely see in use.

Some of them may not be relevant to your service because your table never has values for them. For example, theoretical spectra will typically not give information on positions. The SSAP spec says that such a service should ignore POS rather than returning the empty set. We consider that an unfortunate recommendation that you should ignore; if someone queries your theoretical service with a position, it is highly likely they do not want to see all your spectra.

If you nevertheless think you must ignore certain conditions, you can use the PRUNE active tag. This looks like this:

<ssapCore queriedTable="newdata">
  <FEED source="//ssap#hcd_condDescs">
    <PRUNE id="coneCond"/>
    <PRUNE id="bandCond"/>
  </FEED>
</ssapCore>

Again, do not do this just because you don't have, say position information.

Here is a table of parameter names and ids; you can always check them by inspecting the output of dachs adm dumpDF //ssap:

For APERTURE, SNR, REDSHIFT, TARGETNAME, TARGETCLASS, PUBDID, CREATORDID, and MTIME, the condDesc id simply is <keyname>_cond, e.g., APERTURE_cond.

To have custom parameters, simply add condDesc elements as usual:

<ssapCore queriedTable="newdata">
  <FEED source="//ssap#hcd_condDescs"/>
  <condDesc buildFrom="t_eff"/>
</ssapCore>

For SSAP cores, buildFrom will enable “PQL”-like query syntax such that users can post arguments like 20000/30000,35000 to t_eff. This is in keeping with the general SSAP parameter style, while more modern VO services use 2-arrays for intervals (“DALI style”).

To expose SSAP cores, use the ssap.xml renderer.

Form-based Spectral services

Using the form renderer on SSAP cores is not terribly useful, because the core returns XML directly, and there are far too many parameters no human will ever be interested in anyway. Hence, you will typically define extra browser-based services. The example RD shows a compact way to do that:

<service id="web" defaultRenderer="form">
  <meta name="shortName">\\schema Web</meta>

  <dbCore queriedTable="main">
    <condDesc buildFrom="ssa_location"/>
    <condDesc buildFrom="ssa_dateObs"/>
    <condDesc>
      <inputKey original="data.ssa_targname" tablehead="Star">
        <values fromdb="ssa_targname from theossa.data
          order by ssa_targname"/>
      </inputKey>
    </condDesc>
  </dbCore>

  <outputTable>
    <autoCols>accref, mime, ssa_targname,
      ssa_aperture, ssa_dateObs, datalink</autoCols>
    <FEED source="//ssap#atomicCoords"/>
    <outputField original="ssa_specstart" displayHint="displayUnit=Angstrom"/>
    <outputField original="ssa_specend" displayHint="displayUnit=Angstrom"/>
  </outputTable>
</service>

Essentially, we only select a few fields people might want to query against, and we directly build them out of the query fields; the SSA condDescs are bound to the funny and insufficiently defined SSA input syntax and probably not very useful in interactive applications.

The extra selector for object names with the names actually present in the database is a nice service as long as you only have a few hundred objects or so. Since the query over ssa_targname is executed at each load of the RD, it should be fast, which means that even for medium-sized tables, you should have an index on the object names in the raw_data table, probably like this:

<index columns="ssa_targname"/>

In the output table, we only give a few of the many dozen SSAP output fields, and we change the units of the spectral limits to Angstroms, which will look nicer for optical spectra. For educational reasons you might want to change this to nm (nanometer).

In the template, this form-based service is published as a capability of the SSA service. This is done using the service attribute in the Element publish in the SSAP service element:

<publish render="form" sets="ivo_managed,local" service="web"/>

See Registering Web Interfaces to DAL Services for more background.

SSAP and Obscore

The SSA metadata is not far from the Obscore metadata (cf. publishing anything through obscore), and so an Obscore publication of SSAP data almost comes for free: Minimally, just mix in //obscore#publishSSAPMIXC and set calibLevel. The template does a bit more:

<mixin
  calibLevel="%likely one of 1 for uncalibrated or 2 for uncalibrated data%"
  coverage="%ssa_region -- or remove this if you have no ssa_region%"
  sResolution="ssa_spaceres"
  oUCD="ssa_fluxucd"
  emUCD="ssa_spectralucd"
  >//obscore#publishSSAPMIXC</mixin>

– if you use one of the old hcd or mixc mixins, you do not want oUCD and emUCD.

In particular for larger spectral collections, it is highly recommended to also have the //ssap#simpleCoverage mixin in an obscore-published spectral table; only then will you get indexed queries when there are constraints on s_region, and having these non-indexed will lead to really slow obscore queries.

SSAP and Datalink

Given that you will usually get fairly bizarre inputs and will probably want to publish “repaired” spectra, using Datalink to provide both native and SDM (“Spectral Data Model”) compliant spectra without having to resort to SSAP's ill-thought-out FORMAT feature is a fairly natural thing to do. That is why the SSAP+datalink template comes with almost all that you need to do that; what is left is mainly to write an embedded grammar to parse the spectra (if the parsing is complex, you might to go for an Element customGrammar, which lets you keep the source outside of the RD). Other than that, it is just a few formalities.

So, you first define the table that will later hold your spectrum. Use the //ssap#sdm-instance mixin for that (this continues material from the template):

<table id="instance" onDisk="False">
  <mixin ssaTable="main"
    spectralDescription="%something like 'Wavelength' or so%"
    fluxDescription="%something like 'Flux density' or so%"
    >//ssap#sdm-instance</mixin>
  <meta name="description">%a few words what a spectrum represents%</meta>
</table>

The descriptions you need to enter here typically end up being labels on axes of spectral plots, so it is particularly important to be concise and precise here.

If your spectrum has additional columns (e.g., errors, noise estimates, bin widths), just put more columns in here. The mixin pulls in all the various params that SDM wants to see from the row of the spectrum in the SSAP table.

Note that the table does not have onDisk="True"; these tables are only made for the brief moment it takes to serialise them into what the user receives.

As usual, to fill tables, you want a data element. The template just gives a few hints on how that might work. As a working example, zcosmos/q parses from 1D FITS images like this:

<data id="build_sdm_data" auto="False">
  <embeddedGrammar>
    <iterator>
      <setup imports="gavo.protocols.products, gavo.utils.pyfits"/>
      <code>

      fitsPath = products.RAccref.fromString(
          self.sourceToken["accref"]).localpath
      hdus = pyfits.open(fitsPath)
      ax = utils.getWCSAxis(hdus[0].header, 1)

      for spec, flux in enumerate(hdus[0].data[0]):
        yield {"spectral": ax.pix0ToPhys(spec), "flux": flux}
      hdus.close()
      </code>
    </iterator>
  </embeddedGrammar>
  <make table="spectrum">
    <parmaker>
      <apply procDef="//ssap#feedSSAToSDM"/>
    </parmaker>
  </make>
</data>

The way this figures out the file from which to parse will work if you have the actual file path in the product table. When you hide the upstream format as recommended, you have to follow some custom convention. The SSAP+datalink template has:

sourcePath = urllib.decode(
  self.sourceToken["ssa_pubDID"].split('?', 1)[-1])

This works if you use the macro standardPubDID as in the template.

But you can do arbitrary things here; see the califa/q3 RD for an example for how you can decode the accref to database rows. A more complex scenario with ASCII data stored externally and cached is in theossa/q, a case where multiple orders of echelle spectra are being processed in flashheros/q.

You will notice that the rowmaker in both the example and the template is missing. DaCHS will then fill in the default one, which essentially is idmaps="*". Since you are writing the grammar from scratch, just use the names of the columns defined in the instance table and be done with it. The predefined column names are spectral and flux, so make sure you always have keys for them in the dictionaries you yield from your grammars.

While there is no rowmaker, the make does have an Element parmaker; this is stereotypical, just always use the procDef as here. It copies values from the SSA input row to the params in the instance table.

Finally, you would need to write the service. For SSAP and SODA, what's in the template ought to just work. Add Element metaMaker-s to provide links to, e.g., your raw input files if you want. In that case, please skim the endless chapter on Datalink and SODA in the reference documentation to get an idea of how descriptor generators, data functions, and meta makers play together.

For reasons discussed in Datalinks in Columns, it may be a good idea to include a custom column with a datalink URL in the SSAP table. The ssap+datalink template already has such a column in its source table and fills it in import's rowmaker.

SSAP for Time Series

As long as there is no anointed successor to SSAP explicitly catering to time series, you can use SSAP to publish time series. It is a bit of a hack, but clients like SPLAT do about the right thing with them.

As to examples, check out k2c9vst/q (which parses the data from ASCII files) and gaia/q2 (which stores the actual time series in the database, a technique we believe is a very good idea).

To notify the Registry (and possibly clients, too), that you are producing time series, do two things:

1. Globally declare that you serve time series by setting:

   <meta name="productType">timeseries</meta>
   

near the top of your RD. The ssap+datalink template has more information on the productType meta.

2. have 'timeseries' as ssa_dstype in your view definition.

Building Time Series VOTables

This section is under construction

Since there is no actual agreed-upon standard for the serialisation of time series, you will probably have to produce time series on the fly; DaCHS helps you to produce something that will hopefully follow standardisation efforts in the VO without much additional work later on, using, you guessed it, a mixin. For now, the only mixin available is for photometric timeseries: See the //timeseries#phot-0 mixin to build things. If you have other time series, please write mail to dachs-support.

To see things in action, refer to k2c9vst/q, the instance table and the corresponding makes.

However, there is an additional complication, showcased in gaia/q2 and bgds/l: it is quite common to have time series in multiple bands in one resource. For DaCHS, this is a bit of a problem, because the band influences quite a bit of the table metadata in DaCHS – this is in the mixin, and what you set there is fixed once the table instance is made.

To get around this, look at the technique shown in bgds/l. This first defines a STREAM time-series-template with a macros where the items very between bands:

<STREAM id="time-series-template">
  <table id="instance-\band_short">
    <mixin
      effectiveWavelength="\effective_wavelength"
      filterIdentifier="\band_human"
      longitude="@ra"
      latitude="@dec"

It then uses a LOOP to fill these slots and create one table definition per band:

 <LOOP>
  <csvItems>
    band_short, band_human, band_ucd, effective_wavelength
    i,          SDSS/i,     em.opt.I, 7.44e-7
    r,          SDSS/r,     em.opt.R, 6.12e-7
  </csvItems>
  <events source="time-series-template"/>
</LOOP>

The dispatch between the different table templates then happens in the data function of the tsdl service, using a somewhat obscure feature of rsc.Data: when using the createWithTable class function, you can pass in the table that the data item should make. This obviously only works in specialised circumstances like the one here, but then it's really convenient. So, while the make in make_instance claim to build instance-i, this is really overridden in the datalink service's data function to select the actual table definition:

<dataFunction>
  <setup imports="gavo.rsc"/>
  <code>
    dd = rd.getById("make_instance")
    descriptor.data = rsc.Data.createWithTable(dd,
      rd.getById("instance_"+descriptor.band))
    descriptor.data = rsc.makeData(
      dd,
      data=descriptor.data,
      forceSource=descriptor)
  </code>
</dataFunction>

(where descriptor.band has been pulled from the dataset identifier in the custom descriptor generator of that datalink service; that pattern is probably a good idea when you are in a similar situation).

This will not scale well to many dozens of bands – if you have that, you probably want somewhat more hardcore means –, but for the usual handful of bands this is a relatively reasonable way to produce time series with nice metadata.

Obscore Derived from Typed Service Tables

Obscore defines a single table called ivoa.obscore. In DaCHS, that table typically contains data from a multitude of resources with different metadata structures. To keep that manageable, DaCHS implements the table as a view, where the individual tables are mapped onto that common schema. These mappings are almost always created using a mixin from the //obscore RD. Filling out its parameters will result in SQL DDL fragments that are eventually combined to the view definition. In case you are curious: The fragments are kept in the ivoa._obscoresources table.

There is some documentation on what to put where in the mixin documentation, but frankly, as a publisher, you should have at least passing knowledge of the obscore data model (2017ivoa.spec.0509L).

When you start with a table underlying a typed service, you can get away with just saying something like (using SIAP as an example):

mixin="//obscore#publishSIAP"

to the table definition's start tag. You do not have to re-import a table to publish it to Obscore when you have already imported it – dachs imp -m <rd id> && dachs imp //obscore will include an existing table in the obscore view.

When you import data without the -m flag, the mixins arrange for everything, so you do not need the extra step of importing //obscore.

Since the Obscore data model is quite a bit richer than SIAP's and just a bit richer than SSAP's, you will usually want to add extra metadata through the mixin, for instance:

<mixin
  sResolution="0.5"
  calibLevel="2"
  >//obscore#publishSIAP</mixin>

Again, a dachs imp -m followed by an import of //obscore would be enough to make these changes visible in ivoa.obscore.

See SIAP and Obscore and SSAP and Obscore for more information on how to Obscore-publish typed data.

Obscore Notes

<map key="s_region">pgsphere.SCircle.fromDALI(
   [alpha, delta, aperture]).asPoly(6)</map>

Pure Obscore Tables

You can also have “pure” Obscore tables which do not build on protocol mixins. A live example is the cubes table in the califa/q3 RD within the GAVO data center. Here is a brief explanation of how this works.

Somewhat like with the SSA view, you define a table for the obscore columns varying for your particular data collection. In that tables' definition re-use the metadata given in the global obscore table. A compact way to do that is through a LOOP (see Active Tags) and original references, exploiting the namePath on Element Table:

<table id="cubes" onDisk="True" namePath="//obscore#ObsCore">
  <LOOP listItems="obs_id obs_title obs_publisher_did
    target_name t_exptime t_min t_max s_region
    t_exptime em_min em_max em_res_power">
    <events>
      <column original="\item"/>
    </events>
  </LOOP>

adql="True" is absent here as the obscore mixin will set it later. If you do not have any additional columns (which you can of course have) and just want to have your datasets in the obscore table, consider having <adql>hidden</adql> after the obscore mixin. This will make your table invisible to but still readable by TAP. This is desirable in such a situation because the entire information of the table would already be contained in the obscore table, and thus there is no real reason to query the extra table. In the Califa example cited above, that is obviously not the case; there is a wealth of additional columns in the custom, non-obscore table. We believe this will be the rule rather than the exception.

For a quick overview over what column names you can have in the listItems above, see the obscore table description.

Even with a custom obscore-like table, you will almost always want to have DaCHS manage your products. This works even when all your files are external (i.e., you're entering http URLs in //products#define's path), and so use the //products#table mixin (which you don't see with SIAP and SSAP as their mixins pull it in for you):

<mixin>//products#table</mixin>

Then, apply the //obscore#publish mixin, which is like the protocol-specific mixins except it doesn't pre-set parameters based on what is already in protocol-specific tables:

<mixin
    accessURL="dlurl"
    size="10"
    mime="'application/x-votable+xml;content=datalink'"
    calibLevel="3"
    collectionName="'CALIFA'"
    coverage="s_region"
    dec="s_dec"
    emMax="7e-7"
    emMin="3.7e-7"
    emResPower="4000/red_disp_mean"
    expTime="t_exptime"
    facilityName="'Calar Alto'"
    fov="0.01"
    instrumentName="'PMAS/PPAK at 3.5m Calar Alto'"
    oUCD="'phot.flux;em.opt'"
    productType="'cube'"
    ra="s_ra"
    sResolution="0.0002778"
    title="obs_title"
    tMax="t_min"
    tMin="t_max"
    targetClass="'Galaxy'"
    targetName="target_name"
  >//obscore#publish</mixin>

Essentially, what is constant is given in literals, what is variable is given as a column reference. It is a bit unfortunate that you have to enter quite a few identity mappings in here (and our foolish camel-case mogrification of the parameter names doesn't help). Telling us you are annoyed with this will certainly speed up our efforts to fix things.

That's about it for defining the table. To fill the table, just have a normal rowmaker; since the table contains products, don't forget the //products#define rowfilter in the grammar.

Registering Obscore-Published Datasets

Most of the time, you do not need to worry about telling the Registry anything about what you do with obscore. As long as you have the obscore table, your TAP registry record will tell the Registry about it, and as long as that is published, clients looking for obscore-published data will find your service and thus your datasets (if they match the constraints in the obscore query, that is).

In the other direction, when you register a service for a data collection published via a typed protocol, DaCHS will add a reference such that clients can see that the data is available through obscore, too.

But when you do not register a typed service for your data collection for some reason, you should also register the standalone table as described in publishing DaCHS-managed tables via TAP

SIAP version 2

SIAP version 2 is just a a thin layer of parameters on top of obscore. To publish with SIAP version 2, simply ingest your data as described in publishing images via SIAP and add the //obscore#publishSIAP mixin.

In contrast to SIAP version 1, you do not define or register a service for a SIAPv2-published data collection. Instead, there is a sitewide SIAPv2 service at <root URL>/__system__/siap2/sitewide/siap.xml. It is always there, but it is unpublished by default. To publish it, you should furnish some extra metadata in the userconfig RD and then run:

dachs pub //siap2

Specifically, get the sitewidesiap2-extras stream and follow the instructions there to update the meta items as appropriate; at this point, they are exactly analogous to the ones for SIAP version 1.

Quick Sample EPN-TAP Service

Install PyPDS if you don't have it anyway:

curl -LO https://github.com/RyanBalfanz/PyPDS/archive/master.zip
unzip master.zip
cd PyPDS
python setup.py build
sudo python setup.py install

Get the sample data:

cd `dachs config inputsDir`
curl -O http://docs.g-vo.org/epntap-example.tar.gz
tar -xvzf epntap-example.tar.gz
cd lutetia

Import it and build the previews from the PDS images:

dachs imp q
python bin/makePreview.py

Start the server as necessary. If you go to your local ADQL endpoint (something like http://localhost:8080/adql) and execute queries like:

SELECT * FROM lutetia.epn_core

there.

For access through a standard protocol, start TOPCAT, select VO/TAP Query, and at the bottom of the dialog enter http://localhost:8080/tap (or whatever you configured) in “TAP URL”. Hit “Use Service”, wait until the table metadata is in and then again query something like:

SELECT * FROM lutetia.epn_core

Hit “Run Query”,open the table and play with it. As a little visual treat, in TOPCAT's main window hit “Activation Action”, and configure the preview_url column under “View URL as Image”. Then click on the table rows.

To get into Vespa's query interface, you will have to register your table. Do not do this with the sample data.

Tables

In essence, EPNcore is just a set of columns, some mandatory, some optional. The mandatory ones are pulled into a table by using the //epntap2#table-2_0 mixin, which needs the spatial_frame_type parameter (see the reference for what's supported for it) since that determines the metadata on the spatial columns. Optional columns can be pulled in through the optional_columns mixin parameter, and, as said above, a few of these optional columns are actually required if you want to publish data products through EPN-TAP. The reference documentation lists what is available. You can, of course, define further, non-standard columns as usual.

Filling EPNcore Tables

To populate EPNcore tables, use the //epntap2#populate-2_0, apply identifying the parameters applying to your data collection and setting them as usual (cf. Mapping Data). You may need to refer to the EPN-TAP proposed specification now and then while doing that. Note again that parameter values are python expressions, and so you have to use quotes when specifying literal strings.

If you have to evaluate complex expressions, it is recommended to do the computations in Element var-s and then use the variables set there in the bind``s (as ``@myvar). This also lets you re-use values once computed. Even more complex, multi-statement computations can be done in Element apply with custom code.

On s_region in EPN-TAP

The s_region parameter (see //epntap2#populate-2_0) is essentially a footprint describing the area covered by 2D spatially extended data products. It uses pgshpere types such as spoly, scircle, smoc, or spoint (we advise against the use of spoint as a s_region type: only spatially extended types should be used). The default type is spoly, the others must be specified using the regiontype mixin parameter (see //epntap2#table-2_0).

For more information on how to create values for these regions, see Creating pgSphere Geometries.

Service

EPN-TAP tables are queried through the data center's TAP service. If you have registred that, there is nothing else you need to do to access your data.

For registration, just add:

<publish/>

to your table body and run dachs pub <rd-id>.

Associating Datalink Services

In DaCHS, Datalink services are associated with tables. This association is declared using the _associatedDatalinkService meta item, which consists of a serviceId (a service reference as per referencing in DaCHS) and an idColumn (stating from which column the ID parameter to the datalink service is to be taken from). So, within the table, you add something like:

<meta name="_associatedDatalinkService">
   <meta name="serviceId">dl</meta>
   <meta name="idColumn">pub_did</meta>
</meta>

This implies that the service dl within the current RD will produce a datalink document if passed a string from idColumn. The example implies that this column ought to contain publisher DIDs (see Dataset Identifiers), which is what the standard descriptor generators that come with DaCHS like to see. Since publisher DIDs tend to be a bit unwieldy (they are supposed to be globally unique, after all), the standard descriptor generators will also let you pass in plain accrefs.

If you write your own descriptor generator, you are free to stick whatever you like into the idColumn, just so long the table and the descriptor generator agree on its interpretation.

Datalinks in Columns

The _associatedDatalinkService declaration discussed in the previous section is all it takes when you serve data to datalink-aware clients. If, however, you also want to cater to clients without native datalink support, you may want to add links to the datalink documents in your responses; this is particularly advisable when you have services working through forms in web browsers.

One way to effect that is by defining a column like this:

<column name="datalink" type="text"
    ucd="meta.ref.url"
    tablehead="DL"
    description="URL of a datalink document for this dataset"
    verbLevel="1" displayHint="type=url">
  <property name="targetType"
    >application/x-votable+xml;content=datalink</property>
  <property name="targetTitle">Datalink</property>
</column>

The property declarations add some elements to response VOTables that inform clients like Aladin what to expect when following that link. At this point, this is a nonstandard convention.

You will then have to fill that column in the rowmaker. As long as the product is being managed through the products table and you thus used the //products#define rowfilter in the grammar, all that takes is a macro:

<map key="datalink">\dlMetaURI{dl}</map>

Here, the “dl” in the macro argument must be the id of the datalink service.

This method will retain the datalink columns even in protocol responses. While at this point there is something to be said for that, because users immediately discover that datalink is available, datalink-aware clients will then have both the datalink through _associatedDatalinkService and the in-table column, which, since they cannot know that the two are really the same, will degrade user experience: Why should the same datalink be present twice?

With increasing availability of datalink-aware protocol clients, we therefore prefer a second alternative: produce the extra datalinks only when rendering form responses. To do that, furnish web-facing services with an Element outputTable. In there, do not include the column with your publisher DID but instead produce a link directly to the links response, somewhat like this:

<service id="web" core="siacore">
  ...
  <outputTable>
     <outputField name="dlurl" select="accref"
      tablehead="Datalink Access"
      description="URL of a datalink document for the dataset
        (cutouts, different formats, etc)">
      <formatter>
        yield T.a(href=getDatalinkMetaLink(
          rd.getById("dl"), data)
          )["Datalink"]
      </formatter>
      <property name="targetType"
        >application/x-votable+xml;content=datalink</property>
      <property name="targetTitle">Datalink</property>
    </outputField>

Datalinks as Products

In particular for large datasets, it is usually a good idea to keep people from blindly pulling the data without first having been made aware that what they're accessing is not just a few megabyte of FITS. For that, datalink is a good mechanism by pointing to a links response as the primary document retrieved.

Of course, without a datalink-enabled client people might be locked out from the dataset entirely. On the other hand, DaCHS comes with a stylesheet formatting links responses to be usable in a common web brower, so that might still be preferable to overwhelming unsuspecting clients with large amounts of data.

To have datalinks rather than the plain dataset as what the accref points to, you need to change what DaCHS thinks of your dataset; this is what the //products#define rowfilter in your grammar is for:

<fitsProdGrammar qnd="True">
  <rowfilter procDef="//products#define">
    <bind key="path">\dlMetaURI{dl}</bind>
    <bind key="mime">'application/x-votable+xml;content=datalink'</bind>
    <bind key="fsize">10000</bind>
    [...]
  </rowfilter>
  [...]
</fitsProdGrammar>

The fsize here reflects an estimation of the size of the links response.

When you do this, you must use a descriptor generator that does not fetch the actual file location from the path in the products table, since that column now contains the URI of the links response.

For FITS images, you can use the DLFITSProductDescriptor class as //soda#fits_genDesc's descClass parameter. The base functionality of a FITS cutout service with datalink products would then be:

 <service id="dl" allowed="dlget,dlmeta">
  <meta name="title">My Cutout Service</meta>
  <datalinkCore>
    <descriptorGenerator procDef="//soda#fits_genDesc"
      name="genFITSDesc">
      <bind key="accrefPrefix">'mysvcs/data'</bind>
      <bind key="descClass">DLFITSProductDescriptor</bind>
    </descriptorGenerator>
    <FEED source="//soda#fits_standardDLFuncs"/>
  </datalinkCore>
</service>

If you have something else, you will have to write the resolution code yourself – DLFITSProductDescriptor's sources (in gavo.protocols.datalink) should give you a head start on how to do that); see also the tsdl service bgds/l for how to integrate that into your RD.

Note that DaCHS will not produce automatic previews in this situation. Have a look at Product Previews for what to do instead.

The Server URL, Ports, and Addresses

In today's internet with its reverse proxies and firewalls, a service cannot usually figure out by itself what it can be reached as. That is why you will almost certainly have to tell DaCHS in [web]serverURL.

For the basic setup, we recommend you try things with HTTP first. HTTPS adds quite a bit that can go wrong. See Enabling HTTPS on how to enable HTTPS once the rest runs as you would like it to run.

Also note that serverURL is reflected in several internal tables of DaCHS. If you change it later, remember to run dachs pub -a to update them; the side effect is that (at least if you use DaCHS' built-in publishing registry) this will update the links in the VO Registry as well.

There are a few scenarios:

[web]
bindAddress:
serverURL: http://mydc.example.org:8080

[web]
bindAddress:
serverPort: 80
serverURL: http://mydc.example.org

[web]
serverURL: https://the.proxy.url
adaptProtocol: False

[web]
serverURL: http://the.proxy.url
serverPort: 8081
adaptProtocol: False

The Site Name

The machinery needs some short name for the site in so many places that this is in the basic configuration (rather than the basic metadata, where it should reasonably be). This should be a very terse phrase saying who you are; for Heidelberg, that's “GAVO Data Center”. That phrase goes into the [web]sitename item.

The Maintainer Address

There are several situations when the server wants to contact you (e.g., failed cron jobs). To let it do that, teach the server it is on to send mail (we like the nullmailer package for things like that) and set [general]maintainerAddress to a mail address you regularly read.

Path Configuration

If you would like to (perhaps temporarily, for instance in a ~/.gavorc) override where DaCHS looks for its tree (the /var/gavo we have been assuming here; this is often called $GAVO_ROOT in the reference documentation), set [general]rootDir, perhaps like this:

[general]
rootDir: /home/user/gavo

You can also override other paths (see Section [general]), which are then are interpreted relative to rootDir unless they start with a slash.

Two items you may want to change if your rootDir is on a network file system are [general]tempDir and [general]cacheDir – things probably run more smoothly if they are local.

The Admin Password

You can influence a running DaCHS server through its web interface after authenticating as gavoadmin. The password to use for that is set in the [web]adminpasswd configuration item (and has no default, which means no remote administration is possible). You normally want this on a site that has publications now and then, as having it lets dachs imp and dachs pub tell the web component to release some caches, and it also makes dachs serve expire work (which is sometimes useful to expunge cached RDs).

Authenticated gavoadmin can also access all restricted resources.

The password is stored in clear text in gavo.rc; doing some kind of encryption would only make sense if you were prepared to enter some kind of passphrase every time you start the server. As in other places, DaCHS assumes the machine it runs on is trusted. You can restrict gavo.rc access to the gavo group, but of course that only helps against people who do not need to run the dachs command (which needs access to gavo.rc).

Creating a Userconfig RD

DaCHS has a builtin RD //userconfig that is updated as you update DaCHS. It always contains fallbacks for everything that can be in userconfig used by the core code. If an item is searched in the Userconfig RD, it is first searched in your local copy (which resides in /var/gavo/etc) and then in the distributed userconfig. That way, DaCHS can add new things in the userconfig without forcing you to merge on every update.

To create your local, editable copy, pull the distributed RD into /var/gavo/etc/userconfig.rd (unless, of course, that file already exists):

cd `dachs config configDir`
dachs admin dumpDF //userconfig > userconfig.rd

Maintaining and Changing the Userconfig RD

If you already have a local userconfig.rd and cannot find something referenced in a userconfig in there, instead run:

dachs admin dumpDF //userconfig | less

then pick out the element(s) for whatever you want to change and copy them into your own etc/userconfig.rd.

Changes to userconfig.rd are picked up by DaCHS but will usually not be visible in the RDs they end up in. This is because DaCHS does not track which RDs make use of userconfig, so these will typically need to be reloaded manually. For instance, if you changed TAP examples, you'd need to run:

gavo serve exp //tap

to make your change show up in the web interface. Although usually not necessary, you can reload userconfig itself using:

gavo serve exp %

Note that for gavo serve exp to work, you need to set The Admin Password, and if all that confuses you you can always just restart the server.

Referencing into the Userconfig RD

While Referencing in DaCHS in principle applies to the userconfig RD, normal RDs cannot do the fallback between the local and built-in RD. For this, there is a special syntax that denotes this “magic” RD %. This is what the exp % above was about.

Within an RD, you reference items from userconfig as %#id, as in %#obscore-extracolumns.

You can also define your own material in the userconfig RD. As an example, if you have a list of authors that publish a lot of data collections between them and you want to re-use that declaration in multiple RDs, you would put something like:

<STREAM id="thegang">
  <meta>
    creator.name: Author, F.
    creator.logo: http://example.org/logos/author_f.png
  </meta>
  <meta name="creator">Author, S.; Co-Author, N.; Ontributor, C.</meta>
</STREAM>

into your userconfig.rd and then, in each RD that describes work from them, just have:

<FEED source="%#thegang"/>

Choosing your Authority

In the VO, every publisher has something like a “namespace”, within which they are free to name things; that namespace is called your authority, it it needs to be gobally unique so VO identifiers are globally unique.

To pick one, think of a resonably short string defining your site; use dots to structure this as necessary. In Heidelberg, we are using org.gavo.dc, somewhat reflecting our DNS name. Today, I'd probably choose gavo.dc. Since astronomy is small enough, there is no need for deeply-nested hierarchy.

If you have picked a string, make sure it is not taken yet. A convenient way to do that is through WIRR. There, constrain Resource Type to Authority, then Add Constraint and constrain IVOID to the authority you have chosen (preconfigured WIRR query).

If nothing comes back, the authority is still free and you can enter it in your gavo.rc in the [ivoa]authority item, like this:

[ivoa]
authority: my-auth

Registering your Authority

The authority truly only belongs to you once you have added an authority record to the VO Registry. The chances that someone else will beat you to a non-trivial authority are slim, though, so do not worry if you have to delay a bit before going live with the following. It doesn't hurt to re-run the WIRR check before you actually go live, though.

You will first have do define your authority record. To do that, you have to edit your userconfig RD. See The Userconfig RD if you have not set one up yet.

Once you have your etc/userconfig.rd, look for the registry-interfacerecords stream. This contains two (relevant) elements: a resource record for claiming your authority, and the registry service, which is what the VO registry will later talk to.

While the latter usually does not manual intervention, in the authority, you must replace the the metaString macros defaulting to UNCONFIGURED (you could set the corresponding items in defaultmeta instead, but the additional indirection will not buy you anything), specifically:

• authority.title – A human-readable descriptor of what the authority corresponds to.
• authority.description – A sentence or two on what the authority you are using means. This could be the same as site.description if all you are claiming authority for is that; if you are claiming authority for your institute or organisation, this obviously should be different.
• referenceURL – A URL at which people can learn more about your data center.

As an example, the following could work:

<resRec id="authority">
  <meta>
    resType: authority
    creationDate: \metaString{authority.creationDate}
    title: The Utopia Observatory Data Center Authority
    shortName: \\metaString{authority.shortName}
    subject: Authority
    managingOrg: \\metaString{publisher}
    referenceURL: http://utopia.example.org
    identifier: ivo://\getConfig{ivoa}{authority}
    sets: ivo_managed
  </meta>
  <meta name="description">
    The Data Center at the Observatory of Utopia manages lots of
    substantial data sets created by the bright scientists all over
    the world.  All data collections published there are
    registered under this authority.
  </meta>
</resRec>

You can, of course, override any of what is filled through the remaining macros, too, but you should have a good reason to do so.

Once you have done that, you can proceed to Preparing your Publishing Registry to make the VO Registry see all this. Or you can keep customising your DaCHS installation first.

The Logo

The least you should do is replace the DaCHS logo in /var/gavo/web/nv_static/img/logo_medium.png with something pertaining to you. Scale the bitmap to about 250 pixels width or so (it will typically be used at 100 pt in the CSS).

In general, anything you put into /var/gavo/web/nv_static will be visible underneath /static/ on the web. See also Customising the Web Interface.

The Root Page

When people dereference your data center URI, they will see the root.html template rendered. The default is a simple list of services published in the local set (see Registering Services and Data), together with site.description from your defaultmeta.

This may do when you plan to publish more than a few but less than many services.

If you only have some very few services that furthermore will rarely, if ever, change, it is probably perferable to just write a static root page in plain XHTML in /var/gavo/web/templates/root.html (use nv_static from ancillary material as mentioned in The Logo).

If you expect to have many services, you should probably derive your root page from the root page that we use in Heidelberg; more on all this in templating.html#the-root-template.

Note that for DaCHS to notice you have overridden a built-in file, you must restart it once.

If you have special needs, there is also the [web]root configuration item. Set that to a relative path to replace the root service (which defines custom data functions such as titleList, chunkedServicesList and such used in the root templates) with a completely different service. Typical uses for this include pointing people directly to the service on single-service installations, or pointing to the ADQL form when what you are really doing is run a TAP-only system; in that latter case, you would say:

[web]
root: __system__/adql/query/form

Extra Sidebar Items

It is rather common to want to stick extra material into the sidebar. Overriding the sidebar template is a possibility, and it will probably survive several upgrades without breaking big time.

However, if it is just a few links or lines of text you want in there, there is the _sidebarlocal meta item that the default sidebar templates includes near its foot. There are the sidebarnote and sidebarmicro classes you can use make things fit; and, in order to pull raw HTML all the way through DaCHS's guts, you have to use the raw metadata format. Hence, in etc/defaultmeta.txt, you would put something like (see Stream Metadata for details):

_sidebarlocal:  raw:<div class="sidebarnote">\
    <a href="/adql">Try ADQL</a> to query our data.</div>

Note that you can easily ruin the well-formedness of your document in this way; the material is really included in the HTML without any adaptation (except UTF-8 encoding).

A recommended use for this is to link to a privacy policy. DaCHS tries hard not to store any data that could fall under the GDPR or similar legislation, but it's still nice to tell that to people; if you do process personal data, you must have some declaration like that in most juristictions anyway.

To have a privacy policy, go to /var/gavo/web/nv_static and put the following data into privacy.shtml:

<html xmlns="http://www.w3.org/1999/xhtml"
  xmlns:n="http://nevow.com/ns/nevow/0.1">
<head n:render="commonhead">
<title>Privacy Policy</title>
</head>
<body n:render="withsidebar">
  <h1>Privacy Policy</h1>
  <p>It's nothing personal.  Really.  In particular, while we do
  store our logs for a year, we never keep the source IPs.</p>
</body>

You will see the result at http://localhost:8080/static/privacy.shtml.

To have this linked from your sidebar, just add the following to your default meta:

_sidebarlocal: raw:<div class="sidebarmicro">\
    <p><a href="/static/doc/privpol.shtml">Privacy</a></p>
  </div>

The somewhat odd extension .shtml for DaCHS means that the file will be interpreted as a template over the service //services#root, which means that you can use the usual render functions and data items as per templating.html. This is be an easy way to have your page fit into the look of the remaining site.

You could also use plain .html, in which case you could even escape the necessity to provide well-formed XHTML.

Operator CSS

To override css rules we distribute or add new rules, avoid changing or overriding gavo_dc.css, as that will be a liability when upgrading. Instead, drop a CSS file somewhere (recommended location: /var/gavo/web/nv_static/user.css) and add a configuration item in [web]operatorCSS. With the recommended location, this would work out to be:

[web]
operatorCSS: /static/user.css

in /etc/gavo.rc.

This can also be an external URL, but we recommend against that, as that would force a browser to open one external connection per web page delivered.

By far the most common complaint is that we are limiting the width of p and li elements to 40em. We believe that text lines much longer than that are hard to read and should be avoided. On pages with tables where users might actually want their browsers to fill the entire screen, this choice cannot be made through a sensible choice of the width of the user agent window but requires CSS intervention.

Having said that, if you really think you want window-filling text lines, just put:

p, li {
  max-width: none;
}

into your operator CSS.

Overiding Built-in Web Resources

As mentioned in The Logo, you can put files and directories into /var/gavo/web/nv_static and have them show up on the web under /static. These files will override built-in resources of DaCHS if they overwrite their names. This might be intersting if you would like to fix a bug or try something out.

The originals sit in DaCHS' resource tree under web. Hence, if you want a local help file, in nv_static say:

dachs adm dump web/help.shtml > help.shtml

DaCHS decides between built-in and overridden when a resource is first accessed. Hence, when you first override a given resource, you have to restart the server. After that, DaCHS will pick up updates by itself.

Here are some built-in resources that you might want to override:

• help.shtml – the help file. Unfortunately, we blurb quite a lot about GAVO in there right now. We'll think of something more parametrisable, but meanwhile you may want to have your own version. This changes rarely, so it probably won't hurt to just fork it.
• xsl/dachs-xsl-config.xsl – this lets you customise (somewhat) the responses of various pages that are just XML with XSLT when viewed through the browser. This includes the OAI-PMH pages, UWS management, and VOSI responses.
• css/gavo_dc.css – the central CSS; only override this during bug fixing or similar. For anything more permanent, use Operator CSS.
• js/gavo.js – this is most of the custom javascript that DaCHS uses here and there. Again, only override this for debugging (in which case you will always want to set [web]jsSource to True to keep DaCHS from minifying your code). If you need custom javascript globally on a permanent basis, let us know and we'll do something similar to the operator CSS.

On the meaning of the .shtml extension, see the end of Extra Sidebar Items.

Overridden System RDs

You can also dump system RDs into /var/gavo/inputs/__system__ and edit them there; after a restart, DaCHS will use your copy then. Doing this is even more of a liability for upgrades than overriding the CSS or the Javascript. Only do that when debugging or to fix a transient bug until a fixed package is released.

Register Through purx

purx is a little service run by GAVO that lets people put arbitrary (but valid) VOResource records on some web server and have this service talk to the VO Registry to get them in there. Purx will pick up changes to the records, polling them roughtly daily, and propagate their contents to the VO Registry as necessary.

To use purx, get the URL of your service's VOResource XML as described above. Paste this URL into purx' enrollment field, wait for a mail to arrive, confirm the publication, and you are done. Your service will have a service identifier starting with ivo://purx, but the rest should contain material from your RD id.

To make purx stop publishing your record, you can take down the service and wait for purx to give up after a couple of months. A faster way would be to have the VOResource link return an HTTP 403 status code, which DaCHS so far cannot do (selectively). Let us know if you need it.

The good thing about purx is that once enrolled updates are automatic. Still, the extra HTTP-based polling increases latency and adds fragility. And you don't get to have your own authority in the identifiers, which sometimes is inconvenient (e.g., when Validating your Services).

Register through Web Forms

Both the EuroVO registry at ESAC and the NAVO registry at STScI let you register resources using a web browser and filling out web forms. When you are running DaCHS, this is not terribly attractive, because if you have followed Global Metadata for your RD and the first two sections of Configuring DaCHS, DaCHS already has essentially all information you would enter there.

Going through one of the web-based services still lets you offload authority management to them. And, at least on the EuroVO service, you can simply upload the XML DaCHS generates for you – see the “Create new Resource from File” (or, later, the “Edit Resource” tab in the details for a resource). The material to upload is again what the “VOResource XML” link on your service's info page points to. You will, however, probably have to edit the identifier.

Admin Web Interfaces

Some administrative operations on the data center can (or have to) be done from its web interface. To use these features, you have to set The Admin Password. You can then use the “Log in” link in the side bar using gavoadmin as the user name.

If you are logged in as gavoadmin, you should see an “Admin me”-link in the side bar of services. The page behind that link lets you block all services on the respective RD – where blocking means all requests are rejected until the RD is reloaded – and reload the RD (this is equivalent to running dachs serve exp).

In the form, you can also set scheduled down times. This is for VOSI, an interface clients could use to figure out whether a service can reasonable be expected to work. Since there don't seem to be clients exploiting the VOSI endpoints for such purposes so far, you probably don't need to bother.

You can directly access the administration panel for an RD by accessing /seffe/<rdId>, e.g , seffe/__system__/services.

There are several more or less introspective resources within DaCHS that do not need authentication. Among those, there you should know /browse. This gives a list of all RDs that have (ivo or local) published services or data in them. Links on the RD ids lead to info pages on the RDs, in particular giving tables and services within the RD.

Admin CLI Interfaces

You can also perform various housekeeping operations using gavo admin. Try gavo admin --help. This includes User/Group Management, precomputing automatic previews, and creating registry records for deleted services that got lost.

Sometimes people execute stupid TAP queries on your machine (e.g., constraining ra and dec in separate intervals, which is both slow and physically at least doubtful) and you would like to give them a friendly nudge on how to do it better. This is when gavo admin tapabort is your friend.

Call it with a TAP job id (which you can obtain from /tap/async unless people authenticated; if they did authenticate, you will have to check the tap_schema.tapjobs table directly in postgres) and a helpful (!) text to abort a TAP job and set it to an error state giving a short explanation what happened and the helpful text.

Letsencrypt

To get a certificate that is (hopefully) widely accepted, we strongly recommend you use letsencrypt. DaCHS has built-in support to update these certificates in time.

Here's how to go about it:

  # DaCHS internally calls a tool called acme-tiny; we much rather trust
  # it than any crypto code we'd hack together
$ sudo apt install acme-tiny
  # now create the directory with the key material; we'll do all of this
  # as root; if you don't like this, at least make it a user distinct
  # from gavo and gavoadmin
$ sudo bash
# cd `dachs config rootDir`
# mkdir hazmat
# chmod 700 hazmat
# cd hazmat
  # generate an account key; this is what identifies you against
  # letsencrypt
# openssl genrsa 4096 > account.key
  # Generate your secret key; these are the holy bits if you believe
  # in https.
# openssl genrsa 4096 > server.key
  # now tell dachs to have the certificate signed; warning: this will
  # try to restart the server
# dachs serve updateCertificate
  # Done -- hit ^D to exit the root shell.

Letsencrypt certificates are only good for three months. It's a good idea to renew them every two months. And that won't fly if you don't automate it. Automating it means your server will restart without manual intervention. That's bad because it might be down for a significant time (namely if someone is doing a large download at that time). As said below: HTTPS is an operational liability.

Anyway, add the following to root's crontab (if you installed DaCHS from source, you'll probably have to use the full path to /usr/local/bin/dachs here):

21 4 1 1,3,5,7,9,11 * dachs serve updateCertificate

-- adapt the time (here, 4:21 local time) to avoid times when it's likely you'll have many users. Also, make sure you receive mails from cron from the machine that does this, because there's a lot that can go wrong here. When updating keys, DaCHS will just write error messages to stderr for cron to transport them. That's probably not a big additional liability, as you should teach the DaCHS server to send mails also for several other DaCHS subsystems.

Finally, to register your https endpoints, add:

registerAlternative: True

to the ivoa section in your /etc/gavo.rc. To propagate the information, restart the server and then say dachs pub -a to make the registries re-harvest your site.

If your machine is reachable through different host names (our box in Heidelberg, for instance, is both dc.g-vo.org and dc.zah.uni-heidelberg.de), tell DaCHS about it by listing all the non-primary names (i.e., anything that's not in serverURL) in a comma-separated list in [web]alternateHostnames.

HTTP preferred

We strongly recommend to have an unencrypted HTTP endpoint, and have that as the primary interface even when you support HTTPS unless you actually have authenticated services.

In particular, please to not forcibly redirect to HTTPS versions of pages; it breaks POST requests (which are quite common in the VO) and quite a few other things, too.

DaCHS will give you the same effect in a sane way because (since version 2.2.3) it honours the upgrade-insecure-requests header. That way, clients that are confident they can deal with HTTPS can inform the server, and if the server thinks it's safe to upgrade, it can redirect them. Anything not prepared to deal with or interested in HTTPS will be unconcerned, and everything is under user control, as it should be.

Here's the reasoning for preferring HTTP in the VO: HTTPS is largely ineffective against most privacy breaches: Almost all nation-states can issue certificates that almost all user systems will accept; companies or institutions wanting to eavesdrop on their employees can force http proxies and install their own CA certificates on their employee's machines; and of course most privacy violations are side effects of platforms executing loads of Javascript sailing into the browsers with valid certificates.

On the other hand, networks become a lot more brittle with HTTPS: On the server side, the certificates need to be managed and regularly refreshed. And that's even before operators employ secure key management (which would probably require extra hardware or at least the use of passphrases).

Worse, the client side needs to keep their CA bundles (that's essentially lists of private keys they trust) up to date. With browsers that often carry these along and are updated quite regularly on most boxes, that's marginally manageable. For the VO, clients as a rule are not browsers but tools like TOPCAT or Aladin that may use a CA bundle coming with the Java VM, which often is not as well maintained. And once people start using curl or pyVO, the operating system's CA bundle is used, which on many systems is a mess. The net effect is that a given service may appear to work in the browser and in a script, but not from TOPCAT. Oh my.

Given the miniscule benefits and the serious operational implications, you should provide HTTP endpoints and register your those. If HTTPS is available, DaCHS will tell clients via the Registry's mirrorURL feature. If clients think HTTPS is worth it, they in this way can learn about support for it and use that as they see fit; and for browsers, see above on upgrade-insecure-requests.

Upgrading DaCHS

In general, we try to make upgrades painless, but with a system allowing people to play tricks with intestines like DaCHS guarantees are hard.

If you are running DaCHS for the longer term (i.e., more than 2 year so or), be sure to subscribe to DaCHS-users. We announce new releases there, together with brief release notes pointing to possible spots of trouble.

Ideally, you will have a development system and regression tests in place that let you diagnose problems before going to production.

$ dachs val -c ALL
$ sudo service dachs stop # (or whatever you use to stop the server)
$ dachs upgrade
$ sudo service dachs start # (or whatever you use to start the server)
$ dachs test ALL

$ cd <checkout dir>
$ svn update
<run the restart sequence given above>

$ cd <checkout dir>
$ svn update
$ sudo python setup.py install
<run the restart sequence given above>

Upgrading Postgres

Postgres' on-disk formats change from version to version. This makes upgrades a bit of an adventure, in particular if you have large databases that take days to export and ingest. However, Debian helps a bit; even if you don't run Debian, the following might be helpful.

The text assumes you are using the default cluster, which is called main. If – and that's likely if you installed DaCHS in the olden days – yours has a different name (we used to suggest “pgdata”), you will have to replace “main” with that name throughout. In case you do not remember, at least in Debian you can just ls /etc/postgresql/<pgversion> – what you see there is the name of your cluster.

As it is easy to destroy all your tables at once if you mistype things here, it is wise to dump your database somewhere before starting with this:

pg_dump -f (some descriptive name).pgdump -F c gavo

Let's say you are upgrading from 11 to 12. Since we will need these numbers quite a few times, set them in variables – but be very careful, if you get this wrong, the commands further down will nuke your database:

export OLDVER=11
export NEWVER=12

To upgrade, first install the new engine and extensions, for instance:

apt-get install postgresql-${NEWVER}-pgsphere postgresql-${NEWVER}-q3c postgresql-${NEWVER}

If you use other extensions in your database (e.g., postgis), also install them now. To see what extensions you have installed, a command like:

dpkg-query -W "postgresql-${OLDVER}-*"

is probably helpful. Other versioned packages you may want to install include:

• postgresql-server-dev-$NEWVER if you use locally-built extensions. Rebuild and install those extensions before proceeding with the upgrade.
• postgresql-plpython-$NEWVER if you use python DB extensions
• postgresql-contrib-$NEWVER for several interesting index schemes and the like.

You will probably first have to drop the new cluster the Debian maintainer scripts already have created, as pg_upgradecluster wants a clean slate:

pg_ctlcluster $NEWVER main stop
# this is a good time to see if your data center still works; if not,
# you stopped the currently active cluster and should *not* drop it
pg_dropcluster $NEWVER main

If these give errors to the effect that the “specified cluster does not exist”, that's fine.

During the database update, postgres will be down, and hence your data center will not work. For a large database and a complicated upgrade, the update can take several hours (though, really, the last few upgrades for us were minute jobs although we have about 5 TB of tables). You should therefore let people know what is going on. As long as nobody really does anything with VOSI availability, the next best thing is to put your DaCHS installation in panic mode.

To do this, place a file called MAINT into DaCHS's stateDir, maybe somewhat like this:

cat <<EOF > `dachs config stateDir`/MAINT
We're currently upgrading our database and cannot properly respond to
requests.

We expect to be back online late afternoon UTC, 2021-08-11.
EOF

Then say:

pg_upgradecluster -k -m upgrade -v $NEWVER $OLDVER main /var/lib/postgresql/$NEWVER

You may need to change the last argument, depending on where your database files actually are. You can figure out where the old ones were using:

grep data_directory /etc/postgresql/$OLDVER/main/postgresql.conf

Do not use that old directory for the new cluster.

Note the -k option to the above command: this makes pg_upgrade use hard links instead of copies. This saves a lot of space and time. The downside is that once the upgrade has succeeded, the old server may go haywire because its files have been changed to fit the new version. In our experience, that's a price worth paying.

If this fails, do not panic. Most of the time it's some extension you forgot to install, and once you've done that, everything will be fine. The upgrade script points you to the logs of the operation, which should let you figure out what to do.

When all is done, pg_upgradecluster configures the old cluster to no longer start automatically and the new cluster to listen where the old one was.

You can now probably run your services again, so, remove MAINT in the stateDir, restart the DaCHS server and run your regression tests (you wrote some, didn't you?).

However, quite typically the table stats are wrong or ununderstandable to the new version, so for large tables you may see timeouts and the like. Starting around version 10, pg_upgradecluster helpfully provides a script that does all that's required to update the stats and make things fast again. It is kept together with the upgrade log which you will find by going to /var/log/postgresql, saying ls -lr --sort=time. The last directory shown ought to be your upgrade directory.

In there, start a screen (the script can take a long while, and it's good if you can detach the screen with ^ADD and it keeps running while you are going home) and run it like this:

sudo -u postgres sh ./analyze_new_cluster.sh

Finally, pg_upgrade has printed a command that will remove the old cluster; execute it, and finally deinstall the old postgres packages to avoid confusion – locate them somehow like:

dpkg -l | grep "postgres.*$OLDVER"

When that's all done, take a deep breath and relax.

pg_restore: [archiver (db)] could not execute query:
        ERROR:  argument of negator must be a name
    Command was: CREATE OPERATOR @ (
    PROCEDURE = "sline_contains_point_com",
    LEFTARG = "spoint",
    RIGHTARG = "sline",
    COMMUTAT...

UPDATE pg_operator AS o
SET oprcom=0
WHERE
  NOT EXISTS (
    SELECT 1 FROM pg_operator AS i
      WHERE i.oid=o.oprcom)
  AND oprcom!=0

UPDATE pg_operator AS o
SET oprnegate=0
WHERE
  NOT EXISTS (
    SELECT 1 FROM pg_operator AS i
      WHERE i.oid=o.oprnegate)
  AND oprnegate!=0

$ ls
PG_9.4_201409291/  PG_9.6_201608131/

Authors, or: Nested Sequential Meta

If you read RMI-Style Metadata in the reference, you will notice that the real meta key for the authors of the data is creator.name (please pay particular attention to what we are saying there about the format of that string). However, in the templates and our examples, you will see we are using creator throughout. The reason for this is that creator is complex metadata (there's name, logo, and possibly more), and setting multiple pieces of complex metadata according to the rules discussed in Defining Metadata has a nasty pitfall with these.

Since this pitfall probably will not bite you outside of creator, you might decide to skip this and just never write creator.name in your RDs. But then it is safer to understand the issue and read on.

So, suppose you had more than one creator. What you then want is a metadata structure like this:

+-- creator -- name (Arthur)
|
+-- creator -- name (Berta)

However, if you write:

creator.name: Arthur
creator.name: Berta

or, equivalently:

<meta name="creator.name">Arthur</meta>
<meta name="creator.name">Berta</meta>

by rules of Defining Metadata, you will get this:

+-- creator -- name (Arthur)
       |
       +------ name (Berta)

i.e., one creator with two names, which is both invalid and wrong.

To avoid this, you have to insert a new creator node in between, and write:

creator.name: Arthur
creator:
creator.name: Berta

This yields the upper (correct) meta structure.

Since the creator.name case is so common, DaCHS provides a shortcut, which you should simply always use to define the creators: if you set creator directly, DaCHS will expect a string of the form:

<author1>, <inits1> {; <authorn>, <initsn>}

(i.e., Last, I.-form separated by semicolons, as in “Foo, X.; Bar, Q.; et al”) and split it up into the proper structure. You can mix the two notations, for instance if you want to set a logo on the first creator:

<meta name="creator">
  <meta name="name">Chandrasekhar, S.</meta>
  <meta name="logo">http://sit.in/chandra.png</meta>
</meta>
<meta name="creator">Copernicus, N.; Gallilei, G.</meta>

Table Notes

Frequently, you need to say more about a column than is appropriate in the few-phrase description. In catalogue descriptions and VizieR, such situations are handled using notes, and DaCHS follows suit.

The notes themselves are kept in meta elements belonging to tables. Since the notes tend to be markup-heavy, their default format is reStructuredText. When entering notes in RDs, there is an attribute tag on these meta items:

<table id="demo">
  ...
  <meta name="note" tag="1">
    The meaning of the flag is as follows:

    =====  ==========
    value  meaning
    =====  ==========
    1      value is 2
    2      value is 1
    =====  ==========
 </meta>

 <meta name="note" tag="2">
 ...
</table>

To assoicate a column with a note, use the column's note attribute:

<colum name="crazyflag" type="smallint" ... note="1"/>

As tag, you may use basically any string, but it's a good idea to keep it to numbers or at least characters not requiring URL encoding.

The notes will be present in HTML table heads, table and service descriptions, etc. If you need to link to one, there is the built-in tablenote renderer that takes the table and the note from its query path. The most convenient way to is it is through the built-in vanity name tablenote, where you would access the note above using a URL like http://your.server/tablenote/demoschema.demo/1.

Space-Time Metadata

As soon as you have coordinates, you will want to declare coordinate metadata on them, i.e., reference frames, roles played by columns (say, x is the time derivative of y, and x1 is a galactic latitude), the position for which times and positions are given for, and so on. Collectively, this is known as declaring space-time coordinates or STC for short.

The problem here is that this got a bad start in the VO and went downhill from there. And that is why there is, to this day, no real model for how to express these things.

DaCHS has been using a language called STC-S to let operators declare such metadata. It is clear that we will have to change this at some point, and for time metadata, we are already using something else that will probably be roughly stable. Apologies for this state of affairs. We've tried to fix it, but so far without much success.

<table id="withcoo">
  <stc>
    Position ICRS "ra" "dec" Error "e_ra" "e_dec"
  </stc>
  <stc>
    Position FK4 Epoch J1950.0 "ra_orig" "dec_orig"
  </stc>

  <column name="ra" unit=...
  <column name="dec" ...
  ...
</table>

<stc>
  Position ICRS Epoch J2015.0 "ra" "dec"
      Error "ra_error" "dec_error"
    Velocity "pmra" "pmde" Error "sigpmr" "sigpmd"
</stc>
<stc>
  Position ICRS Epoch "epu" "ra_ucac" "de_ucac"
    Velocity "pmra" "pmde" Error "sigpmr" "sigpmd"
</stc>

<stc>
  Position Galactic "glon" "glat"
</stc>

<stc>
  Polygon ICRS [s_region]
</stc>

<stc>
  PositionInterval ICRS [bbox]
</stc>

<stc>
  Position J2000 "raj2000" "dej2000"
  Redshift OPTICAL "redshift"
</stc>

<stc>
  Time TT 2000-01-01T00:00:00
  Position ICRS Epoch J2000 "alphaFloat" "deltaFloat"
    Error "RAErr" "DecErr"
    Velocity "pmRA" "pmDE"
      Error "PMRAErr" "PMDEErr"
</stc>

<stc>
  Time TT "detection_time"
  Position ICRS "raj2000" "dej2000"
  Spectral "energy_cor" unit keV
</stc>

<stc>
  TimeInterval UTC TOPOCENTER "time_min" "time_max"
</stc>


<stc>
  Position ICRS SPHER3 Epoch J2010 "alpha" "delta" "distance"
  Velocity "mualpha" "mudelta" "radialvelocity"
</stc>

<stc>
  Position ICRS SPHER3 "raj2000" "dej2000" "dist"
    Size "rcluster" "rcluster" -1
    Velocity "pmra" "pmde" "rv" Error "e_pm" "e_pm" "e_rv"
</stc>

<dm id="obs_pos">
  (stc2:Coords) {
    time: (stc2:TimeCoordinate) {
      frame:
        (stc2:TimeFrame) {
          timescale: TCB
          refPosition: BARYCENTER
          time0: 0 }
      location: @obs_time
    }
  }
</dm>

>>> from gavo import api
>>> import datetime
>>> api.dateTimeToJdn(datetime.datetime(1970, 1, 1))
2440587.5

Defining Views

It is quite regularly preferable to represent data in multiple tables (for instance, for normal form reasons); however, outside of TAP you cannot expose such table structures in VO protocols unless you re-join the various tables to some de-normalised table or, in SQL terms, a view. Other reasons to define views include the mapping of nonstandard data to standard table structures, which is what DaCHS does internally for SSAP tables and when publishing anything through obscore.

If you want to manually define a view, declare a table as usual and then give it a viewStatement – done. In practice, there are a few things you should keep in mind to make such a view definition maintainable.

First, you will obviously define the tables involved, for instance:

<table onDisk="True" id="master" mixin="//scs#q3cindex"
    primary="catno" adql="True">
  <column name="catno" type="integer" required="True"
    ucd="meta.id;meta.main"
    tablehead="id#"
    description="Identification number in the ARIGFH master catalogue"
    verbLevel="1"/>
  <column name="raj2000" type="double precision"
  ...

<table onDisk="True" id="identified" adql="True">
  <column name="dist" type="double precision"
    ucd="pos.angDistance" unit="deg"
    tablehead="Offset"
    description="Offset between master catalogue position at catalogue
      epoch and equinox and the catalogue position"
    verbLevel="1" displayHint="displayUnit=mas,sf=1"/>
  <column name="masterNo" type="integer" required="True"
    ...

Note, in particular, that postgres does not let you define indexes over views. This means that anything you want to index must be indexed on the table level (hence the //scs#q3cindex mixin on the master table). You still should declare the underlying indexes; you could write index statements as usual (add metaOnly="True"), but it is generally faster and safer to use //procs#declare-indexes-from. Insert it right above your viewDefinition.

A good approach is to keep the columns that will later show up in the view into a STREAM and then replay that stream in both the table definitions and the view definition. This is used in, for instance gedr3mock/q. An alternative that lets you keep the table definition as-is is to go through original; this is what we do here, inspired by arigfh/q.

One reason we are going for original rather than STREAM is that with original, we can straightforwardly rename some columns, which here we want because in the view, we want to mark columns coming from the master table, like this:

<table onDisk="true" id="id" adql="True">
  <meta name="description">
    The stars from the gfh table having counterparts in the master
    catalogue, together with those counterparts.
  </meta>
  <column original="master.catno" name="master_no"/>
  <column original="master.component" name="comp_master"/>
  <column original="master.raj2000"/>
  <column original="master.dej2000"/>
  <column original="master.pmra" name="pmra_master"/>
  <column original="master.pmde" name="pmde_master"/>
  <column original="master.mv" name="mv_master"/>
  <column original="master.mb" name="mb_master"/>

When just copying columns, active tags come in handy to copy over column definitions from the two tables; note the use of <tablename>.<columnname> in referencing:

<LOOP listItems="catid catan dist iq">
  <events>
    <column original="identified.\item"/>
  </events>
</LOOP>

<LOOP>
  <codeItems>
    for col in context.getById("gfh"):
      yield {'item': col.name}
  </codeItems>
  <events>
    <column original="gfh.\item"/>
  </events>
</LOOP>

As shown in the second loop, you can go as far as to programmatically obtain the column names for your source tables. Only do that if you are sure you actually want to follow changes in source table structure in the view.

After declaring the columns, what is left is actually defining the view (see above for the FEED):

  <FEED source="//procs#declare-indexes-for"
    sourceTables="master gfh"/>

  <viewStatement>
    CREATE VIEW \curtable AS (
      SELECT \colNames FROM
        (SELECT catno, raj2000, dej2000,
          pmra AS pmraMaster,
          pmde AS pmdeMaster,
          mv AS mvMaster,
          mb AS mbMaster,
          component AS compMaster FROM \schema.master) AS m
      JOIN
        \schema.identified AS idf
      ON (masterNo=catno)
      JOIN \schema.gfh
      USING (catid, catan))
  </viewStatement>
</table>

The viewStatement essentially is more or less arbitrary SQL. For robustness against changes of table structure or schema or table name changes (as in: you should get errors quickly if things are broken), however, you should probably always start with:

CREATE VIEW \curtable AS (
  SELECT \colNames FROM

The macro \curtable will automatically adjust to whatever schema and table name is given in the RD, and \colNames gives all the names of the columns; using it, you will get errors instead of silent failures if you add or remove columns in the table definition but fail to adjust the view statement.

When making these tables, it pays to be a bit careful with the data elements, as of course the view depends on the source tables. In particular, when you re-import one of the source tables, the view will get dropped, and it is a good idea to tell DaCHS to automatically re-make it. The recommended setup for this looks like this:

<data id="import_master" recreateAfter="gfhtables">
  ...
  <make table="master"...></data>

<data id="import_identified" recreateAfter="gfhtables">
  ...
  <make table="identified"...></data>


<data id="gfhtables" auto="False">
  <make table="id"/>
</data>

– essentially, you make the view in a non-auto data which gets imported every time the source tables are re-made. Note that making the data for one source table when the other doesn't exist and will not be made in the same import will lead to an error in the view creation. This is harmless for the import of the table you imported, as data creation on recreateAfter happens in a separate transaction.

Also note that dachs limits, when run on an RD, will ignore views, partly because their columns will typcially already have statistics in the originating tables, partly because Postgres cannot TABLESAMPLE views, which may make column statistics really expensive. If your views contain, for instance, unions, that will yield table metadata that is seriously wrong. While you can update the statistics by passing the identifier of the table descriptor into a dachs limits call, it is preferable to not this in the RD. For that, add:

<property name="forceStats">True</property>

to the table element describing the view.

Doing Incremental Imports

When running archives of ongoing projects, it is usually desirable to import often, but ignore items that are already in the database to make the imports quick and cheap.

The key ingredient to accomplish that is the Element ignoreSources, which lets you tell DaCHS which files of those it found in sources it should not ingest (again). What this deals with are paths relative to DaCHS' inputsDir. Unless you did something special, the accref column in tables having data products contains exactly these inputsDir-relative paths.

There are various ways to communicate accrefs to ignore: from lines in a file, by a pattern, or from the database. This last case is what most commonly is used in incremental imports, in the form of the fromdb attribute of ignoreSources.

DaCHS, however, when you import something, will always tear down all tables being made, so just ignoring some sources will not be sufficient. To make DaCHS keep tables mentioned in make, give the Element data an update attribute.

In sum, the pattern is:

<data id="import" updating="True">
  <sources pattern="data/*.fits">
    <ignoreSources fromdb="select accref from myschema.mytable"/>
  </sources>...

Of course, you have to adapt the source pattern and the table to get the accrefs from.

For non-data product sources (e.g., files you feed into catalogs), you will have to use some other technique, because these do not have accrefs and you will have to manage the sources already processed manually. An example for how this can be done is the rr.imported table in rr/q.

Even more on this is found in the reference documentation's Updating Data Descriptors.

Creating pgSphere Geometries

DaCHS lets you store spherical geometries (polygons, circles, and MOCs) in database columns. Obscore and EPN-TAP have the s_region column holding such geometries, and you can have custom columns with geometry types that facilitate writing interesting TAP queries. Also, DaCHS is moving towards using pgSphere's spoint for spatial indices (instead of q3c, which has been the preferred spatial index until DaCHS2).

To feed such columns, you first have to choose a type for the column; DaCHS' geometry columns are type-clean, i.e., if a column has a polygon in one row, it all other rows will have to have a polygon (or NULL) there, too – rather than, say, a circle or a point.

The choice of types is:

• spoint – a simple point. You should not have spoints in s_region, as common queries will run contains or intersects against them, and that rarely does what users expect with points.
• scircle – a spherical circle. pgSphere limits those to radii smaller than pi/2.
• spoly – a spherical polygon, defined through a number of edges. pgSphere currently has rather severe limitations on what these can be; in particular, you cannot have intersecting edges, and their maximum size has limits, too. For complex geometries, prefer smoc.
• smoc – a Multi-Order Coverage map (MOC). MOCs are, conceptually, lists of HEALPixes and thus can represent arbitrarily areas on the sky. If you have possibly large, possibly non-convex, possibly non-simply connected areas, they are what you are looking for.

Creating values of these types in DaCHS would usually use classes from the gavo.utils.pgsphere module, which is available as pgsphere in rowmakers. These are usually constructed with arguments in rad; for this and other reasons, we recommend using their fromDALI class methods. These take the canonical representations forseen by DALI for VOTables, which are usually just arrays for floating point values in degrees.

So, to fill an spoint column, you would say something like:

<var key="center"
  >pgsphere.SPoint.fromDALI([
    float(@centerLong),
    float(@centerLat)])</var>

A circle with a radius given in arcminutes would be:

<var key="s_region"
  >pgsphere.SPoint.fromDALI([
    float(@ra), float(@dec),
    float(@radius)/60])</var>

A polygon is constructed with longitudes and latitudes interleaved:

<var key="s_region"
  >pgsphere.SPoly.fromDALI([@x1, @y1,
    @x2, @y2, @x3, @y3])</var>

The DALI representation of a MOC is a string of whitespace-separated order/pixel lists. The details are in section 2.3.2 of the MOC specification. There are various ways in which you can come up with such strings.

If you have a list of points, decide on the order you want your MOC to give the coverage on (6 is about one degree, and each order gives you a factor of two in linear resolution). Then, make circles of about that resolution around your points, turn these into circles and join the MOCs. For instance:

<apply name="compute_coverage">
  <setup>
    <bind name="order">10</bind>
  </setup>
  <code>
    circles = []
    for x, y in @points:
      circles.append(
        pgsphere.SCircle.fromDALI([x, y, 6/order]))

    m = circles.pop().asSMoc(order=order)
    for c in circles:
      m.moc += c.asSMoc(order=order)

    @s_region = m
  </code>
</apply>

This is based on the fact that a pgsphere.SMoc's moc attribute (currently) contains a pymoc.MOC instance; even if we change the backend implementation (which is likely), the pattern should keep working.

Similarly, for a bunch of polygons, you could do this:

<apply name = "compute_region">
  <setup>
    <bind name="order">10</bind>
  </setup>
  <code>
    @s_region = pgsphere.SPoly.fromDALI(polys[0]
      ).asSMoc(order)
    for vertices in polys[1:]:
      @s_region.moc += pgsphere.SPoly.fromDALI(vertices
        ).asSMoc(order)
   </code>
</apply>

Note that the HEALPix library DaCHS uses currently (2020) cannot deal with non-convex polygons and may slightly over- or underestimate the coverage of circles and polygons.

If either limitation bothers you: The CDS HEALPix library provides a nice alternative for which there is a convenient wrapper in CDS-HEALPix-python. If you produce an ASCII MOC from whatever you compute using these tools, you can pass them to pgsphere.SMoc.fromASCII. See also openngc/q for sample usage of the relatively naked rust HEALPix library.

Skipping Things

Sometimes there are a few “broken” things in material that is otherwise just fine: A couple of broken FITSes, or some lines of a CSV that should not make it into a database table.

While the -c flag to dachs import will let you conveniently ignore individual sources (note that this is rather certainly not what you want with the bad rows), it will also hide genuine errors and thus should only be used in development. In production, you should skip the bad sources or rows.

<fitsProdGrammar>
  <rowfilter>
    <code>
      if "_failed" in rowIter.sourceToken:
        raise SkipThis("Ignoring failed observation")
      yield row
    </code>
  </rowfilter>
  ...

<fitsProdGrammar>
  <rowfilter>
    <code>
      if not "CD1_1" in row:
        raise SkipThis("Ignoring uncalibrated observation")
      yield row
    ...

<fitsProdGrammar>
   <sourceFields>
      <code>
        try:
          _ = pyfits.open(
            os.path.join(
              base.getConfig("inputsDir"),
              sourceToken))
        except Exception as ex:
          raise SkipThis("Skipping utterly broken {}: {}".format(
            sourceToken, ex))
        return {}
      </code>
   </sourceFields>

Skipping Rows

To skip individual rows of a source, raise IgnoreThisRow in a procedure application, like this:

<rowmaker id="build_mytable" idmaps="*">
  <apply>
    <code>
      if int(@colX)>22:
        raise IgnoreThisRow()
    </code>
  </apply>
</rowmaker>

Note that if you raised SkipThis here, you would skip the entire rest of the source, which is probably not what you want.

Row makers also admit the Element ignoreOn mentioned in Skipping Sources; as for them, we believe the declarative means rather typically are are not flexible enough, and skipping using Python code is usually about as clear and evolvable.

Source Fields

It is a fairly common situation that you want to make available certain properties of a source file to the row makers; the most common case is that the data providers have encoded information like object names, times, or whatever, in file names, or that there is some auxiliary information in a different files.

In these situations, it is often efficient to use Element sourceFields. It contains a standard procedure definition (i.e., you could predefine those and bind parameters), but usually you will just fill in the code.

This code is called once for each source processed, and receives the sourceToken as argument. It must return a dictionary, the key/value pairs of which will be added to all rows returned by the row iterator.

In addition to the sourceToken, you also have access to the data that will be fed from the grammar. This can be used to, e.g., retrieve the resource directory (data.dd.rd.resdir) or data descriptor properties (data.dd.getProperty("whatever")).

As a trivial example, here is how to only parse a field identifier from a source name once per source (rather than have this expression in a row maker and to that once per row):

<!-- this sits in a grammar element -->
<sourceFields>
  <code>
    return {"field":
      int(sourceToken.split("bul_sc")[-1].split(".pm.gz")[0])}
  </code>
</sourceFields>

A more complex example, taken from lightmeter/q, looks up some calibration values from a different table based on an identifier again pulled from the source file – clearly, you wouldn't want to do one query per row when there's thousands of rows coming from one source:

<sourceFields>
  <code>
    stationId = os.path.split(os.path.split(sourceToken)[0])[1]
    if re.match("[0-9]+$", stationId):
      raise base.SkipThis("Archive directory")
    try:
      with base.getTableConn() as conn:
        stationPars = list(conn.query(
          "SELECT calibA, calibB, calibC, calibD, timeCorrection"
          " FROM lightmeter.stations"
          " WHERE stationId=%(stationId)s", locals()))[0]
      calibA, calibB, calibC, calibD, timeCorrection = stationPars
    except IndexError:
      raise base.ValidationError("No station data for %s.  Import"
        " stationsdata first."
        %stationId, "stationId", stationId)
    if calibA is None:
      raise base.SkipThis("No calibration for %s."%stationId)
    srcKey = "/".join(utils.getRelativePath(sourceToken,
      base.getConfig("inputsDir"), liberalChars=True).split("/")[2:])
    return locals()
  </code>
</sourceFields>

On getTableConn, see Database Queries.

Note, however, that you need a special pattern when you want to query the table that you are filling. This is because sourceFields run in the midst of a transaction updating the table. So, something like:

<code>
  <!-- will deadlock, don't do it like this -->
  with base.getTableConn() as conn:
    conn.queryToDicts("select ... from <table mentioned in make>")
</code>

will wait for the transaction to finish. But the transaction is waiting for data that will only come when the query finishes – this is a deadlock, and dachs imp will just sit there and wait (see also commonproblems.html#deadlocks).

To get around this, you need to query using the data's connection. So, instead write:

<code>
  yield data.connection.queryToDicts(...)
</code>

preFilters

DaCHS Grammars parsing from files have a preFilter attribute. This can contain a pipeline that has to read from stdin and produce its output on stdout.

The typical use case is to decompress input files without bothering DaCHS, e.g., using preFilter="zcat (for which there's the historical gunzip="True" alias – don't use that any more), preFilter="bzcat", or preFilter="xzcat".

However, if you have some weird format for which there is a dumping tool that produces something that can be turned into CSV from a stream coming in from stdin something like the following provides an elegant way to get the data in with minimal code:

<csvGrammar preFilter="dump-weird-format-to-stuff | stuff-to-csv"/>

reGrammars

The Element reGrammar is the swiss army knife of text-parsing grammars when neither element columnGrammar nor csvGrammars will do. The idea here is that you give one regular expression each to separate the file into records and to split the the records into fields, and that you simply enumerate the names used in the mapping.

In the simplest case – whitespace separated columns in lines containing no other whitespace, with one comment line at the top –, such a grammar could be specified like this:

<reGrammar topIgnoredLines="1">
  <names>raMin, raMax, decMin, decMax, EVI, AV, AI</names>
</reGrammar>

reGrammars have a few tricks built-in to make them a bit more versatile. The following lets you simulate properly parsing some bzipped database dump by making some strong assumptions:

<reGrammar topIgnoredLines="15" preFilter="bzcat">
   <fieldSep>"?,(\s*")?</fieldSep>
   <recordCleaner>\((.*)\)</recordCleaner>
   <names>primflag, measure_no, paper_id, source_no, source_name</names>
</reGrammar>

The recordCleaner used here is an RE matched against every record and that contains exactly one capturing group. If it matches, the group will be used to apply the separators; in this case, it simply discards the parentheses you have in your typical database dump. Records not matching at all are discarded (which, admittedly, makes it easy to miss errors in ways that will lose data silently).

An alternative, and somewhat safer, way to get rid of junk in the input is to use the commentPat attribute. That is an RE that is removed from record before trying to parse it. The reference documentation mentions the classic case: (?m)^#.*$ will throw away lines starting with a hash.

Another gung-ho feature of reGrammars is the lax attribute. Normally, reGrammars will fail on records that do not have exactly one field per name you put in. Some ugly input may skip “optional” fields at the end – with lax="True", DaCHS will map those to empty strings.

Finally, sometimes files have junk at the end. This is when stopPat comes in handy: When a record matches that RE, the entire remainder of the input is ignored.

fitsProdGrammars

This grammar exposes FITS headers as rawdicts. Since both represent essentially the same data structure – sequences of key-value pairs, you can get away with just:

<fitsProdGrammar/>

– and that would cover a lot of use cases that read FITS images.

If you have FITS tables, this grammar is not for you; use Element fitsTableGrammar instead, and if you have large FITS tables, have a look at Grammars in C (even if you don't know C).

DaCHS uses pyfits to parse the headers and thus supports all the major conventions (CONTINUE cards are transparent, as are HIERARCH-style long keywords). Neither HISTORY nor COMMENT cards are present in the rawdicts.

There are some snags, anyway. For example, DaCHS by default reads the header bytes using specialised code that is somewhat more robust and has more desirable behaviour with odd files than the pyfits one (you can deselect it by setting qnd="False" if necessary). This gives up collecting header blocks if it has not found the end card after maxHeaderBlocks blocks. The default maxHeaderBlocks is chosen to be 40, which is reasonable for reasonable FITSes. However, we have seen in the wild massive abuse of comment cards to hold entire tables. If you're unlucky enough to have to handle such files, you may have to raise this.

You also have to use qnd="False" if you parse from compressed FITS images.

It is fairly common for FITS keywords to contain a dash (-). Since rawdict keys are supposed to be python identifier (e.g., for @-referencing), fitsProdGrammars translate these to underscores. If further cleanup is necessary, there is Element mapKeys that lets you write things like:

<mapKeys>
  <map key="properName">PROP NAM</map>
</mapKeys>

The element should only be used to fix crazy names. Actual mapping of names should be performed in rowmakers, because that's where you will look for them in two months.

FITS files are somewhat more complex, and fitsProdGrammars expose some of this. If you need to parse from a header other than the primary one, give the 0-base extension number in hdu attribute. For the full power of pyfits (but also all incompatibilities that are introduced by using that power), there is the hdusField attribute. This gives the key under which the pyfits HDU is visible in the rawdict. Be warned that using this might make your grammar dramatically slower, in particular when it operates on gzipped FITS files (which are, incidentally, transparently supported if their file names end in “.gz”).

csvGrammars

csvGrammar parses from files containing comma separated values. It actually is a thin wrapper around python's csv module, and thus it is fairly forgiving about the idiosyncrasies of CSV (e.g., quotes around all, some, or no values). You can configure the delimiter character via the same-named attribute (it defaults to, expectably, comma).

CsvGrammars by default expect input files to follow the convention that the first row contains the column names, and it will use whatever is there as rawdict keys. If these names are not usable as Python identifiers, use Element mapKeys to map them to other keys so you can have nice @-references in your row maker.

If the first CSV line does not contain the names, you must use the names child to give these names, like this:

<csvGrammar names="ra, dec, mag"/>

The csvGrammar will not complain if there are more fields in the file than you give names for; any extra values are reachable through @NOTASSIGNED (which should probably only be used to catch exceptional circumstances).

If your input files follow the (silly) convention of having the colum names in a comment in the first line, like this:

# ra, dec, mag
13,2, 14.45, -0.33

you have to give names as above and make the csvGrammar ignore the first line with topIgnoredLines="1".

There is currently no way to ignore comments in any other way; if you need that, you should probably retrofit that functionality in Python's csv module.

Python-defined grammars

Sometimes inputs are so nasty that DaCHS' grammars really can't be used to parse them. For these situations, DaCHS has Element customGrammar and Element embeddedGrammar that let you just slap in pieces of Python yielding dictionaries as grammars.

First off: before you go there, think hard if one of DaCHS' built-in grammars really can't do the job (including Element binaryGrammar). Long experience has shown that custom code is much more likely to break in the evolution of your system, Python, and DaCHS itself.

Custom grammars come in modules of their own, which is a good thing because you can test them outside of DaCHS. But they break the principle of “all you need is in the RD”, so don't use them unless you must. An introduction to them is in Writing Custom Grammars in the reference documentation.

Embedded grammars are somewhat more light-weight. They in particular come in handy when you need to generate a few rows, like this:

<embeddedGrammar>
  <iterator>
    <setup>
      <par name="testData">"a"*4</par>
    </setup>
    <code>
      for i in self.sourceToken:
        yield {'index': i, 'data': testData}
    </code>
  </iterator>
</embeddedGrammar>

Essentially, they are usual DaCHS procedures that yield rawdicts in whatever way you like. They see the source to be parsed in self.sourceToken. While this usually is a file name (that you woujld then pass to open, you can also use the string for something else, as is done here. Combined with:

<sources><item>ab</item><item>cd5</item></sources>

the grammar above would produce the rawdicts:

{'index': 'a', 'data': 'aaaa'}
{'index': 'b', 'data': 'aaaa'}
{'index': 'c', 'data': 'aaaa'}
{'index': 'd', 'data': 'aaaa'}
{'index': '5', 'data': 'aaaa'}

Embedded grammars are intended to be used for small and simple tasks, and if anything goes wrong, the error messages produced by default are not very useful. So, when things get a bit more complicated, program defensively and keep a note of where in the source you are, somehwhat like this:

with open(self.sourceToken) as f:
  for line_number, line in f:
    try:
      ...
    except Exception as ex:
      raise utils.SourceParseError(
        "Bad line: {}".format(ex),
        offending=line,
        location=str(line_number+1),
        source=self.sourceToken)

Grammars in C

If you have large data collections (millions to billions of rows), going through the all the steps of parsing, row making, and postgres ingestion with lots of Python in between can become prohibitively slow. In such cases, you can use Element directGrammar (or “a booster” for short). They let you bypass everything DaCHS and almost everything Postgres and produce binary dump material that is just slurped into the database at maximum speed.

This normally requires some knowledge of C, although DaCHS tries to help you if you're not terribly C-savvy. The one exception are FITS binary tables; DaCHS can generate C code for pulling them in without further intervention.

Boosters are a longer story, which we tell in :dachsdoc:booster.html.

Automatic and manual control

The standard idiom introduced above:

<condDesc buildFrom="somecol"/>

adapts to protocols; for instance, if somecol is a floating point variable, the condDesc will accept VizieR-like expressions in HTML forms, PQL-like intervals (“min/max”) in legacy DAL protocols and DALI-like intervals (“min max”) in modern DAL protocols.

Sometimes you need more control, for instance, when there should always be just one single string regardless of what renderer the service is used through. In that case, define the input key directly:

<condDesc>
  <inputKey original="primaryId" required="False"/>
</condDesc>

The extra attribute is there to make sure that even a column that is required in the table remains optional in the query (you can skip that if your column is not required in the first place). You can also define the input key from scratch:

<condDesc>
  <inputKey
    name="specType"
    type="text"
    tablehead="Spectral Type"
    description="Spectral type of the target object"/>
</condDesc>

If you add an enumeration of possible values, like this:

<condDesc>
  <inputKey name="color" type="text" required="True">
    <values multiOk="True">
      <option title="Red">R</option>
      <option title="Green">G</option>
      <option title="Blue">B</option>
    </values>
  </inputKey>
</condDesc>

DaCHS will try to communicate the choices to clients; in HTML forms, for instance, this will translate into a pop-up menu.

Limiting condDescs by renderer

In order to let you emulate buildFrom behaviour of making different input key types depending on the renderer even when you cannot use buildFrom, there are two properties on inputKeys are defined to only show inputs for certain renderers, viz., onlyForRenderer and notForRenderer.

Both have single strings as values. This is intended mainly for cases like SIAP and SCS where there are “human-oriented” versions of the input fields available. The built-in SCS and SIAP conditions already to that, so you can give both scs and humanSCS conditions in a core. Here is how you would define an input key that is only used for the form renderer:

<inputKey original="color">
  <property name="onlyForRenderer">form</property>
</inputKey>

Phrase makers

In all the condDescs we have seen so far, the SQL generated becomes conjunction (i.e., OR expression) of the condDesc's input key's individual SQL expressions – unless you set the joiner attribute to AND, that is (incidentally, constraints from different condDescs are always combined using AND in DaCHS).

The SQL expression derived from an input key is a simple comparison for equality for plain types, a test for overlap for interval-valued parameters, and even more complex expressions for VizieR-like expressions in the form renderer.

For complete control over what SQL is generated, condDescs may contain code in Element phraseMaker. This, again, is a procedure application, quite like with rowmaker procs, except that the signature of condDesc code is different.

Phrase maker code has the following names available:

• inputKeys – the list of input keys for the parent CondDesc
• inPars – a dictionary mapping inputKey names to the values provided by the user
• outPars – a dictionary that is later used as the parameter dictionary to the query.

The code should amend the outPars dictionary with the keys mentioned in the conditions. The conditions themselves are yielded. So, a very simple condDesc with generated SQL could look like this:

<condDesc> <!-- don't do it like this, see below -->
  <inputKey name="val"/>
  <phraseMaker>
    <code>
      outPars["xxyy"] = "x"*inPars.get("val", 20)
      yield "someColumn=%(xxyy)s"
    </code>
  </phraseMaker>
</condDesc>

However, using fixed names in outPars is not recommended, if only because condDescs could be used multiple times.

The recommended way uses the base.getSQLKey(name, value, outPars) function, where name is more or less arbitrary but should usually be the input key name, value is what you would like to see in the SQL, and outPars is a dictionary that you get passed into your phrase maker. All that you should do with outPars is pass it to this function.

getSQLKey will return a key unique to the query in question and enter the value into the outPars dictionary under that key. While that sounds complicated, it is actually rather harmless, as shown in the following real-world example that lets users input date, time and an interval in split-up form (e.g., when you cannot hope anyone will try to write the equivalent vizier-like expressions):

<condDesc>
  <inputKey name="date" tablehead="Date" type="date"
    multiplicity="single"
    required="True"/>
  <inputKey name="time" tablehead="Time (UTC)" type="time"
    multiplicity="single"
    required="True"/>
  <inputKey name="within" required="True" type="integer"
    multiplicity="single"
    tablehead="plus/minus" unit="minutes"
    description="Give measurements within this many minutes
     of your chosen date and time.  The sampling rate is 20 minutes">
    <values default="11"/>
  </inputKey>
  <phraseMaker>
    <code>
      baseTS = datetime.datetime.combine(inPars["date"], inPars["time"])
      dt = datetime.timedelta(minutes=inPars["within"])
      yield "date BETWEEN %%(%s)s AND %%(%s)s"%(
        vizierexprs.getSQLKey("date", baseTS-dt, outPars),
        vizierexprs.getSQLKey("date", baseTS+dt, outPars))
    </code>
  </phraseMaker>
</condDesc>

TAP examples

The most immediately useful examples are those for TAP/ADQL. While examples normally reside in the service elements they illustrate, that doesn't work for TAP, because the service is defined in the built-in RD //tap.

Rather than change that (with all the trouble that would give when upgrading DaCHS), instead add the TAP examples in userconfig.rd; to prepare for that, see The userconfig RD.

Once you have your userconfig.rd, locate the STREAM with the id tapexamples. One example is already given in the userconfig.rd of the distribution; take that as a template for your own examples.

Each example resides in an _example meta element. This has a mandatory title attribute, and its content format is fixed to ReStructuredText. The built-in example looks like this:

<meta name="_example" title="tap_schema example">
  To locate columns “by physics”, as it were, use UCD in
  :taptable:`tap_schema.columns`.  For instance,
  to find everything talking about the mid-infrared about 10µm, you
  could write:

  .. tapquery::

    SELECT * FROM tap_schema.columns
      WHERE description LIKE '%em.IR.8-15um%'
</meta>

In addition to title, the only thing mandatory is a block marked .. tapquery:: that contains the ADQL query you want to show. A typical client would fill this into whatever its UI provides to write queries.

Of course, you should explain what the query does and how it does this. ReStructuredText lets you have basic markup, including, if necessary, section headings if your example gets long enough.

Optionally, you can give the client a hint what table the example pertains to using the :taptable: interpreted text role. A client would typically use that to restrict the display of the example to states in which it assumes the user wishes to runs queries against that particular table.

Note again that DaCHS does not pick up changes to userconfig automatically. Hence, after adding or changing examples, you have to run:

dachs serve exp % //tap

on the server (if you have set The Admin Password).

To see your shiny new example(s), point your browser to <server url>/tap/examples.

Datalink examples

DaCHS also contains provisional support for examples associated with datalink services. We cannot recommend a client to try them with at this point, but of course having services with useful examples will make client support more attractive.

To add an example to a datalink service, add an _example meta with a title attribute to the service definition; for instance:

<service id="sdl" allowed="dlget,dlmeta">
  <meta name="title">FEROS Datalink Service</meta>
  <meta name="_example" title="Usage Example">
    On published datasets like
    :dl-id:`ivo://org.gavo.dc/~?feros/q/f04031.bdf`,
    this service lets you to cutouts, translations into FITS binary
    tables, ASCII, and possibly more, as well as simple recalibration.
  </meta>

There is only one interpreted text role in there so far, dl-id. That's a PubDID the service will generate a datalink document for.

To see the example, point your browser to the service URL with the examples renderer. If the above fragment were in the RD flash/q, the URL would thus work out to be <server url>/flash/q/sdl/examples. Since DaCHS picks up changes to “normal” RDs automatically, you will immediately see changes to your example text.

Generic examples

The DALI standard defines a term generic-parameter which can be used to annotate all kinds of services; this may come in particularly handy with the api renderer.

To use it, you can write something like:

<meta name="_example" title="Dataset identifier">
  Publisher dataset identifiers have a query part, but the
  IVORN part still has to resolve:
  :genparam:`uri(ivo://org.gavo.dc/~?feros/data/f89411.vot)`
</meta>

<meta name="_example" title="Standard identifier">
  New-style standardIds use fragments to refer to standard keys within
  vstd:Standard records, as in
  :genparam:`uri(ivo://org.gavo.dc/std/glots#tables-1.0)`
</meta>

As you can see, generic parameters are defined using a genparam interpreted text role; the content of that role is the parameter name and then, in parentheses, the parameter value. Don't worry if your parameter value has a closing parenthesis in it; DaCHS only recognises the parenthesis that ends the genparam if it is at the end of the interpreted text.

Note that such examples best sit in the service rather than top-level in the RD; if they were direct children of the RD, they would appear in all services defined in the RD.

DaCHS does not yet have support for the capability and continuation properties defined by DALI. Ask if you need them.

General Hints

• If you get error messages, be sure to check our hints on common problems – many commonly encountered problems, in particular the ones that give particularly baffling messages, are explained there with suggestions for how to fix them.

• The dachs command takes a --hints switch. With it, error messages are frequently accompanied with – you guessed it – hints on what might cause the problem and possible solutions. Note that --hints, like the other debugging switches, goes between dachs and the subcommand name, as in dachs --hints serve.

• Validate your RD. This is, in general, a good idea before doing anything with the RD, since it will allow you to more easily catch errors than the in all likelihood even more byzantine error messages that may arise when something goes wrong later. The dachs val subcommand takes one or more RDs. If you don't understand its output, complain to gavo@ari.uni-heidelberg.de – the command is really intended to help you catch errors, and if it doesn't do so, it's a bug.

• Check the logs. They are in /var/gavo/logs by default, and there's dcErrors and dcInfos (you'll want to look at both).

• When hunting bugs, it's usually a good idea to enable the logging of (many) tracebacks by passing the --debug flag to dachs.

• If you're trying to figure out server behaviour, don't run the server daemonized but use dachs serve debug instead; this won't detach and log to stdout. The user executing the serve command needs to be in the gavo group, or you'll get permission problems.

• With this (or if the problem is in non-server DaCHS code in the first place), the Python debugger is your friend. The dachs command has an option --enable-pdb that will dump you into the debugger where an uncaught exception happened (as the server should never let an uncaught exception through, that's not useful with dachs serve). If that doesn't help you, you can set breakpoints (e.g., in your own in-RD procedure defininitions) by writing import pdb; pdb.set_trace().

• To see what SQL is actually sent to the database, set the GAVO_SQL_DEBUG environment variable to any value. This could look like this:

  env GAVO_SQL_DEBUG=1 dachs imp q create
  

  Don't be too alarmed by the deluge of queries you'll see while DaCHS starts up; this is just making sure that the tables DaCHS cannot live without are present.

• If dachs serve start doesn't actually cause the server to run, something went wrong after detaching from the controlling terminal. The messages are in the logs directory, in the file serverStderr.

Debugging Rowmakers

When debugging rowmakers, it helps keeping in mind that there are two dictionaries that play a role here: The incoming rawdict and the rowdict that is being built. Having a clear image which key is where at which point is helpful. Also, it is important to remember that first all var elements are processed, then all apply elements, and finally all map elements. Within each class, elements are processed in the order of definition, with simpleMaps and idmaps after all explicit maps.

Since only applys contain larger amounts of code, debugging will mostly focus on them. If you want to look at one of your own applys, just set breakpoints as discussed above (import pdb; pdb.set_trace()). If you'd just like to have a look at what vars and results look like at some point, just add a <apply procDef="//procs#debug"/>. This will, again, drop you into a debugger.

One thing that may baffle you when you write row makers are unexpected primary key clashes (or indeed, violations of other constraints enforced by the database) or just failures where invalid data is passed to the database. Since DaCHS ships out rather large batches of rows (5000 by default) to the database if you let it, the errors will typically occur far from the lines that cause them.

In such a situation, pass -b 1 to dachs imp – this will slow down the import quite dramatically, but the import will fail exactly at the source or row that causes the problem.

Debugging Services

Debugging services is of course an extra challenge since code runs deep in the bowels of a complex system, with timeouts, threads, and all kinds of nastyness involved. The first adivce is: Pull whatever is mysterious into your development system and run dachs serve debug. You can even drop into the python debugger while the server processes a request (the server will of course block while you are in the debugger). It's a bit tricky to communicate with the debugger in between the log messages of dachs serve debug, and there's no readline support since pdb doesn't think it's running within a terminal there, but it's definitely doable.

Note that in python3, the debugger by default changes SIGINT processing so pressing Control-C will drop you back into the debugger. With dachs serve debug, that behaviour is unwelcome, because it makes terminating the server a bit of a steeplechase. Hence, when setting breakpoints in server code, use:

import pdb;pdb.Pdb(nosigint=True).set_trace()

Another challenge is that sometimes problems manifest themselves in a running server. In that case it's sometimes useful to open a manhole into the server. One reasonably convenient way to do this is to put a special RD somewhere (e.g., __tests/manhole.rd) and use some RD-embedded code to introspect the server. We usually use a datalink service for this since it keeps things nicely self-contained – an obvious alternative with less bending could be a python core returning a pair of text/plain and a string.

Here's how something that fiddles out a column property would look like:

<service id="cq" allowed="qp">
  <property name="queryField">prop</property>
  <pythonCore>
    <inputTable>
      <inputKey name="prop" type="text"/>
    </inputTable>
    <coreProc>
      <code>
        from gavo import base
        col = base.resolveCrossId("arihip/q#main.hipno")
        payload = "Result: %s"%(
          getattr(col, inputTable.getParam("prop"), "(err)"))
        return ("text/plain", payload.encode("ascii"))
      </code>
    </coreProc>
  </pythonCore>
</service>

See the qp renderer for how the queryField magic is to be understood. The net effect is that looking at http://localhost:8080/manhole/cq/qp/type you can find out what DaCHS things the type of arihip.main's hipno column is – and you can try description, ucd, or whatever in that last URI segment, too.

Note that you can edit manhole.rd, save it, and reload the page; the server will notice your changes and display the new result without a restart.

Case Studies

In this section we will damage the arihip RD in various ways, show you how the errors manifest themselves, and how you could try and figure them out. It's highly recommended to play the scenarios; it's very well possible that the actual messages will look differently for you if we've changed and hopefully improved the software. We're thankful if you point us to outdated examples.

arihip > dachs imp q
*** Error: In /home/msdemlei/gavo/inputs/arihip/q.rd: mismatched tag:
line 674, column 2

/arihip > xmlstarlet val -e q.rd
q.rd:674.12: Opening and ending tag mismatch: meta line 72 and resource
</resource>
           ^
q.rd - invalid

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
     primary="hipno">
   <wonder>foo</wonder>

msdemlei@victor:/home/msdemlei/gavo/inputs/arihip > dachs imp q
*** Error: At /home/msdemlei/gavo/inputs/arihip/q.rd, (112, 4): table
elements have no wonder attributes or children.

<meta name="note" tag="tabflags"><![CDATA[

arihip > dachs imp q
*** Error: In /home/msdemlei/gavo/inputs/arihip/q.rd: not well-formed
(invalid token): line 411, column 24

<meta name="note" tag="src">
  The srcSel field indicates which catalogue the astrometric solution
was taken from using the following codes:

arihip > dachs --debug imp q
*** Error: At /home/msdemlei/gavo/inputs/arihip/q.rd, (317, 4): Bad
text in meta value (Bad indent in line u'    was taken from using the
following codes:')

<map dest="kbin">parsWithNull(@kbin, str, "9")</map>

arihip > dachs imp q
Making data arihip/q#import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
Done /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz, read 1
*** Error:   Row
  alphaHMS -> '0  0  0.216002'
  ddeHIP -> '+     0.93'
  [...lots of lines...]
  t_ra_mod -> '91.45'
  vrad -> ''

Field kbin: While building kbin in <rowmaker without id>: name
'parsWithNull' is not defined

<make table="main">
  <rowmaker idmaps="*" id="make_main_row">

Field kbin: While building kbin in make_main_row: name
'parsWithNull' is not defined

<map dest="vrad">parseWithNull(
   @vrad, lambda a:float(killBlanks(a)), "")</map>

/arihip > dachs imp -M 12 q
[...]
Field vrad: While building vrad in None: could not
convert string to float: +   8.3

vrad:      188-195

arihip > dachs imp -M 12 q
Making data import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
  Source hit import limit, import aborted.
Done /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz, read 13
Shipped 13/13
Create index Primary key on arihip.main
Create index main_mv
Create index main_q3c_main
Rows affected: 13

arihip > dachs info arihip/q#main
 col           min         avg     max       hasnulls
 [...]
 vrad          None        None    None      True
 [...]

<column name="vrad" ucd="phys.veloc;pos.heliocentric"
  required="True"
  tablehead="v_rad" verbLevel="20" unit="km/s"
  description="Radial velocity as used in calculating the foreshortening
  effect."/>

arihip > dachs imp -M 12 q
Making data arihip/q#import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
Done /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz, read 1
*** Error:   Row
alphaHMS -> '0  0  0.216002'
[...]
t_ra_mod -> '91.45'

Field vrad: While building vrad in <rowmaker without id>: Key 'vrad'
not found in a mapping.

 <apply>
  <code>
    ddt
  </code>
</apply>

arihip > dachs imp q
[...]
Field procf11fed4c: While executing procf11fed4c in <rowmaker without
id>: name 'ddt' is not defined

<apply>
   <code>
     a = 1
     b = 2/a
     c = 3/(a+b)
     d = 4/(a+b+c+1)
     e = 5/d
   </code>
 </apply>

While executing proc985706c in None: integer division or modulo by zero

2013-09-23 15:57:37,304 [INFO 24430] Swallowed the exception below, re-raising Field proc985706c: While executing proc985706c in None: integer division or modulo by zero
Traceback (most recent call last):
  File ".../gavo/rscdef/rmkdef.py", line 583, in __call__
    exec self.code in self.globals, locals
  File "generated mapper code", line 13, in <module>
  File "<string>", line 9, in proc985706c
ZeroDivisionError: integer division or modulo by zero

<apply>
   <code>
     import pdb; pdb.set_trace()
     ...

User/Group Management

The DaCHS user administration is fashioned a bit after the Unix user/group model, except that there always is a group corresponding to a user. To create a user and its group, use dachs admin adduser, like this:

dachs admin adduser kroisos notsecret “Remove when xy is public”

This command adds the user kroisos with the password notsecret and an optional comment reminding future operators what to do with the identity.

To add existing users to groups, use dachs admin addtogroup, like this:

dachs admin addtogroup kroisos happy

– this adds kroisos to the happy group, and whoever can authenticate as kroisos will be allowed access to any products or services restricted to happy.

To discover further commands manipulating the user table, try:

dachs admin --help

Important: When you use authentication, please set the [web]realm configuration item to some string reasonably characteristic for your site. DaCHS sends that realm with the authentication challenge, and even if these days, it's not terribly common to show it any more since it's been abused in various ways, it may still be shown to users.

Protecting Services

To password-protect entire services, use the limitTo child of the service element, for instance:

<service id="scs" core="scsCore" allowed="form,scs.xml"
  limitTo="happy">
  ...
</service>

Any access to the service will then challenge the client to authenticate as a user belonging to the group given in limitTo. Only one such group can be given. If you forsee the need for complex authorization schemes (rather than “there's one user on my system, and whoever's authorized get its credentials”), it is probably a good idea to create one user per service and add “real” users to the corresponding group as necessary.

Embargoing Products

DaCHS' products subsystem has the notion of owners and embargo periods, which allows public services to deliver metadata on products during their proprietarity period, while handing out the data itself only to authorized clients. The embargo will automatically be lifted once the proprietarity period is over.

To make a ”product” (e.g., spectrum or image) proprietary, in the //products#define rowfilter, set the owner and embargo keys. Owner is the name of a user created as described above, embargo must eventually become a timestamp, so you will in general come up with an ISO datetime string or a Python datetime.datetime instance. Here is an example that says images become public a year after the observation:

<fitsProdGrammar qnd="True">
  <rowfilter procDef="//products#define">
    <bind key="embargo">parseTimestamp(row["DATE_OBS"]
      )+datetime.timedelta(days=365)</bind>
    <bind key="owner">"danish"</bind>
    <bind key="table">"danish.data"</bind>
  </rowfilter>
</fitsProdGrammar>

This is, in our view, an acceptable policy, but many observers want weird policies (try to talk them out of it, since such behaviour is not nice, and it leads to a bad user experience in the VO as a whole). You can get as fancy (or antisocial) as you like using custom rowfilters, as in the following example that sets a default embargo for the end of 2008, except for calibration frames and the observations of two objects made in 2003:

<fitsProdGrammar qnd="True">
  <rowfilter procDef="//products#define">
    <setup><code>
      <![CDATA[
      def getEmbargo(row):
        res = '2008-12-31'
        if (row.get("ARI_TYPE")!="SCIENCE" or
            row["ARI_OBJC"]=='Q2237+0305'
            or row["ARI_OBJC"]=='SBSS 1520+530'):
          if '2003-01-01'<=row['DATE_OBS']<='2003-12-31':
            res = '2005-12-31'
        return res
      ]]>
    </code></setup>
    <bind key="owner">"maidanak"</bind>
    <bind key="embargo">getEmbargo(row)</bind>
    <bind key="table">"maidanak.rawframes"</bind>
  </rowfilter>

An embargoed product can only be retrieved by the owner until the embargo period is over. What you give as owner is a group name; if someone can authenticate as the member of a group, they can access the data – see above for details on how to create users and groups.

STREAM and FEED

When you have a sequence of elements that you need in several places, you can put these elements into an Element STREAM and replay them with Element FEED in as many places as you want. You have seen several examples of that in this tutorial, for instance, in:

<FEED source="//scs#coreDescs"/>

To see what this plays back, look for an element with id coreDescs in the RD scs, which in turn you can most easily access with:

dachs adm dumpDF //scs | less

You will see that this element looks like this:

<STREAM id="coreDescs">
        <doc>[...]</doc>
        <condDesc original="//scs#humanInput"/>
        <condDesc original="//scs#protoInput"/>
</STREAM>

The doc element is local to the STREAM; make sure that you use it to drop a few words on what the stream is supposed to do – your future self will be grateful.

All other elements will end up exactly like typed there whereever there is a corresponding FEED.

Another example where a simple duplication of elements is useful is in fk6/q, where a catalogue has been split into two parts. Both parts share a lot of the columns, but there are some extra columns specific to each part. To deal with this, the RD defines a STREAM:

<STREAM id="commonFrontFields" doc="Field 1..23 are (about) the same
  for parts I and III.">

  <column name="hipno" type="integer" required="True"
    ucd="meta.id"
    tablehead="HIP#"
    description="Number of the star in the HIPPARCOS Catalogue (ESA 1997)"
    verbLevel="25"/>
  <column name="comname" type="text"
  ...

and then, further down, replays it into each table:

<table onDisk="True" id="part1" adql="True" primary="localid"
    mixin="//scs#q3cindex">
  <mixin>//scs#pgs-pos-index</mixin>
  <meta name="description">Part I of the FK6 (successor to Basic FK5)</meta>
  <nrows>878</nrows>
  <FEED source="commonFrontFields"/>
  ...

<table onDisk="True" id="part3" adql="True">
  ...
  <FEED source="commonFrontFields"/>

However, more commonly there will be slight variations between the replayed elements. Consider, for example, the license declaration we have seen above:

<FEED source="//procs#license-cc-0" what="ARIHIP"/>

Since CC-0 recommends that you explitictly state what you distribute, we have to put in something different every time we replay the stream. If you look at the stream (adm dump is your friend again), you will see how this works:

<STREAM id="license-cc0">
  <doc>[...]</doc>
  <meta name="rights" format="rst">\\RSTcc0{\what}</meta>
  <meta name="rights.rightsURI"
    >http://creativecommons.org/publicdomain/zero/1.0/</meta>
</STREAM>

The basis here is that all attributes to FEED are available for macro expansion. On the other hand, FEED does one level of macro expansion, which explains the double backslash in front of RSTCcc0: after the first macro expansion, this becomes a single backslash. Hence with what="ARIHIP" as above, the FEED's net effect is the same as if someone had typed:

<meta name="rights" format="rst">\RSTcc0{ARIHIP}</meta>
<meta name="rights.rightsURI"
  >http://creativecommons.org/publicdomain/zero/1.0/</meta>

Another example for a parameterised STREAM is in sp_ace/q, where multiple output fields of a code have possibly asymmetric confidence intervals:

<STREAM id="valWithConfidence">
  <doc>
    A param with a confidence interval (give name, ucd, unit, tablehead
    and description and get three params including asymmetric error
    bars.
  </doc>
  <param name="\name"
    ucd="\ucd" unit="\unit"
    tablehead="\tablehead"
    description="\description of the best-fit modelled spectrum."/>
  <param name="\name\+_low"
    ucd="stat.error;stat.min;\ucd" unit="\unit"
    tablehead="Lower \tablehead"
    description="Lower limit of 68% confidence interval of \description"/>
  <param name="\name\+_up"
    ucd="stat.error;stat.max;\ucd" unit="\unit"
    tablehead="Upper \tablehead"
    description="Upper limit of 68% confidence interval of \description"/>
</STREAM>

This is used later on like this:

<FEED source="valWithConfidence"
  ucd="phys.temperature.effective" unit="K"
  name="teff" tablehead="T_eff"
  description="Effective Temperature"/>
<FEED source="valWithConfidence"
  ucd="phys.gravity" unit="cm/s**2"
  name="logg" tablehead="Log g"
  description="Log of gravity"/>

LOOP

Element LOOP takes the idea of having multiple FEED elements further and makes it easy to generate lots of fillers, possibly even programmatically.

The simplest way to use LOOP is by giving a space-separated list of “items”:

<LOOP listItems="a b c">
  <events>
    <column name="\item"/>
  </events>
</LOOP>

The events child of the LOOP element is something like an immediate STREAM; you could use a stream instead and reference it in a source attribute, but that's rarely easier to read.

These events are then replayed to the parser for each item in the LOOP's listItems attribute. Each occurrence of the \item macro is replaced with the current item. So, in the resulting RD tree, the fragment above will have the same result as:

<column name="a"/>
<column name="b"/>
<column name="c"/>

Sometimes the list items are used in multiple places in the same document. To avoid having to maintain multiple lists, you can define macros using RD's macDef element; this could look like this:

<resource schema="foo">
  <macDef name="bands">U B V R</macDef>
  <table id="mags">
    <LOOP listItems="\bands">
      <events>
        <column name="mag\item"/>
      </events>
    </LOOP>
  </table>
  <rowmaker id="build_mags">
    <LOOP listItems="\bands"/>
      <events>
        <map dest="mag\item">parseFromString(MAG_\item)</map>
      </events>
    </LOOP>
  </rowmaker>
...
</resource>

Note that macro names must be at least two characters long.

Frequently, the loop variable should not just take on a single string. For such cases you can feed in tuples. The most convenient way to do this is LOOP's csvItems attribute. The content of this is a string literal containing comma separated values with labels, i.e., parsable with Python's csv.DictReader. In your events, you can then refer to the labeled items using macros. For example:

<resource schema="foo">
  <macDef name="bands">
    band,source
    U 10-12
    V 13-16
  </macDef>
  <table id="mags">
    <LOOP csvItems="\bands">
      <column name="mag\band"/>
    </LOOP>
  </table>
  <data id="magscontent">
    <columnGrammar>
      <LOOP csvItems="\bands">
        <col key="mag\band">\source</col>
      </LOOP>
    </columnGrammar>
    <make table="mags"/>
  </data>
</resource>

You can take this even further and generate the replacements with Python code. For instance, in califa/q3, I needed to pull dozens of columns through a parseWithNull in a row maker. To maintain the list of such columns was already a challenge, and so I had the computer do it by pulling the table out of the parse context (which is a complex object you usually don't see; codeItems runs while the RD is being parsed, and so there are additional limits to what you can see) and retrieving all ”flag” columns. Those are the ones that need a uniform parsing:

<LOOP>
  <codeItems>
    for col in context.getById("cubes"):
      if col.name.startswith("flag_"):
        yield {"item": col.name}
  </codeItems>
  <events>
    <map dest="\item">parseWithNull(@\item, int, -1)</map>
  </events>
</LOOP>