Defining Tables

A major part of the metadata DaCHS deals with is the table structure. It is defined in table elements, which usually are direct children of the resoource element. A resource element may contain multiple table definitions. See http://dc.g-vo.org/arihip/q/cone/static for what upstream documentation we had when we made the service.

Skip over the macDef elements for now to where the table element begins. What you see is something like:

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
  primary="hipno">

The id attribute of the table doubles as the name of the database table; make sure you use something that works as a valid simple SQL identifier (i.e., [A-Za-z_][A-Za-z0-9_]*) – DaCHS does not support for delimited identifiers as table names.

Be sure to always specify onDisk="True" unless you're going for special effects – without it, the table will end up only in memory. The adql attribute says that TAP queries should be allowed on the table; leave it out for tables not suitable for "raw" consumption by your clients.

For the mixin attribute, see Indices and Mixins.

Finally, using the primary attribute you can specify an explicit primary key of the table (if it is made up of several columns, concatenate their names with commas). This is made into a primary key for postgres straightforwardly, which means that the database makes sure there are no two rows with the same value for the primary key. Also, the database creates an index for efficient queries using the primary key.

What follows is a definition of the structure of the space-time coordinates:

<stc>
  Position ICRS Epoch J2000.0 "raj2000" "dej2000" Error "err_ra" "err_de"
    Velocity "pmra" "pmde" Error "err_pmra" "err_pmde"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raHIP" "deHIP" Error "err_raHIP" "err_deHIP"
    Velocity "pmraHIP" "pmdeHIP" Error "err_pmraHIP" "err_pmdeHIP"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raSTP" "deSTP" Error "err_raSTP" "err_deSTP"
    Velocity "pmraSTP" "pmdeSTP" Error "err_pmraSTP" "err_pmdeSTP"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raLTP" "deLTP" Error "err_raLTP" "err_deLTP"
    Velocity "pmraLTP" "pmdeLTP" Error "err_pmraLTP" "err_pmdeLTP"
</stc>

What this says is that there's a number of coordinate structures in the table, grouping together positions, errors, and proper motions in a specific reference frame. This is again a topic of its own, discussed in STC

<column name="hipno" type="integer" ucd="meta.id;meta.main"
  tablehead="HIP id" verbLevel="1"
  description="Number of the star in the HIPPARCOS Catalogue (ESA 1997)."
  required="True"/>
<column name="srcSel" type="text" ucd="meta.flag"
  tablehead="Source" verbLevel="25"
  description="Source of the astrometric solution"
  note="src"/>

<column name="n_obs" type="integer"
    description="Number of
    observations, NULL if interpolated data">
  <values nullLiteral="-1"/>
</column>

Parsing Input Data

Going further down in the RD, you will find a data element. Their main purpose is to describe how certain input files fill the table(s) defined above. It starts like this:

<data id="import">
  <sources>data/data.txt.gz</sources>

Data elements can have ids which can be used to indiviually reference them from a gavo imp command line; this is useful if you just want to import one part of a multi-table data collection. The default of gavo imp default is to build all data elements except those having an auto="False" attribute.

It is recommended that the id is a short verb phrase, as data basically contains instructions for an action. You might rightly argue that we have not chosen the element name too aptly (recipe might have been more appropriate), but we feel it's too late to change it now.

The sources element lets you specify the names of the input files to be processed. There are several ways to do that; in this case, there's just one input file, which is given as element content, with the path interpreted relative to the resource directory. If the data was distributed into several files in two directories, something like the following specification would do the trick:

<sources>
  <pattern>inp2/*.txt</pattern>
  <pattern>inp1/*.txt</pattern>
</sources>

The sources element also has a recurse (boolean) attribute that makes DaCHS search for the pattern in the subdirectories of the path part of the pattern.

Now have a look at the input file:

zless data/data.txt.gz

You'll see that we have a plain ASCII file with aligned columns, header lines, and even a cell separator ("|"). That's still a fairly common format for raw data, but by no means the only one. To give DaCHS the flexibility to deal with anything upstream cares to throw at you, DaCHS has the concept of a grammar.

There are many grammars defined, e.g., for getting values from FITS files, VOTables, or using column-based formats; you can also write specialized grammars in python. All grammars read "something" and emit a mapping from names to (mostly) string values; those unprocessed string-to-string mappings are called "rawdicts" in DaCHS jargon (distinguished from "rowdicts" ready for database ingestion, which sport processed and typed data).

It is often useful to inspect what a grammar emits. You can do that using import's --dump flag. During development, it is frequently convenient to just import a few rows and watch what they produce; this would look like this:

gavo imp -M 100 --dump q.rd | less

(if you interrrupt the above command, your table should be unscathed – check with gavo info –; otherwise just re-run the full import).

With a source that has both a separator character and aligned columns, there are several valid choice for which grammar to use on this particular file. In the RD, it next says:

<columnGrammar topIgnoredLines="9" preFilter="zcat">
  <colDefs>
    hipno:     3-8
    srcSel:    47-49
    alphaHMS:  59-73
    deltaDMS:  77-91
    pmra_mas:  95-103
    pmde_mas:  107-115
    t_ra_mod:  119-123
    err_ra_mas:127-131
    err_pmra_mas:135-139
    t_de_mod:  143-147
    err_de_mas:151-155
    err_pmde_mas:159-163
    parallax_mas:167-172
    e_parallax_mas:176-180
    kp:        184
    vrad:      188-195
    mv:        199-203
    km:        207
    kbin:      211-212
    kdmu:      216
    kae:       220
    flags:     882-901
  </colDefs>

– so we went for a columnGrammar. Those cut up every input line along character indices that are here, following the display in most editors, are counted from 1 upwards. Note that in ranges, the last column is included in the string – these are no python slices but basically a representation of the character ranges in VizieR-style "byte-by-byte"-descriptions.

The assignment of names to column ranges can happen both in a colDefs element as shown above – one specification per line, label mapped to a column or a range of columns.

Alternatively, have several col elements, each of which has a key attribute that gives a name. This could be the name of a target column in the simplest case, or it can be an auxillary identifier that is later processed in a rowmaker. These individual specifications are interesting when combined with RD macros, and that's where they come in in the arihip RD (again, using LOOPs).

Grammars also have various attributes; the ones parsing from text files support, for example, topIgnoredLines, which allows you to skip header lines, and preFilter that allows running the input through a shell command before it is processed using DaCHS (if you find yourself doing more than just decompression in such a preFilter, you should probably look for a different solution).

Mapping data

The arihip RD then goes on with:

<make table="main">
  <rowmaker idmaps="*">
    <var name="raj2000">hmsToDeg(@alphaHMS, None)</var>
    <var name="dej2000">dmsToDeg(@deltaDMS, None)</var>
    ...
    <map dest="kbin">parseWithNull(@kbin, str, "9")</map>
    <map dest="vrad">parseWithNull(
      @vrad, lambda a:float(killBlanks(a)), "")</map>

The make element brings together a table (in the table attribute) with a recipe how to fill it from the output of the grammar (the row maker).

Incidentally, there can be multiple make elements in a single data element if multiple tables (using different row makers) are generated from the same grammar output. This is particularly useful in combination with dispatching grammars that let, in effect, the grammar choose which make to use.

Makes can also carry scripts in SQL or python, at various points of the building process. These let you perform all kinds of higher magic. For details, see the chapter on scripting in the reference.

As explained above, output of grammars and hence the input to a make is a sequence of mappings from names to strings (the "rawdicts"). The database, on the other hand, wants typed values, i.e., integers, time stamps, etc. Also, data in input tables is frequently given in inconvenient formats (e.g., sexagesimal angles), deprecated or inconsistent units, or values may be distributed over multiple columns (e.g., date and time of an observation when we want a single timestamp). To cover these and more tasks, DaCHS has row makers, the results of which are then called rowdicts (note the subtle difference from the rawdicts coming in from the grammars: "row makers turn rawdicts into rowdicts").

Basically a row maker consists of

• var elements -- assignments of expression values names in the rawdict.
• map elements -- simple mappings of (python) expressions to values in the destination rowdict
• procedure applications (see apply) -- manipulations of both rawdicts and rowdicts in python code

The fragment above shows one of several ways to use both var and map (which work exactly the same way, except that vars end up in the rawdict, and map in the rowdict): generating values from python expressions, where there is the special syntax @identifier which expands to whatever value the rawdict has for that key (or raises a KeyError if the key is not present in the rawdict).

The rawdict manipulations that var does are useful if you want to re-use whatever you compute. The map element, on the other hand, writes directly into the results dictionary, the keys of which directly correspond to the column names.

When building a rowdict for ingestion into the database, a row maker first binds var names, then applies procedures and finally performs the mappings. In the bodies of the mappings, you can use all built-in python functions plus a set of useful rowmaker functions documented in the reference documentation, as well as everything from the python standard library modules datetime, math, os, re, sys, time, and urllib (you need to give the module name when referring to names from these modules as in, e.g., re.sub). Furthermore, the gavo modules base, stc, and utils are in the namespace of the mapping code, as well as the submodule utils.pgsphere. TODO: link to useful documentation for them here.

For simple cases, maps will suffice; frequently, you can do without python expressions by giving a src attribute specifying a rawdict key instead of element content (which is preferable if possible – as elsewhere, less code is better in RDs, too). The rawdict string will in this case be converted to a typed value using "sane" defaults (e.g., integers will be converted by python's int constructor, where empty strings are mapped to None, datetimes are parsed as ISO strings, etc)

If you match the keys in the rawdicts with the names of the database columns their content is supposed to end up with and the content needs no further manipulations, a row maker like:

<rowmaker>
  <map dest="evi" src="evi"/>
  <map dest="av" src="av"/>
  <map dest="ai" src="ai"/>
</rowmaker>

would to the trick. Since this is a bit unwieldy, DaCHS provides a shortcut:

<rowmaker> <simplemaps>evi:evi,av:av,ai:ai</simplemaps> </rowmaker>

which expands to exactly what is written above. The keys in each pair do not need to be identical; the first item of each pair is the table column name, the second the rawdict key.

The case where the names of rawdict and rowdict keys are identical is so common (since the RD author in general controls both) that there is yet another shortcut for this:

<rowmaker>
  <idmaps>evi,av,ai</idmaps>
</rowmaker>

Idmaps sets up one map element each with both dest and src set to the value for every name in the comma separated list idmaps.

You can abbreviate this further to:

<rowmaker idmaps="*"/>

– so, idmaps values can contain shell patterns. They will be matched to the column names in the target table. For every column for which there is no explicit mapping, an identity mapping (with type conversion) will be set up with this specification.

Of course, you can have values that do not even depend on grammar output:

<map dest="dateIngested">datetime.datetime.now()</map>

Null values are always troublesome. Within DaCHS, the null value (almost) always is python's None. There is the row maker function parseWithNull to help you come up with those; if your upstream was devious enough to use 99.99 as a null value for a magnitude, you could say:

<map dest="Vmag">parseWithNull(@VmagSrc, float, "99.99")</map>

Note that the null value here is a literal matched against the string coming from the grammar; due to the rounding errors when converting from decimal to binary floating points, you can only safely compare against relatively few floating point numbers (99.99 is not among them), so you shouldn't do that if you can avoid it.

If you need to scale this (or if null values are chosen that they are invalid literals to begin with), a feature that lets you null out a value when an specific type of exception is raised comes in handy. This is map's nullExcs attribute, which is just a comma separated list of exceptions that should be caught and interpreted as "this is null". If, in the example above, the source would give the magnitude in millimags to save a comma, you could use:

<map dest="Vmag" nullExcs="TypeError"
  >parseWithNull(@VmagSrc, float, "99999")/1000.</map>

If parseWithNull here returns None, a TypeError will be raised and caught, and Vmag will be None.

You can turn more than one exception into None. For example example, if magicOffset has been parsed before and could be None, while magicLit is to be parsed and has the empty string as a NULL literal, you could write:

<map dest="magic" nullExcs="ValueError,TypeError"
  >@magicOffset+float(@magicLit)</map>

If magicOffset is None, magic will be None via the TypeError, whereas empty magicLits will result in Nones via a ValueError.

We defer the discussion of apply elements to the discussion of how to build SIAP services.

rowmaker elements may also be direct children of resource; this is for when they are used in more than one data. You would then give the rowmaker an id attribute and say something like <make rowmaker="id-of-rowmaker" table=.../> However, for the standard case it's best to keep everything related to a given import together in the make element.

Indices and Mixins

We have so far deferred the discussion of the mixin attribute in arihip's opening table element:

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
  primary="hipno">

Mixins are DaCHS' primary tool to endow tables with "everything needed to serve a standard" (e.g., a minimal set of columns, certain indices, or metadata). For instance, an image table must have a certain structure determined by the SIA protocol. The //siap#pgs mixin makes sure that tables have this structure, and it makes sure that the table containing information on all the file-like datasets in the data center (which is called dc.products) is updated when the table is filled.

The content of the mixin element (or the attribute value when you give the mixin property as an attribute) is a reference to a mixin definition. These references typically go into some system descriptor (though you could define your own mixins), and the double slash in a DaCHS reference means "system descriptor" (in actual truth, it's just an abbreviation for __system__/). The reference documentation contains a chapter on DaCHS' public mixins. For the curious: you can have a look at the actual definitions by admin's dumpDF subcommand, e.g., like this:

gavo admin dumpDF //scs

The //scs#q3cindex mixin referenced here arranges for spatial indexing of tables having some sort of spherical coordinates. To identify which columns to index, DaCHS inspects the UCDs of the columns; what it looks for here are columns with UCDs of pos.eq.(ra|dec);meta.main` as index columns [2]. Contrary to mixins for other standard protocols, it does not automatically insert these columns (and neither the only other required column in SCS, the main row identifier with the UCD meta.id;meta.main).

Since in addition to spatial queries, we also expect a lot of queries constraining the mv column, we ask for an index on it using DaCHS' index element, which is a child of table:

<index columns="mv"/>

This is the simplest, but mostly sufficient, form of defining an index; for advanced usage, please refer to the reference documentation. If you decide to add an index to a table later on, or to initiate re-indexing after a lot of data changed, see the -I option of gavo imp.

Cores and Services

The last but one part of the RD deals with how to get the data out of the database again, i.e., the services exposing the data. This part is fairly simple for arihip:

<service id="cone" allowed="scs.xml,form">
  <meta name="shortName">arihip cone</meta>
  <meta name="testQuery">
    <meta name="ra">9.4076</meta>
    <meta name="dec">9.6414</meta>
    <meta name="sr">1.0</meta>
  </meta>

  <dbCore queriedTable="main">
    <FEED source="//scs#coreDescs"/>
    <condDesc buildFrom="mv"/>
    <condDesc>
      <inputKey original="hipno" required="False"/>
    </condDesc>
  </dbCore>

  <publish render="scs.xml" sets="ivo_managed"/>
  <publish render="form" sets="ivo_managed,local"/>
  <outputTable verbLevel="20"/>
</service>

To understand what's going on here, some basic understanding of DaCHS' service architecture is required; it consists of:

• cores; they actually do the computation or database query
• renderers; these digest the data coming in from the service and (in general) format the result in some way requested by the user. There are renderers for web forms, VO protocols, imges, etc. Frequently – as in the example –, you can use the same core for both a VO protocol and a form-based service by just allowing different renderers.
• The service; it holds together the core and the renderer, can reformat core results, controls the metadata, etc.

The renderers are referenced by name in the service's allowed attribute. What can be given there (concatenated by commas) is listed in the reference documentation's renderer chapter. As you have seen above, the renderer is selected via the URL. If a client tries to retrieve a URL with a renderer that is not in the service's allowed list, DaCHS will respond with a 403 forbidden HTTP code (excepting certain "unchecked" renderers like info that typically expose service metadata). In addition, not all cores can be combined with all renderers even if you list them in allowed. For example, the ssap.xml renderer will not (usefully) work on anything but an ssapCore.

The most common core for catalog services (and the one you'll typically use for SCS services) is the dbCore, as used here. See cores available in the reference documentation for more predefined cores -- e.g., to run ADQL queries or to upload files. For special functionality, you can even write your own core.

The dbCore generates a (single-table) query from condition descriptors and returns a table that you describe through an output table. Cores are defined as direct children of the resource (as with grammars, you can also have them in resource and then write core="id-of-element", which makes sense when a single core is shared by several services).

dbCores need a queriedTable attribute, the value of which must be a table reference. This is the table the query will run against.

The condition descriptors (or condDescs for short) define input fields (for the form renderer, these will be rendered as form items people can fill in). Most commonly, you will either define them using the original attribute (when inheriting from predefined condDescs) or using buildFrom. The first case is typically used in connection with protocols and on tables having mixins; such condDescs result in zero or more input fields, and they typically inspect the queried table. For example, the //scs#humanScs condDesc locates the "main" positions as identified by UCDs and generates queries against them using two input fields, one it tries to guess a position from, and another for the search radius.

When you define a condDesc using buildFrom, the result is usually one or more input field(s) constraining values in the column named in the buildFrom attribute. The software tries to make some useful input definition from that column, depending on the renderer. Renderers with a parameter style (this is given in each renderer's description the reference documentation) of "form", for example, let users query string-like columns using Vizier-like string expressions, real and double precision columns using Vizier-like float expressions, and so on. The pql style allows a specification like for SSAP (e.g., "range_min/range_max").

The service element must have an id attribute that is used to select the service run in the access URL. Furthermore, there should be certain pieces of metadata useful in later registration. First, there's shortName, which is typically used by clients in space-restricted displays. It must not be longer than 16 characters, so something like an acronym and a very terse role identifier is the best you can do (hence the "arihip cone" here). Frequently, a title meta is also useful, in particular when an RD contains multiple services, in which case one could be "Cone search for ARI's HIPPARCOS re-reduction" and the other, say, "Autocorrelation on ARI's HIPPARCOS re-reduction".

See the data checklist for more information on useful generic metadata and remember that services inherit whatever is defined within resource when not specified within the element.

Many standard VO protocols require additional, protocol-specific metadata. In the case of arihip, we have a Simple Cone Search service, which, as laid down in the section on the scs.xml renderer in the reference documentation, requires the parameters of a test query returning a nonempty result.

General Hints

• If you get error messages, be sure to check our hints on common problems – may commonly encountered problems are explained there with suggestions for how to fix them.

• The gavo command takes a --hints switch. With it, error messages are frequently accompanied with – you guessed it – hints on what might cause the problem and possible solutions. Note that --hints, like the other debugging switches, goes between gavo and the subcommand name, as in gavo --hints serve.

• Validate your RD. This is, in general, a good idea before doing anything with the RD, since it will allow you to more easily catch errors than the in all likelihood even more byzantine error messages that may arise when something goes wrong later. The gavo val subcommand takes one or more RDs. If you don't understand its output, complain to gavo@ari.uni-heidelberg.de -- the command is really intended to help you catch errors, and if it doesn't do so, it's a bug.

• Check the logs. They are in /var/gavo/logs by default, and there's dcErrors and dcInfos (you'll want to look at both).

• When hunting bugs, it's usually a good idea to enable the logging of (many) tracebacks by passing the --debug flag to gavo.

• If you're trying to figure out server behaviour, don't run the server daemonized but use gavo serve debug instead; this won't detach and log to stdout.

• With this (or the problem is in normally-running DaCHS code in the first place), the python debugger is your friend. The gavo command has an option --enable-pdb that will dump you into the debugger where an uncaught exception happened (as the server should never let an uncaught exception through, that's not useful with gavo serve). If that doesn't help you, you can set breakpoints (e.g., in your own in-RD procedure defininitions by writing import pdb; pdb.set_trace. If you install the python-ipdb package, and write ipdb instead of pdb, you'll get a nicer debugger commandline with tab completion and similar frills.

• To see what SQL is actually sent to the database, set the GAVO_SQL_DEBUG environment variable to any value. This could look like this:

  env GAVO_SQL_DEBUG=1 gavo imp q create
  

  The first couple of requests are for internal use (like checking that some meta tables are present).

• If gavo serve start doesn't actually cause the server to run, something went wrong after detaching from the controlling terminal. The messages are in the logs directory, in the file serverStderr.

Debugging Services

Debugging services is of course an extra challenge since code runs deep in the bowels of a complex system, with timeouts, threads, and all kinds of nastyness involved. The first adivce is: Pull whatever is mysterious into your development system and run gavo serve debug. You can even do the import pdb;pdb.set_trace() trick then. It's a bit tricky to communicate with the debugger in between the log messages of gavo serve debug, and there's no readline support since pdb doesn't think it's running within a terminal there, but it's definitely doable.

Another challenge is that sometimes problems manifest themselves in a running server. In that case it's sometimes useful to open a manhole into the server. One reasonably convenient way to do this is to but a special RD somewhere (e.g., __tests/manhole.rd) and use some RD-embedded code to introspect the server. We ususally use a datalink service for this since it keeps things nicely self-contained – an obvious alternative with less bending could be a custom renderer.

Here's how something that fiddles out a column property would look like:

<resource schema="test">
  <service id="look" allowed="dlget">
    <datalinkCore>
      <descriptorGenerator>
        <code>
          return ProductDescriptor(None, None, None, 'text/plain', )
        </code>
      </descriptorGenerator>
      <dataFunction>
        <code>
          descriptor.data = "debugging"
        </code>
      </dataFunction>
      <dataFormatter>
        <code>
          class DebugResult(Page):
            def renderHTTP(self, ctx):
              request = IRequest(ctx)
              request.setHeader("content-type", "text/plain")
              return "%s"%base.caches.getRD("maidanak/res/rawframes"
                ).getById("rawframes").getColumnByName("accref").displayHint
          return DebugResult()
        </code>
      </dataFormatter>
    </datalinkCore>
  </service>
</resource>

All but the data formatter is just blind code for hiding our true intentions from the datalink machinery. In the data formatter, you can return arbitrary text. You can now access http://localhost:8080/__tests/manhole/look?ID=0 and see whatever gets returned from renderHTTP (the value of ID obviously is arbitrary here, although you could use it to transmit information into your debugging code; not that we think that's a good idea).

Note that you can edit manhole.rd, save it, and reload the page; the server will notice your changes and display the new result without a restart.

Case Studies

In this section we will damage the arihip RD in various ways, show you how the errors manifest themselves, and how you could try and figure them out. It's highly recommended to play the scenarios; it's very well possible that the actual messages will look differently for you if we've changed and hopefully improved the software. We're thankful if you point us to outdated samples.

arihip > gavo imp q
*** Error: In /home/msdemlei/gavo/inputs/arihip/q.rd: mismatched tag:
line 674, column 2

/arihip > xmlstarlet val -e q.rd
q.rd:674.12: Opening and ending tag mismatch: meta line 72 and resource
</resource>
           ^
q.rd - invalid

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
     primary="hipno">
   <wonder>foo</wonder>

msdemlei@victor:/home/msdemlei/gavo/inputs/arihip > gavo imp q
*** Error: At /home/msdemlei/gavo/inputs/arihip/q.rd, (112, 4): table
elements have no wonder attributes or children.

<meta name="note" tag="tabflags"><![CDATA[

arihip > gavo imp q
*** Error: In /home/msdemlei/gavo/inputs/arihip/q.rd: not well-formed
(invalid token): line 411, column 24

<meta name="note" tag="src">
  The srcSel field indicates which catalogue the astrometric solution
was taken from using the following codes:

arihip > gavo --debug imp q
*** Error: At /home/msdemlei/gavo/inputs/arihip/q.rd, (317, 4): Bad
text in meta value (Bad indent in line u'    was taken from using the
following codes:')

<map dest="kbin">parsWithNull(@kbin, str, "9")</map>

arihip > gavo imp q
Making data import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
Failed /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
*** Error: Row {u'pmdeLTP': None, u'srcSel': 'T2H', u'err_raHIP':
[abridged]
.. .... ... ...', u'ddeLTP': '-     2.37', u'pmra_mas': '-    4.85',
u'raHIP': None} Field kbin: While building kbin in None: name
'parsWithNull' is not defined

<make table="main">
  <rowmaker idmaps="*" id="make_main_row">

Field kbin: While building kbin in make_main_row: name
'parsWithNull' is not defined

<map dest="vrad">parseWithNull(
   @vrad, lambda a:float(killBlanks(a)), "")</map>

/arihip > gavo imp -M 12 q
[...]
Field vrad: While building vrad in None: could not
convert string to float: +   8.3

vrad:      188-195

arihip > gavo imp -M 12 q
Making data import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
  Source hit import limit, import aborted.
Done /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz, read 13
Shipped 13/13
Create index Primary key on arihip.main
Create index main_mv
Create index main_q3c_main
Rows affected: 13

arihip > gavo info arihip/q#main
 col           min         avg     max       hasnulls
 [...]
 vrad          None        None    None      True
 [...]

<column name="vrad" ucd="phys.veloc;pos.heliocentric"
  required="True"
  tablehead="v_rad" verbLevel="20" unit="km/s"
  description="Radial velocity as used in calculating the foreshortening
  effect."/>

arihip > gavo imp -M 12 q
Making data import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
  Source hit import limit, import aborted.
Done /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz, read 1
*** Error: Row {u'pmdeLTP': None, u'srcSel': 'T2H', u'err_raHIP':
[...]
u'ddeLTP': '-     2.37', u'pmra_mas': '-    4.85', u'raHIP': None}
Field vrad: While building vrad in None: Key 'vrad' not found in a
mapping.

 <apply>
  <code>
    ddt
  </code>
</apply>

arihip > gavo imp q
[...]
u'raHIP': None} Field proc98577cc: While executing proc98577cc in
None: global name 'ddt' is not defined

<apply>
   <code>
     a = 1
     b = 2/a
     c = 3/(a+b)
     d = 4/(a+b+c+1)
     e = 5/d
   </code>
 </apply>

While executing proc985706c in None: integer division or modulo by zero

2013-09-23 15:57:37,304 [INFO 24430] Swallowed the exception below, re-raising Field proc985706c: While executing proc985706c in None: integer division or modulo by zero
Traceback (most recent call last):
  File ".../gavo/rscdef/rmkdef.py", line 583, in __call__
    exec self.code in self.globals, locals
  File "generated mapper code", line 13, in <module>
  File "<string>", line 9, in proc985706c
ZeroDivisionError: integer division or modulo by zero

<apply>
   <code>
     import pdb; pdb.set_trace()
     ...

Automatic and manual control

However, most condDescs correspond to one input key, and the input key is mostly derived from a table column. This is effected by the standard idiom:

<condDesc buildFrom="somecol"/>

where somecol is a column in the table queried by the core. This construct will cause the an input key to be built from somecol. While doing this, the type will be mapped automatically. The primary rules are:

• Numeric types will get mapped to numeric vizier-like expressions
• Datetimes will get mapped to date vizier-like expressions
• text and chars will get mapped to string vizier-like expressions
• enumerated values (i.e., columns with value elements giving options) will not become vizier-like expressions but input keys that yield selection widgets.

To have more control (e.g., if you do not want to allow vizier-like expressions, give the input key yourself):

<condDesc>
  <inputKey original="primaryId" required="False"/>
</condDesc>

(which would make a column required in the table optional in the query), or:

<condDesc>
  <inputKey name="specType" tablehead="Spectral Type"
    type="text" description="Spectral type of the target object">
</condDesc>

(which creates an input key matching everything literally), or even:

<condDesc>
  <inputKey name="color" type="text" required="True">
    <values multiOk="True">
      <option title="Red">R</option>
      <option title="Green">G</option>
      <option title="Blue">B</option>
    </values>
  </inputKey>
</condDesc>

-- if the input key is required, queries not giving it will be rejected. The title attribute on option gives the label of an option in the HTML input widget; if it's missing, a string representation of the value will be used.

In all those cases, the SQL generated from the condDesc is a conjunction of the input key's individual SQL expressions. Those, in turn, are simply comparisons for equality for plain types and more or less arbitrary expressions for vizier expression types.

Incidentally, two properties on inputKeys are defined to only show inputs for certain renderers, viz., onlyForRenderer and notForRenderer. Both have single strings as values. This is intended mainly for cases like SIAP and SCS where there are "human-oriented" versions of the input fields available. The built-in SCS and SIAP conditions already to that, so you can give both scs and humanSCS conditions in a core. Here is how you would define an input key that is only used for the form renderer:

<inputKey original="color">
  <property name="onlyForRenderer">form</property>
</inputKey>

Phrase makers

For complete control over what SQL is generated, condDescs may contain code called a phrase maker. This, again, is a procedure application, quite like with rowmaker procs, except that the signature of condDesc code is different.

Phrase maker code has the following names available:

• inputKeys -- the list of input keys for the parent CondDesc
• inPars -- a dictionary mapping inputKey names to the values provided by the user
• outPars -- a dictionary that is later used as the parameter dictionary to the query.

The code should amend the outPars dictionary with the keys mentioned in the conditions. The conditions themselves are yielded. So, a very simple condDesc with generated SQL could look like this:

<condDesc> <!-- don't do it like this, see below -->
  <inputKey name="val"/>
  <phraseMaker>
    <code>
      outPars["xxyy"] = "x"*inPars.get("val", 20)
      yield "someColumn=%(xxyy)s"
    </code>
  </phraseMaker>
</condDesc>

However, using fixed names in outPars is not recommended, if only because condDescs could be used multiple times. The recommended way uses the vizierexprs.getSQLKey function. It takes a name, a value, and the outPars dictionary. It will return a key unique to the query in question and enter the value into the outPars dictionary under that key. While that sounds complicated, it is actually rather harmless, as shown in the following real-world example that lets users input date, time and an interval in split-up form (e.g., when you cannot hope anyone will try to write the equivalent vizier-like expressions):

<condDesc>
  <inputKey name="date" tablehead="Date" type="date"
    multiplicity="single"
    required="True"/>
  <inputKey name="time" tablehead="Time (UTC)" type="time"
    multiplicity="single"
    required="True"/>
  <inputKey name="within" required="True" type="integer"
    multiplicity="single"
    tablehead="plus/minus" unit="minutes"
    description="Give measurements within this many minutes
     of your chosen date and time.  The sampling rate is 20 minutes">
    <values default="11"/>
  </inputKey>
  <phraseMaker>
    <code>
      baseTS = datetime.datetime.combine(inPars["date"], inPars["time"])
      dt = datetime.timedelta(minutes=inPars["within"])
      yield "date BETWEEN %%(%s)s AND %%(%s)s"%(
        vizierexprs.getSQLKey("date", baseTS-dt, outPars),
        vizierexprs.getSQLKey("date", baseTS+dt, outPars))
    </code>
  </phraseMaker>
</condDesc>

Tables

In principle, SCS can expose any table that has a exactly one column each with the UCDs pos.eq.ra;meta.main, pos.eq.dec;meta.main, and meta.id;meta.main, where the coordinates must be real or double precision, and the id must be either some integral type or text; the standard requires the id to be text, but the renderer will automatically convert integral types. The main query is then ran against the position specified in this way.

You almost always want to have a spatial index on these columns. To do that, use the //scs#q3cindex mixin on the tables, like this:

<table id="forSCS" onDisk="true" mixin="//scs#q3cindex"> ...

Finally, note that to have a valid SCS service, you must make sure the output table always contains the three required columns (as defined by the UCDs given above. To ensure that, these columns' verbLevel attribute must be 10 or less (we advise to have it at 1).

Cores

The SCS core simply is a dbCore. You must include the SCS condDesc, like this:

<dbCore queriedTable="main">
  <condDesc original="//scs#protoInput"/>
</dbCore>

There is an alternative condDesc more suitable for humans. They can be used in parallel. The form renderer will then use the human-oriented one, the DAL renderer the protocol one. You'll get this by writing:

<dbCore id="xlcore" queriedTable="main">
  <condDesc original="//scs#humanInput"/>
  <condDesc original="//scs#protoInput"/>
</dbCore>

Although not required by SCS, we recommend to also include a MAXREC argument that lets people change the match limit in the SCS service (for the web service, the database widget already provides this functionality). A usable definition for it is given in the SCS RD in a STREAM with the id coreDescs, together with the two condDescs above. So, here's the recommended way to build a bare-bone SCS service:

<dbCore id="xlcore" queriedTable="main">
  <FEED source="//scs#coreDescs"/>
</dbCore>

SCS allows more query parameters; you can usually use condDesc's buildFrom attribute to directly make one from an input column. If you want to add a larger number of them, you would use an active tag:

<dbCore id="xlcore" queriedTable="main">
  <condDesc original="//scs#humanInput"/>
  <condDesc original="//scs#protoInput"/>
  <LOOP listItems="ipix bmag rmag jmag pmra pmde">
    <events>
      <condDesc buildFrom="\item"/>
    </events>
  </LOOP>
</dbCore>

Note that most current SCS clients are not good at discovering them, since for SCS this requires going through the registry. In TOPCAT, for example, users would have to manually edit the cone search URL.

Service

To expose that core through a service, just allow the scs.xml renderer on it. As the core is built, you can have a web-based form interface for free:

<service id="cone" allowed="scs.xml,form">
  <meta name="title">Nice Catalog Cone Search</meta>
  <meta name="shortName">NC Cone</meta>
  <meta name="testQuery.ra">10</meta>
  <meta name="testQuery.dec">10</meta>
  <meta name="testQuery.sr">0.01</meta>
  <dbCore id="xlcore" queriedTable="main">
    <FEED source="//scs#coreDescs"/>
    <LOOP listItems="ipix bmag rmag jmag pmra pmde">
      <events>
        <condDesc buildFrom="\item"/>
      </events>
    </LOOP>
  </dbCore>
</service>

The meta information given is used when generating registration records. In particular, you should make sure that a query with the given ra, dec, and sr actually returns some data.

Quick Start

Check out a sample resource directory:

cd `dachs config inputsDir`
svn co http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/emi
cd emi
mkdir data

Now fetch some files to populate the data directory so you have something to import:

cd data
SRCURL=http://dc.g-vo.org/emi/q/s/siap.xml
curl -s $SRCURL"?POS=163.3,57.8&SIZE=20,20&MAXREC=5&weighting=uniform" \
  | tr '<TD>' '\n' \
  | grep "^http://" \
  | sed -e 's/<[^>]*>//g' \
  | xargs -n1 curl -sO

(no, this is not in general the way to operate SIAP services; use a proper client for real work, and we didn't show you this).

Run the import:

cd ..
gavo imp q

Now start the server as necessary (see above), and start TOPCAT and Aladin. In TOPCAT, open VO/SIA Query, enter your new service's access URL (it's http://localhost:8080/emi/q/s/siap.xml unless you did something cunning and should know better yourself) under "SIA URL" pretty far down in the dialog.

Then have "Lockman Hole" as Object Name and resolve it, or manually enter 161.25 and 58.0 as RA and Dec, respectively, and have 2 as Angular Size. Send off the request. You'll get back a table that you can send to Aladin (Interop/Send to/Aladin), which will respond by presenting a load dialog. Doubleclick and load as you like (yes, the images look a bit like static noise; that's all right here; but do combine these images with, say, DSS colored optical imagery and marvel at the wonders of modern VLBI interferometry).

Indicentally, we made the detour through TOPCAT since there's no nice UI to query non-registred SIAP services in Aladin.

Tables

SIAP-capable tables should mix in //siap#pgs. This mixin provides all the columns necessary for valid SIAP responses.

So, in the simplest case, a table that's going to be published through SIAP would look like this:

<table id="images" onDisk="True" mixin="//siap#pgs"/>

(of course, you can add more columns if you need them, and you might need metadata and all that, but this is all it really takes).

In the case of the Lockman Hole RD, things are a bit more verbose:

<table id="main" onDisk="True">
  <mixin>//siap#pgs</mixin>
  <mixin
    calibLevel="3"
    collectionName="'VLBA LH sources'"
    facilityName="'VLBA'"
    oUCD="'phot.flux.density;em:radio.750-1500MHz;phys.polarisation.Stokes.I'"
    polStates="'/I/'"
    targetName="object"
    tResolution="5000000"
    targetClass="'AGN'"
  >//obscore#publishSIAP</mixin>

  <column name="object" type="text"
    ucd="meta.id"
    tablehead="Object"
    description="Source name as in
      2009MNRAS.397..281I (VizieR J/MNRAS/397/281)"
    verbLevel="1"/>
  <column name="obsra"
    ucd="pos.eq.ra" unit="deg"
    description="Antenna pointing, RA"
    verbLevel="18"/>
  <column name="obsdec"
    ucd="pos.eq.dec" unit="deg"
    description="Antenna pointing, Dec"
    verbLevel="18"/>
  <column name="weighting" type="text"
    ucd="meta.code"
    description="Natural or uniform, according to weighting method."
    verbLevel="18">
    <values><option>natural</option><option>uniform</option></values>
  </column>
</table>

More on the obscore mixin below. otherwise, you see that additional, non-siap columns are simply added as usual. Anything with a verbLevel lower or equal 20 will be included in standard SIAP replies.

To fill such tables, the normal //products#define rowfilter is necessary, and there are two procDefs from //siap that come in handy:

<data id="import_main">
  <sources recurse="True">
    <pattern>data/*.fits</pattern>
  </sources>
  <fitsProdGrammar qnd="True">
    <maxHeaderBlocks>80</maxHeaderBlocks>
    <mapKeys>
      <map key="object">OBJECT</map>
      <map key="obsdec">OBSDEC</map>
      <map key="obsra">OBSRA</map>
    </mapKeys>
    <rowfilter procDef="//products#define">
      <bind key="table">"emi.main"</bind>
    </rowfilter>
  </fitsProdGrammar>

  <make table="main" >
    <rowmaker id="gen_rmk" idmaps="object, obsra, obsdec">
      <apply procDef="//siap#computePGS"/>
      <apply procDef="//siap#setMeta">
        <bind name="bandpassLo">0.207</bind>
        <bind name="bandpassHi">0.228</bind>
        <bind name="bandpassId">"1.4 GHz"</bind>
        <bind name="bandpassRefval">0.214</bind>
        <bind name="bandpassUnit">"m"</bind>
        <!-- since the images are fairly complex mosaics, there's no way we
        can have sensible dates; this one here "plays a special role
        in the calibration" (Middelberg) -->
        <bind name="dateObs"
          >dateTimeToMJD(datetime.datetime(2010, 7, 4))</bind>
        <bind name="instrument">@INSTRUME</bind>
        <bind name="title">"VLBA 1.4 GHz "+@object</bind>
      </apply>

      <apply name="fixObjectName">
        <setup>
          <code>
            import csv
            with open(rd.getAbsPath(
                "res/namemap.csv")) as f:
              nameMap = dict(csv.reader(f))
          </code>
        </setup>
        <code>
          @object = nameMap[@object]
        </code>
      </apply>

      <map key="weighting">\inputRelativePath.split("_")[-1][:-5]</map>
    </rowmaker>
  </make>
</data>

This does, step by step:

• The sources element is as always – with image collections, the recurse attribute usually comes in particularly handy.
• When ingesting images, you will (at least for a while, still) almost always read from FITS images, i.e., FITS primary headers. A fitsProdGrammar delivers the key-value-pairs from a header as a rawdict.
• The qnd attribute of the grammar is recommended. It makes some (weak) assumptions to yield significant speedups with large images, just so long as you can make do with the primary header.
• The fitsProdGrammar will map keys with hyphens to names with underscores, which allows for smoother action with them in rowmakers. The mapKey` element can produce additional mappings; in this case, we abuse it a bit to let us have idmaps (rather than simplemaps) in the rowmaker. And, actually, to illustrate the feature, as this data does not need that facility, really.
• The grammar further needs a rowfilter. The products#define rowfilter lets you add keys on owners and embargo in case you want password protection for images, but most importantly it defines what table the data is destined for. This is crucial information, and if you ever get it wrong, you need to manually connect to the database and issue a command like DELETE FROM products WHERE sourcetable='<your wrong table>'. So, always bind table. Make sure to include the quotes, this is supposed to be a valid python expression yielding a string.
• You then need to define a rowmaker that must apply two procs. For one, you need //siap#computePGS (if you mixed in //siap#pgsSIAP). No bindings are required here.
• The second proc application required is //siap#setMeta . Try to give all its keys somewhat sensible values, you will make your users' lives much easier.
• Typically, many values coming in the FITS headers will be messy and fouled up. You'll spend some quality time fixing these values in the typical case. Here, we translate somewhat broken object names using a simple mapping file that was provided by the author. In other circumstances there's the procs#mapValue procApp helping you.
• As is usual in procApps, you can access the embedding RD as rd. Here, we use that to let DaCHS find the input file independently of where the program was started.

Warning: Do not use idmaps="*" with SIAP, since the auto-generated mappings will clobber the work of the xSIAP procs.

Cores

There are two cores you may want for SIAP services:

• dbCore, to which you add the necessary condDescs manually as below, for "normal" SIAP services.
• siapCutoutCore, which speaks SIAP but returns cutouts rather than full images; the size of these cutouts is determined by the SIZE argument (i.e., the region of interest).

To furnish these cores with the parameters required by the standard, use the //siap#protoInput condDesc. If you want to re-use the core for a form-based service, use the //siap#humanInput condDesc as well. Both are written in a way that they'll sense if they run under a SIAP renderer or not.

So, a basic core with a couple of additional fields would look like this:

<dbCore id="query_images" queriedTable="main">
  <condDesc original="//siap#protoInput"/>
  <condDesc original="//siap#humanInput"/>
  <condDesc buildFrom="dateObs"/>
  <condDesc buildFrom="bandpassId" />
  <condDesc>
    <inputKey name="object" type="text"
        tablehead="Target Object"
        description="Object being observed, Simbad-resolvable form"
        ucd="meta.name" verbLevel="5" required="True">
        <values fromdb="object FROM lensunion.main"/>
    </inputKey>
  </condDesc>
</dbCore>

Service

If you wrote the core to work for both SIAP and form as described above, there's little more to say except you'll want to use the siap.xml renderer, and you need some additional metadata for VO registration. The latter is described in the siap.xml reference.

With this, the service definition would look like this:

<service id="im" allowed="form,siap.xml" core="query_image">
  <meta name="shortName">sample images</meta>
  <meta name="title">Sample Image Archive</meta>
  <meta name="sia.type">Pointed</meta>

  <meta name="testQuery.pos.ra">230.444</meta>
  <meta name="testQuery.pos.dec">52.929</meta>
  <meta name="testQuery.size.ra">0.1</meta>
  <meta name="testQuery.size.dec">0.1</meta>

  <publish render="siap.xml" sets="ivo_managed"/>
  <publish render="form" sets="local,ivo_managed"/>
</service>

(where again you can just write the above core inline rather than referencing it; that's the style we usually recommend).

SSAP Quick Start

Check out a sample resource directory into your inputs directory:

cd `dachs config inputsDir`
svn co http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/zcosmos
cd zcosmos
mkdir data

The checkout does not contain actual data, so let's fetch a file:

cd data
curl -O \
  http://dc.g-vo.org/getproduct/zcosmos/data/1D/zCOSMOS_BRIGHT_DR2_000826802_ZCMRa26_M1_Q4_13_1.fits
cd ..

This dataset needs obscore, and you should obscore-publish your spectra as well. So, make sure you have the obscore table:

dachs imp //obscore

Run the input and the regression tests:

dachs imp q
dachs test q

One regression test should fail since you've not yet pre-generated the previews (which are optional but recommended for your datasets, too):

python bin/makepreviews.py
dachs test q

If the regressions tests don't pass now, please complain to the authors of this tutorial.

From here on, you can point your favourite spectral client (fallback: TOPCAT's SSA client; note that TOPCAT itself cannot open this service's native format and you'd have to go through datalink, which TOPCAT can't yet do) to http://localhost:8080/zcosmos/q/ssa/ssap.xml and do your queries (if you don't know anything else, do a positional query for 149.734, 2.28216).

Please drop the dataset again when you're done playing:

dachs drop zcosmos/q

Since the dataset is in the obscore table, it would otherwise be globally discoverable, and that'd be bad.

The remainder of this chapter discusses what's in the RD. It is probably a good idea to have it open while reading this.

SSAP Tables

Tables open for SSAP querying need a certain structure. Use the //ssap#mixc mixin to have DaCHS fill it in:

<table id="data" onDisk="true">
  <mixin
    fluxUnit=" "
    fluxUCD="phot.flux.density"
    spectralUnit="Angstrom">//ssap#mixc</mixin>
  <mixin>//ssap#simpleCoverage</mixin>
</table>

As usual, you can define extra columns manually if necessary; the one you'll find in the actual RD is discussed below in SSAP and Datalink. There's also another mixin in what you checked out, which is discussed in SSAP Obscore Publication

This mixin has lots of parameters that define lots of things. Simply use the parameter list in the reference documentation (see the link above) as a checklist and leave out what you don't have. You must give a fluxUnit, fluxUCD and a spectralUnit, though, as they are used in the metadata of some of the columns. When your spectra are not flux calibrated, use a single blank as the unit.

Note that despite their name, the timeSI, fluxSI, and spectralSI mixin parameters do not contain actual unit strings. Instead, they are intended to contain conversion factors in a custom syntax called “Osuna-Salgado convention”. We recommend to not set these fields and instead provide useful VOUnit-compliant units where appropriate.

Building SSA Tables

SSAP tables contain products, so you'll (realistically) have to have a //products#define rowfilter in your grammar. Unless you're very sure you don't want that, also use the //ssap#setMeta and //ssap#setMixcMeta applys.

Unfortunately, there's no real standard for spectra that is widely used. If you're lucky, you'll get 1-D arrays in FITS format (IRAF liked to write such spectra; declare those with a media type of image/fits, since that's what they really are) or FITS tables with spectral/flux pairs (use a media type of application/fits for them).

Our example data set indeed comes as FITS, so we're using an Element fitsProdGrammar. The following bindings already prepare for making and serving previews, which is discussed in more detail in Product Previews in the DaCHS reference; see there for everything mentioning “preview”:

<data id="import">
  <property key="previewDir">previews</property>
  <sources pattern="data/1D/zCOSMOS_BRIGHT_DR2_*.fits"/>
  <fitsProdGrammar qnd="True">
    <rowfilter procDef="//products#define">
      <bind name="table">"\schema.data"</bind>
      <bind name="mime">"image/fits"</bind>
      <bind name="preview">\standardPreviewPath</bind>
      <bind name="preview_mime">"image/png"</bind>
    </rowfilter>
  </fitsProdGrammar>

The rowmaker uses the two applys mentioned above. Here, the spectral axis is along a WCS-described array axis. There's a nifty class handling the calculations with a factory function getWCSAxis. Otherwise, writing the rowmaker essentially means using the parameter lists of the applys as checklists. Here, this works out to:

<make table="data">
  <rowmaker idmaps="ssa_*">

    <var name="specAx">getWCSAxis(@header_, 1)</var>
    <var name="specLen">int(@NAXIS1)</var>

    <apply procDef="//ssap#setMeta">
      <bind name="dstitle">"%s %s"%(@TELESCOP, vars["ESO OBS ID"])</bind>
      <bind name="pubDID">\standardPubDID</bind>
      <bind name="targname">"ICRS %s %s"%(@RA, @DEC)</bind>
      <bind name="alpha">@RA</bind>
      <bind name="delta">@DEC</bind>
      <bind name="aperture">vars["ESO INS ADF SKYREG"]/3600.</bind>
      <bind name="cdate">@DATE</bind>
      <bind name="bandpass">"IR, Optical, UV"</bind>
      <bind name="targclass">"X"</bind>
      <bind name="dateObs">@DATE_OBS</bind>
      <bind name="timeExt">@EXPTIME</bind>
      <bind name="length">@specLen</bind>
      <bind name="specstart">@specAx.pixToPhys(1)*1e-10</bind>
      <bind name="specend">@specAx.pixToPhys(@specLen)*1e-10</bind>
    </apply>

    <apply procDef="//ssap#setMixcMeta">
      <bind name="collection">"zcosmos"</bind>
      <bind name="creationType">"archival"</bind>
      <bind name="dataSource">"survey"</bind>
      <bind name="fluxCalib">"RELATIVE"</bind>
      <bind name="specCalib">"ABSOLUTE"</bind>
      <bind name="binSize">2.5e-10</bind>
    </apply>
  </rowmaker>
</make>

Just make sure you do not forget the idmaps="ssa_*" at the top (you'll see rather opaque “key (something) not found in a mapping” messages if you do).

Caution: In the ssa table, the spectral axis must be a wavelength in meters. You must convert all values manually if necessary. For the spectra themselves you could use different units, but in our experience that's more confusing than helpful.

In contrast to images where delivering FITS is likely all you need, there's a plethora of formats spectra are delivered in. To help a bit, you should make sure one of the formats you offer are VOTables conforming to the spectral data model (see Making SDM Tables in the reference documentation and SSAP and Datalink below). If you want to deliver the “native” format as well, you'll have to have two rows for each spectrum. The standard way to achieve that is through a rowmaker in the grammar importing the spectra, like this:

<rowfilter name="generateFormatLinks">
  <code>
    baseAccref = os.path.splitext(row["prodtblPath"])[0]
    row["prodtblAccref"] = baseAccref
    row["prodtblMime"] = "image/fits"
    # this is the file as delivered from upstream
    yield row
    row["prodtblAccref"] = baseAccref+".vot"
    row["prodtblPath"] = \fullDLURL{sdl}
    row["prodtblMime"] = "application/x-votable+xml"
    # this is our processed SDM VOTable
    yield row
  </code>
</rowfilter>

(we don't do that here; if we did, you could have used a spectrum immediately in TOPCAT in the quick test above). For this to work, you will of course need a datalink service spitting out an appropriate VOTable with an id of sdl (the argument to the fullDLURL macro) in your RD – we will discuss this below.

SSAP's FORMAT parameter lets clients select what they want. The way the default FORMAT argument works, only application/x-votable+xml records are considered compliant.

SSAP Services

Use the element ssapCore for SSAP services. You must manually feed in the condition descriptors for the SSAP parameters; the simplest way to do that is to FEED the ssap#hcd_condDescs stream. Additionally, there is a set of SSA-specific metadata that is discussed with the ssap.xml renderer. The result for our example is:

<service id="ssa" allowed="form,ssap.xml">
  <meta name="shortName">zCosmos SSAP</meta>
  <meta name="title">zCosmos Bright
    Spectroscopic Observations DR2</meta>
  <meta name="ssap.dataSource">pointed</meta>
  <meta name="ssap.testQuery">MAXREC=1</meta>
  <meta name="ssap.creationType">archival</meta>
  <meta name="ssap.complianceLevel">query</meta>
  <publish render="ssap.xml" sets="ivo_managed"/>
  <publish render="form" sets="ivo_managed,local" service="web"/>

  <property name="datalink">sdl</property>

  <ssapCore queriedTable="data">
    <FEED source="//ssap#hcd_condDescs"/>
  </ssapCore>
</service>

The hcd_condDescs STREAM includes condition descriptors for all mandatory and optional parameters that we can remotely see in use.

Some of them may not be relevant to your service because your table never has values for them. For example, theoretical spectra will typically not give information on positions. The SSAP spec says that such a service should ignore POS rather than returning the empty set.

If you think you must ignore certain conditions, you can use the PRUNE active tag. This looks like this:

<ssapCore queriedTable="newdata">
  <FEED source="//ssap#hcd_condDescs">
    <PRUNE id="coneCond"/>
    <PRUNE id="bandCond"/>
  </FEED>
</ssapCore>

Do not do this just because you don't have position information – this would mean that you would dump your complete archive for (typical) queries with a position, and that is neither required by the spec (even if you might think so at first reading) nor desirable.

Here is a table of parameter names and ids; you can always check them by inspecting the output of dachs adm dumpDF //ssap:

For APERTURE, SNR, REDSHIFT, TARGETNAME, TARGETCLASS, PUBDID, CREATORDID, and MTIME, the condDesc id simply is <keyname>_cond, e.g., APERTURE_cond.

To have custom parameters, simply add condDesc elements as usual:

<ssapCore queriedTable="newdata">
  <FEED source="//ssap#hcd_condDescs"/>
  <condDesc buildFrom="t_eff"/>
</ssapCore>

For SSAP cores, buildFrom will enable “PQL”-like query syntax such that users can post arguments like 20000/30000,35000 to t_eff. This is in keeping with the general SSAP parameter style, while more modern VO services use 2-arrays for intervals.

To expose SSAP cores, use the ssap.xml renderer. Using the form renderer on SSAP cores is not terribly useful, because the core returns XML directly, and there are far too many parameters no human will ever be interested in anyway. So, you'll typically define extra browser-based services. The example RD shows a compact way to do that.

SSAP Obscore Publication

SSA tables are not terribly far from obscore tables, and so it's fairly easy to also publish the spectra through ObsTAP. Therefore, just do it by mixing in //obscore#publishSSAPMIXC. This just needs a calibLevel parameter, everything else is pre-set from SSAP or optional. Thus, the complete table definition for our example RD is:

<table id="data" onDisk="true">
  <mixin
    fluxUnit=" "
    fluxUCD="phot.flux.density"
    spectralUnit="Angstrom">//ssap#mixc</mixin>
  <mixin>//ssap#simpleCoverage</mixin>
</table>

SSAP and Datalink

Given that you'll usually get fairly bizarre inputs and will probably want to publish “repaired” spectra, using Datalink to provide both native and SDM (“Spectral Data Model”) compliant spectra without having to resort to SSAP's ill-thought-out FORMAT feature is a fairly natural thing to do. In DaCHS, that's not too hard; essentially, you have to write an embedded (or custom) grammar to parse the spectra. Other than that, it's just a few formalities.

So, you first define the table that will later hold your spectrum. Use the //ssap#sdm-instance mixin for that (all examples are still from zcosmos/q):

<table id="spectrum">
  <mixin ssaTable="data"
    fluxDescription="Relative Flux"
    spectralDescription="Wavelength"
    >//ssap#sdm-instance</mixin>
</table>

If your spectrum has additional columns (e.g., errors, noise estimates, bin widths), just put more columns in here. The mixin, in particular, pulls in all the various params that SDM wants to see.

Note that the table does not have onDisk="True"; these tables are only made for the brief moment it takes to serialise them into what the user receives.

As usual, to fill tables, you want a data element. In our example, this is:

<data id="build_sdm_data" auto="False">
  <embeddedGrammar>
    <iterator>
      <setup>
        <code>
          from gavo.protocols import products
          from gavo.utils import pyfits
        </code>
      </setup>
      <code>

      fitsPath = products.RAccref.fromString(
          self.sourceToken["accref"]).localpath
      hdus = pyfits.open(fitsPath)
      ax = utils.getWCSAxis(hdus[0].header, 1)

      for spec, flux in enumerate(hdus[0].data[0]):
        yield {"spectral": ax.pix0ToPhys(spec), "flux": flux}
      hdus.close()
      </code>
    </iterator>
  </embeddedGrammar>
  <make table="spectrum">
    <parmaker>
      <apply procDef="//ssap#feedSSAToSDM"/>
    </parmaker>
  </make>
</data>

Starting with the rowmaker: it's missing. DaCHS will then fill in the default one, which essentially is idmaps="*". Since you're writing the grammar from scratch, just use the names of the columns defined in the instance table and be done with it. The predefined column names are spectral and flux, so make sure you always have keys for them in the dictionaries you yield those from your grammars.

Then there's a Element parmaker; this is stereotypical, just always use the procDef as here. It copies values from the SSA input row to the params in the instance table.

The thing you have to vary is the code in the embedded grammar. It must, from its input (self.sourceToken, which here is the SSA row) figure out where the data file is (in this case, we can get it from the products table, which is the code with products.RAccref. More typically, just make a convention how to get from accref to the local file and use that. An example for that is in feros/q (the complication there is that the datalink products actually have accrefs themselves, and for those the path in the product table is the datalink URL; that, of course, wouldn't bring the grammar any closer to finding the file.

But you can do arbitrary things here; see the califa/q3 RD for an example for how you can decode the accref to database rows.

Once you have that just build dictionaries with at least spectral and flux keys; at least with the default rowmaker you need to make sure that the values match the units you define in the instance table.

Finally, write the service; you should at least have skimmed the endless chapter on Datalink and SODA in the reference documentation to get an idea of how descriptor generators, data functions, and meta makers play together:

<service id="sdl" allowed="dlget,dlmeta">
  <meta name="title">zCosmos Datalink Service</meta>

  <datalinkCore>
    <descriptorGenerator procDef="//soda#sdm_genDesc">
      <bind name="ssaTD">"\rdId#data"</bind>
    </descriptorGenerator>
    <dataFunction procDef="//soda#sdm_genData">
      <bind name="builder">"\rdId#build_sdm_data"</bind>
    </dataFunction>
    <FEED source="//soda#sdm_plainfluxcalib"/>
    <FEED source="//soda#sdm_cutout"/>
    <FEED source="//soda#sdm_format"/>
  </datalinkCore>
</service>

Datalink services (almost) always need to be enabled for both dlget and dlmeta. The //soda#sdm_genData data function needs a reference to the data element filling the instance table. The actual operations you can simply pull from the soda RD; sdm_plainfluxcalib offers a maximuum=1 calibration of the spectral values (which probably is not terribly useful for you), sdm_cutout does cutouts, which may come handy when you have long spectra, and sdm_format finally is the format translation that's frequently useful with spectra.

In particular as long as only few, if any, clients evaluate datalink blocks in DAL responses, it may be a good idea to include a custom column with a datalink URL in the SSAP table. If you've been following this exposition in the example RD, you will have noticed the custom column definition in the data table:

<column name="datalink" type="text"
  ucd="meta.ref.url;meta.data.datalink"
  tablehead="Datalink"
  description="A link to a datalink document for this spectrum."
  verbLevel="15" displayHint="type=url"/>

(we're cheating a bit with the UCD; it's actually not legal right now, but we're trying). To compute the datalink URL, there's again a helpful macro taking the datalink service id:

<map name="datalink">\dlMetaURI{sdl}</map>

DaCHS for now delivers XSLT with datalink (or does the XSLT translation itself if it suspects its client is actually a web brower), so there's no harm (but potentially quite a bit benefit) to feed this URL to tables displayed by web browsers, as we do here in the web service.

SSAP with Views

In a typical SSA table, most data will be constant. We don't consider this a large problem, as most collections are rather small, and so the memory overhead is negligible. However, in particular considering I/O bandwidth (Postgres is not a column store), you may still want to keep constant things out of the rows as your collections grow larger. We no longer support the old //ssap#hcd mixin that did something pretty much like this (albeit somewhat unflexibly). Instead, we recommend going through (non-materialised, or there'd be no benefit) views.

This works by importing your metadata into a custom table and then have the mixin-ed table be a view. The tricky part is to figure out how to write the view given that the columns are implict. Here's how to get around this. You can see the result in context in k2c9vst/q.

Warning: This introduces a fairly complex dependency scheme, and when dropping, DaCHS doesn't take into account dependencies yet. Thus, dropping the tables belonging to a service discribed here essentially needs to happen manually (gavo purge on the tables). We intend to fix this; complaints will make us do that quicker.

First, write a table definition and a null data item:

<table id="timeseries" onDisk="True">
        <mixin
                fluxUnit="adu"
                spectralUCD=" "
                spectralUnit=" "
                >//ssap#mixc</mixin>
</table>

<data id="make_ts">
        <make table="timeseries"/>
</data>

Do a metadata import of this – you need to import the table, or DaCHS will not find its metadata, and you should only import the metadata, or DaCHS will now create a table and later try to drop a view, which will give you an error (that you can fix by running dachs purge <tablename>):

$ dachs imp -m k2c9vst/q make_ts

Now make sure the server is running and open the metadata inspection page for your RD. In this case, the URL would be http://localhost:8080/browse/k2c9vst/q (it's always browse and the RD id). You'll see a link to the table, and you can see the columns.

Then, in the table, write a view statment. Always follow this pattern:

<table id="timeseries" onDisk="True">
        <mixin
                fluxUnit="adu"
                spectralUCD=" "
                spectralUnit=" "
                >//ssap#mixc</mixin>
        <viewStatement>
                CREATE VIEW \curtable AS (
                        SELECT \colNames FROM (
                                SELECT
                                  'http://foo.bar.baz/'||dataid as accref,
                                  'application/x-votable+xml'::text as mime,
                                  ...
                        ) AS q
                )
        </viewStatment>
</table>

This makes sure you're actually creating the view DaCHS is expecting (the \curtable macro), and the view has the structure declared in the RD (the \colNames macro). In the innermost select, you can then simply fill the columns one by one, in any order convenient to you as long as you get the names right.

Unless one of the joined tables mixes in products#table, you'll have to directly give complete URLs here rather than DaCHS accrefs (which always are resolved through the products table).

Note that you must give all types of literals explicitly in the view definition, either using CAST or the shortcut notation :: (when writing the view by hand, it doesn't really matter which one you use):

...
'LensingEv'::text as ssa_targclass,
CAST (NULL as real) as ssa_redshift,
...

If you want the result to be ADQL-searchable, tables used in the view must be accessible to an unprivileged database user. To do that, use adql="hidden" in the respective tables:

<table id="mymeta" onDisk="True" adql="hidden">
  ... your columns ...
</table>

Typcially, you'll still be importing source tables from somewhere. When you do this, the view will be torn down. You should make sure that DaCHS recreates the view when one of the tables making up the view is re-made. Simply declare the view-making data item in the importing data item as recreateAfter:

<data id="import" recreateAfter="make_ts">
  (normal, custom importing rules)
</data>

Datalinks in columns

The simplest way to let people discover datalink documents is by including links in normal tables (as for DAL tables) and keep the normal access URL in the accref column. To do that, define a column like this:

<column name="datalink" type="text"
  ucd="meta.ref.url"
  tablehead="DL"
  description="URL of a datalink document for this dataset"
  verbLevel="1" displayHint="type=url"/>

and then fill it in the corresponding rowmaker. For the normal case where a product corresponds to a disk file, there is a macro to help you:

<map key="datalink">\dlMetaURI{dl}</map>

Here, the “dl” in the macro argument must be the id of the datalink service as discussed below.

In such situations, it is still usually advisable to use the datalink URL rather than the naked accref in the obscore table – for instance, when you are using a SIAP cutout on large image (obscore cannot do cutouts) or to allow format conversions. To do that, you need to advise the obscore to use the datalink response; besides changing the access URL, you will also need to adjust the access_format (so clients know they will have to deal with a datalink) and the access_estsize (because they will not have to download the entire dataset in one go). For the latter, it is hard to predict how large the datalink stream will end up being. Guessing “about 10 kB” should be good enough, though.

In total, the obscore mixin would have to look like this:

 <mixin
  mime="'application/x-votable;content=datalink'"
  accessURL="datalink"
  size="10"
  ...
>//obscore#publish(WHATEVER)</mixin>

(in case you're wondering: the datalink given here for accessURL is just the column name as given above).

Datalinks as accrefs

In particular for large datasets, it is probably a good idea to keep people from blindly pulling the data without first having been made aware that what they're accessing is not just a little CCD frame. Having the datalink document as the primary document retrieved is a fairly good way to do that; of course, without a datalink-enabled client people might be locked out from the dataset entirely. On the other hand, DaCHS comes with a stylesheet that enables datalink operation from a common web brower, so that's perhaps not too bad.

To have datalinks rather than the plain dataset as what the accref points to, you need to change what DaCHS thinks of your dataset; this is what the //products#define rowfilter in your grammar is for:

<fitsProdGrammar qnd="True">
  <rowfilter procDef="//products#define">
    <bind key="path">\dlMetaURI{dl}</bind>
    <bind key="mime">'application/x-votable+xml;content=datalink'</bind>
    [...]
  </rowfilter>
  [...]
</fitsProdGrammar>

When you do this, you must use a datalink-aware descriptor generator. When you use the recommended setup, where the accref is the inputsDir-relative path to the main file, and you're dealing with FITS, you can use the DLFITSProductDescriptor class (in other cases, its implementation (in gavo.protocols.datalink) will show you how to handle these cases). Thus, the base functionality of a FITS cutout service with datalink products would be:

 <service id="dl" allowed="dlget,dlmeta">
  <meta name="title">My Cutout Service</meta>
  <datalinkCore>
    <descriptorGenerator procDef="//soda#fits_genDesc"
      name="genFITSDesc">
      <bind key="accrefPrefix">'mysvcs/data'</bind>
      <bind key="descClass">DLFITSProductDescriptor</bind>
    </descriptorGenerator>
    <FEED source="//soda#fits_standardDLFuncs"/>
  </datalinkCore>
</service>

Simple FITS Datalink Services

A fairly typical case for a datalink service is a file that has a few related datasets and perhaps a cutout. To define such a thing, write something like:

<service id="dl" allowed="dlget,dlmeta">
  <meta name="title">HDAP Datalink</meta>
  <meta name="description">This service lets you access cutouts
    from HDAP plates and retrieve scaled versions.</meta>
  <datalinkCore>
     <FEED source="//soda#fits_standardDLFuncs"
      accrefPrefix="lswscans"/>
    <metaMaker>
      <code>
        yield descriptor.makeLink(
          makeProductLink(descriptor.accref+"?scale=4"),
          contentType="image/fits",
          description="FITS, scaled by 1/4",
          semantics="#science",
          contentLength=descriptor.estimateSize()/16.)
      </code>
    </metaMaker>
  </datalinkCore>
</service>

TODO: Explain what's going on here.

TBC

Quick Start

Install PyPDS if you don't have it anyway:

curl -LO https://github.com/RyanBalfanz/PyPDS/archive/master.zip
unzip master.zip
cd PyPDS
python setup.py build
sudo python setup.py install

Get the sample data:

cd `dachs config inputsDir`
curl -O http://docs.g-vo.org/epntap-example.tar.gz
tar -xvzf epntap-example.tar.gz
cd lutetia

Import it and build the previews from the PDS images:

dachs imp q
python bin/makePreview.py

Start the server as necessary. If you go to your local ADQL endpoint (something like http://localhost:8080/adql) and execute queries like:

SELECT * FROM lutetia.epn_core

there.

For access through a standard protocol, start TOPCAT, select VO/TAP Query, and at the bottom of the dialog enter http://localhost:8080/tap (or whatever you configured) in “TAP URL”. Hit “Use Service”, wait until the table metadata is in and then again query something like:

SELECT * FROM lutetia.epn_core

Hit "Run Query",open the table and play with it. As a little visual treat, in TOPCAT's main window hit "Activation Action", and configure the preview_url column under “View URL as Image”. Then click on the table rows.

To get into Vespa's query interface, you will have to register your table. Do not do this with the sample data.

Tables

In essence, EPN-TAP is just a set of columns, some mandatory, some optional. The mandatory ones are pulled into a table by using the //epntap2#table-2_0 mixin, which needs the spatial_frame_type parameter (see reference for what's supported there) since that determines the metadata on the spatial columns.

This treatment distinguishes the two cases mentioned above: Files served by someone else first, files served by DaCHS next.

Externally Served Products

The mixin will already open the table for ADQL querying, and to make that work, it is also declared onDisk. What's not declared is the table name, although it's fixed at epn_core. Hence, the minimal definition of an EPN-TAP table is:

<table id="epn_core">
  <mixin
    spatial_frame_type="body"
  >//epntap2#table-2_0</mixin>
</table>

If you want to use optional columns from the EPN-TAP specification, give them in optional_columns. In the typcial case that you publish data products, you'll at least want the access columns, which would translate into this:

<table id="epn_core">
  <mixin
    optional_columns="access_url access_format access_estsize"
    spatial_frame_type="body"
  >//epntap2#table-2_0</mixin>
</table>

You can of course add further custom columns as necessary using the normal Element column.

Filling the table from what comes from the grammar happens through idmaps="*" on the rowmaker (which you want as the apply doesn't actually map anything into the results and there's too many columns to make enumerating fun) and an application of //epntap2#populate-2_0 . The mixin's documentation is intended as a checklist; go through the parameters one by one and see which ones correspond to something in your input and then say:

<par key="param-name">@source-key</par>,

possibly using python functions and expressions to adapt your inputs to what should end up in the database.

You may need to refer to the EPN-TAP proposed specification now and then while filling things out. Note again that these are python expressions, and so you have to use quotes when specifying literal strings.

More complex operations should probably go to var declarations rather than the bind bodies; again, see the examples to see how.

The real work is populating the table. Let's first assume you data products and use DaCHS to publish them. You will then use a grammar, and as EPN-TAP then deals with data products, the grammar must use the rowfilter products#define rowfilter as discussed above in SIAP; the lutetia example shows how this can be done while also providing for DaCHS-generated previews.

Externally Served Products

When you can store the data products you describe locally and have DaCHS hand them out, that's usually preferable. It also lets you do things like enforcing embargoes, make previews, etc.

To be able to manage the products, you need to add the //epntap2#localfile-2_0 mixin, as well as the optional columns starting with access_ in the table definition, so minimally:

<table id="epn_core">
  <mixin
    spatial_frame_type="celestial"
    optional_columns="access_url access_format access_estsize"
    >//epntap2#table-2_0</mixin>
  <mixin>//epntap2#localfile-2_0</mixin>
</table>

To let DaCHS manage the files, you need the //products#define rowfilter in your grammar. You will minimally have to tell it the table you're writing to (this is so DaCHS can remove the entries when you drop the table); since it by default assumes you're serving FITS files, which in planetary sciences is probably untrue, you should also give the media type explicitly, for instance:

<sources pattern="data/*.pds" recurse="True"/>
<pdsGrammar>
  <rowfilter procDef="//products#define">
    <bind name="table">"\schema.epn_core"</bind>
    <bind name="mime">"image/x-pds"</bind>
  </rowfilter>
</pdsGrammar>

(the \schema will just put in the current schema name defined at the top of your RD). This is also where you'd configure previews, embargos, etc.

Finally, in the rowmaker you need to make sure that the new columns are filled. That's done using a paramterless apply:

<rowmaker>
  ...
  <apply procDef="//epntap2#populate-localfile-2_0"/>
</rowmaker>

About s_region

The s_region parameter (see //epntap2#populate-2_0) is essentially a footprint describing the area covered by 2D spatially extended data products. It uses pgshpere types such as spoly, scircle, sbox or spoint (We advise against the use of spoint as a s_region type: only spatially extended types should be used). The default type is spoly, the others must be specified using the regiontype mixin parameter (Please refer to //epntap2#table-2_0).

Please note that the spoly type has some limitations that can trigger unclear errors when not respected: the segments can not be crossed and, more important, the maximum dimension of a polygon must be less than 180°.

Here are some example that shows how to implement it with a sbox:

First the regiontype:

<table id ="epn_core">
...
  <mixin ... regiontype="sbox">//epntap2#table-2_0</mixin>
</table>

Then the s_region parameter, defined in the rowmaker (see Mapping data):

<rowmaker idmaps="*">
  <map key = "s_region">
    pgsphere.SBox(
      pgsphere.SPoint.fromDegrees(float(@c1_min),float(@c2_min)),
      pgsphere.SPoint.fromDegrees(float(@c1_max),float(@c2_max)))
  </map>
</rowmaker>

If you want an “aperture” (point and radius), you'd write something like:

<table id ="epn_core">
...
  <mixin ... regiontype="scircle">//epntap2#table-2_0</mixin>
</table>

<rowmaker idmaps="*">
  <map key = "s_region">
    pgsphere.SCircle(
      pgsphere.SPoint.fromDegrees(float(@c1_min),float(@c2_min)),
      float(@aperture)*DEG)
  </map>
</rowmaker>

The radius of the circle must be given in radians here; if it's in degrees, just multiply with DaCHS' internal DEG symbol (which is pi/180).

Service

EPN-TAP tables are queried through the data center's TAP service. If you have registred that, there is nothing else you need to do to access your data.

For registration, just add:

<publish/>

to your table body and run gavo pub <rd-id>.