Defining Tables

A major part of the metadata DaCHS deals with is the table structure. It is defined in table elements, which usually are direct children of the resoource element. A resource element may contain multiple table definitions. See http://dc.g-vo.org/arihip/q/cone/static for what upstream documentation we had when we made the service.

Skip over the macDef elements for now to where the table element begins. What you see is something like:

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
  primary="hipno">

The id attribute of the table doubles as the name of the database table; make sure you use something that works as a valid simple SQL identifier (i.e., [A-Za-z_][A-Za-z0-9_]*) – DaCHS does not support for delimited identifiers as table names.

Be sure to always specify onDisk="True" unless you're going for special effects – without it, the table will end up only in memory. The adql attribute says that TAP queries should be allowed on the table; leave it out for tables not suitable for "raw" consumption by your clients.

For the mixin attribute, see Indices and Mixins.

Finally, using the primary attribute you can specify an explicit primary key of the table (if it is made up of several columns, concatenate their names with commas). This is made into a primary key for postgres straightforwardly, which means that the database makes sure there are no two rows with the same value for the primary key. Also, the database creates an index for efficient queries using the primary key.

What follows is a definition of the structure of the space-time coordinates:

<stc>
  Position ICRS Epoch J2000.0 "raj2000" "dej2000" Error "err_ra" "err_de"
    Velocity "pmra" "pmde" Error "err_pmra" "err_pmde"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raHIP" "deHIP" Error "err_raHIP" "err_deHIP"
    Velocity "pmraHIP" "pmdeHIP" Error "err_pmraHIP" "err_pmdeHIP"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raSTP" "deSTP" Error "err_raSTP" "err_deSTP"
    Velocity "pmraSTP" "pmdeSTP" Error "err_pmraSTP" "err_pmdeSTP"
</stc>
<stc>
  Position ICRS Epoch J2000.0 "raLTP" "deLTP" Error "err_raLTP" "err_deLTP"
    Velocity "pmraLTP" "pmdeLTP" Error "err_pmraLTP" "err_pmdeLTP"
</stc>

What this says is that there's a number of coordinate structures in the table, grouping together positions, errors, and proper motions in a specific reference frame. This is again a topic of its own, discussed in STC

<column name="hipno" type="integer" ucd="meta.id;meta.main"
  tablehead="HIP id" verbLevel="1"
  description="Number of the star in the HIPPARCOS Catalogue (ESA 1997)."
  required="True"/>
<column name="srcSel" type="text" ucd="meta.flag"
  tablehead="Source" verbLevel="25"
  description="Source of the astrometric solution"
  note="src"/>

<column name="n_obs" type="integer"
    description="Number of
    observations, NULL if interpolated data">
  <values nullLiteral="-1"/>
</column>

Parsing Input Data

Going further down in the RD, you will find a data element. Their main purpose is to describe how certain input files fill the table(s) defined above. It starts like this:

<data id="import">
  <sources>data/data.txt.gz</sources>

Data elements can have ids which can be used to indiviually reference them from a dachs imp command line; this is useful if you just want to import one part of a multi-table data collection. The default of dachs imp default is to build all data elements except those having an auto="False" attribute.

It is recommended that the id is a short verb phrase, as data basically contains instructions for an action. You might rightly argue that we have not chosen the element name too aptly (recipe might have been more appropriate), but we feel it's too late to change it now.

The sources element lets you specify the names of the input files to be processed. There are several ways to do that; in this case, there's just one input file, which is given as element content, with the path interpreted relative to the resource directory. If the data was distributed into several files in two directories, something like the following specification would do the trick:

<sources>
  <pattern>inp2/*.txt</pattern>
  <pattern>inp1/*.txt</pattern>
</sources>

The sources element also has a recurse (boolean) attribute that makes DaCHS search for the pattern in the subdirectories of the path part of the pattern.

Now have a look at the input file:

zless data/data.txt.gz

You'll see that we have a plain ASCII file with aligned columns, header lines, and even a cell separator ("|"). That's still a fairly common format for raw data, but by no means the only one. To give DaCHS the flexibility to deal with anything upstream cares to throw at you, DaCHS has the concept of a grammar.

There are many grammars defined, e.g., for getting values from FITS files, VOTables, or using column-based formats; you can also write specialized grammars in python. All grammars read "something" and emit a mapping from names to (mostly) string values; those unprocessed string-to-string mappings are called "rawdicts" in DaCHS jargon (distinguished from "rowdicts" ready for database ingestion, which sport processed and typed data).

It is often useful to inspect what a grammar emits. You can do that using import's --dump flag. During development, it is frequently convenient to just import a few rows and watch what they produce; this would look like this:

dachs imp -M 100 --dump q.rd | less

(if you interrrupt the above command, your table should be unscathed – check with dachs info –; otherwise just re-run the full import).

With a source that has both a separator character and aligned columns, there are several valid choice for which grammar to use on this particular file. In the RD, it next says:

<columnGrammar topIgnoredLines="9" preFilter="zcat">
  <colDefs>
    hipno:     3-8
    srcSel:    47-49
    alphaHMS:  59-73
    deltaDMS:  77-91
    pmra_mas:  95-103
    pmde_mas:  107-115
    t_ra_mod:  119-123
    err_ra_mas:127-131
    err_pmra_mas:135-139
    t_de_mod:  143-147
    err_de_mas:151-155
    err_pmde_mas:159-163
    parallax_mas:167-172
    e_parallax_mas:176-180
    kp:        184
    vrad:      188-195
    mv:        199-203
    km:        207
    kbin:      211-212
    kdmu:      216
    kae:       220
    flags:     882-901
  </colDefs>

– so we went for Element columngrammar. Those cut up every input line along character indices that are here, following the display in most editors, are counted from 1 upwards. Note that in ranges, the last column is included in the string – these are no python slices but basically a representation of the character ranges in VizieR-style "byte-by-byte"-descriptions.

The assignment of names to column ranges can happen both in a colDefs element as shown above – one specification per line, label mapped to a column or a range of columns.

Alternatively, have several col elements, each of which has a key attribute that gives a name. This could be the name of a target column in the simplest case, or it can be an auxillary identifier that is later processed in a rowmaker. These individual specifications are interesting when combined with RD macros, and that's where they come in in the arihip RD (again, using LOOPs).

Grammars also have various attributes; the ones parsing from text files support, for example, topIgnoredLines, which allows you to skip header lines, and preFilter that allows running the input through a shell command before it is processed using DaCHS (if you find yourself doing more than just decompression in such a preFilter, you should probably look for a different solution).

Mapping data

The arihip RD then goes on with:

<make table="main">
  <rowmaker idmaps="*">
    <var name="raj2000">hmsToDeg(@alphaHMS, None)</var>
    <var name="dej2000">dmsToDeg(@deltaDMS, None)</var>
    ...
    <map dest="kbin">parseWithNull(@kbin, str, "9")</map>
    <map dest="vrad">parseWithNull(
      @vrad, lambda a:float(killBlanks(a)), "")</map>

The make element brings together a table (in the table attribute) with a recipe how to fill it from the output of the grammar (the row maker).

Incidentally, there can be multiple make elements in a single data element if multiple tables (using different row makers) are generated from the same grammar output. This is particularly useful in combination with dispatching grammars that let, in effect, the grammar choose which make to use.

Makes can also carry scripts in SQL or python, at various points of the building process. These let you perform all kinds of higher magic. For details, see the chapter on scripting in the reference.

As explained above, output of grammars and hence the input to a make is a sequence of mappings from names to strings (the "rawdicts"). The database, on the other hand, wants typed values, i.e., integers, time stamps, etc. Also, data in input tables is frequently given in inconvenient formats (e.g., sexagesimal angles), deprecated or inconsistent units, or values may be distributed over multiple columns (e.g., date and time of an observation when we want a single timestamp). To cover these and more tasks, DaCHS has row makers, the results of which are then called rowdicts (note the subtle difference from the rawdicts coming in from the grammars: "row makers turn rawdicts into rowdicts").

Basically a row maker consists of

• var elements – assignments of expression values names in the rawdict.
• map elements – simple mappings of (python) expressions to values in the destination rowdict
• procedure applications (see apply) – manipulations of both rawdicts and rowdicts in python code

The fragment above shows one of several ways to use both var and map (which work exactly the same way, except that vars end up in the rawdict, and map in the rowdict): generating values from python expressions, where there is the special syntax @identifier which expands to whatever value the rawdict has for that key (or raises a KeyError if the key is not present in the rawdict).

The rawdict manipulations that var does are useful if you want to re-use whatever you compute. The map element, on the other hand, writes directly into the results dictionary, the keys of which directly correspond to the column names.

When building a rowdict for ingestion into the database, a row maker first binds var names, then applies procedures and finally performs the mappings. In the bodies of the mappings, you can use all built-in python functions plus a set of useful rowmaker functions documented in the reference documentation, as well as everything from the python standard library modules datetime, math, os, re, sys, time, and urllib (you need to give the module name when referring to names from these modules as in, e.g., re.sub). Furthermore, the gavo modules base, stc, and utils are in the namespace of the mapping code, as well as the submodule utils.pgsphere. TODO: link to useful documentation for them here.

For simple cases, maps will suffice; frequently, you can do without python expressions by giving a src attribute specifying a rawdict key instead of element content (which is preferable if possible – as elsewhere, less code is better in RDs, too). The rawdict string will in this case be converted to a typed value using "sane" defaults (e.g., integers will be converted by python's int constructor, where empty strings are mapped to None, datetimes are parsed as ISO strings, etc)

If you match the keys in the rawdicts with the names of the database columns their content is supposed to end up with and the content needs no further manipulations, a row maker like:

<rowmaker>
  <map dest="evi" src="evi"/>
  <map dest="av" src="av"/>
  <map dest="ai" src="ai"/>
</rowmaker>

would to the trick. Since this is a bit unwieldy, DaCHS provides a shortcut:

<rowmaker> <simplemaps>evi:evi,av:av,ai:ai</simplemaps> </rowmaker>

which expands to exactly what is written above. The keys in each pair do not need to be identical; the first item of each pair is the table column name, the second the rawdict key.

The case where the names of rawdict and rowdict keys are identical is so common (since the RD author in general controls both) that there is yet another shortcut for this:

<rowmaker>
  <idmaps>evi,av,ai</idmaps>
</rowmaker>

Idmaps sets up one map element each with both dest and src set to the value for every name in the comma separated list idmaps.

You can abbreviate this further to:

<rowmaker idmaps="*"/>

– so, idmaps values can contain shell patterns. They will be matched to the column names in the target table. For every column for which there is no explicit mapping, an identity mapping (with type conversion) will be set up with this specification.

Of course, you can have values that do not even depend on grammar output:

<map dest="dateIngested">datetime.datetime.now()</map>

Null values are always troublesome. Within DaCHS, the null value (almost) always is python's None. There is the row maker function parseWithNull to help you come up with those; if your upstream was devious enough to use 99.99 as a null value for a magnitude, you could say:

<map dest="Vmag">parseWithNull(@VmagSrc, float, "99.99")</map>

Note that the null value here is a literal matched against the string coming from the grammar; due to the rounding errors when converting from decimal to binary floating points, you can only safely compare against relatively few floating point numbers (99.99 is not among them), so you shouldn't do that if you can avoid it.

If you need to scale this (or if null values are chosen that they are invalid literals to begin with), a feature that lets you null out a value when an specific type of exception is raised comes in handy. This is map's nullExcs attribute, which is just a comma separated list of exceptions that should be caught and interpreted as "this is null". If, in the example above, the source would give the magnitude in millimags to save a comma, you could use:

<map dest="Vmag" nullExcs="TypeError"
  >parseWithNull(@VmagSrc, float, "99999")/1000.</map>

If parseWithNull here returns None, a TypeError will be raised and caught, and Vmag will be None.

You can turn more than one exception into None. For example example, if magicOffset has been parsed before and could be None, while magicLit is to be parsed and has the empty string as a NULL literal, you could write:

<map dest="magic" nullExcs="ValueError,TypeError"
  >@magicOffset+float(@magicLit)</map>

If magicOffset is None, magic will be None via the TypeError, whereas empty magicLits will result in Nones via a ValueError.

We defer the discussion of apply elements to the discussion of how to build SIAP services.

rowmaker elements may also be direct children of resource; this is for when they are used in more than one data. You would then give the rowmaker an id attribute and say something like <make rowmaker="id-of-rowmaker" table=.../> However, for the standard case it's best to keep everything related to a given import together in the make element.

Indices and Mixins

We have so far deferred the discussion of the mixin attribute in arihip's opening table element:

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
  primary="hipno">

Mixins are DaCHS' primary tool to endow tables with "everything needed to serve a standard" (e.g., a minimal set of columns, certain indices, or metadata). For instance, an image table must have a certain structure determined by the SIA protocol. The //siap#pgs mixin makes sure that tables have this structure, and it makes sure that the table containing information on all the file-like datasets in the data center (which is called dc.products) is updated when the table is filled.

The content of the mixin element (or the attribute value when you give the mixin property as an attribute) is a reference to a mixin definition. These references typically go into some system descriptor (though you could define your own mixins), and the double slash in a DaCHS reference means "system descriptor" (in actual truth, it's just an abbreviation for __system__/). The reference documentation contains a chapter on DaCHS' public mixins. For the curious: you can have a look at the actual definitions by admin's dumpDF subcommand, e.g., like this:

dachs admin dumpDF //scs

The //scs#q3cindex mixin referenced here arranges for spatial indexing of tables having some sort of spherical coordinates. To identify which columns to index, DaCHS inspects the UCDs of the columns; what it looks for here are columns with UCDs of pos.eq.(ra|dec);meta.main` as index columns. Contrary to mixins for other standard protocols, it does not automatically insert these columns (and neither the only other required column in SCS, the main row identifier with the UCD meta.id;meta.main).

Since in addition to spatial queries, we also expect a lot of queries constraining the mv column, we ask for an index on it using DaCHS' index element, which is a child of table:

<index columns="mv"/>

This is the simplest, but mostly sufficient, form of defining an index; for advanced usage, please refer to the reference documentation. If you decide to add an index to a table later on, or to initiate re-indexing after a lot of data changed, see the -I option of gavo imp.

The pos.eq UCDs in the SCS condDescs are hardcoded; this is because the standard simple cone search protocol specifies that the coordinates passed in are ICRS, and hence other systems – galactic, say – make little sense here. Also, the web form-variants of the protocols let users enter Simbad-resolvable identifiers rather than positions. In sum, making these things generic for other systems would be an unreasonable implementation effort.

If you want to allow queryies in other systems, just write the index statement inline, e.g., for galactic coordinates in the columns lambda, beta:

<index name="q3c_gal" cluster="True" columns="lambda,beta"
  >q3c_ang2ipix(lambda,beta)</index>

A simple condDesc that searches using this index could be:

<condDesc>
  <inputKey original="lambda" required="True"/>
  <inputKey original="beta" required="True"/>
  <inputKey name="sr" tablehead="Search Radius">
    <values default="0.001"/>
  </inputKey>
  <phraseMaker>
    yield q3c_radial_query(lambda, beta, %%(%s)s, "
      "%%(%s)s, %%(%s)s)")%(
      base.getSQLKey("lambda", inPars["lambda"], outPars),
      base.getSQLKey("beta", inPars["beta"], outPars),
      base.getSQLKey("sr", inPars["sr"], outPars))
  </phraseMaker>
</condDesc>

Cores and Services

The last but one part of the RD deals with how to get the data out of the database again, i.e., the services exposing the data. This part is fairly simple for arihip:

<service id="cone" allowed="scs.xml,form">
  <meta name="shortName">arihip cone</meta>
  <meta name="testQuery">
    <meta name="ra">9.4076</meta>
    <meta name="dec">9.6414</meta>
    <meta name="sr">1.0</meta>
  </meta>

  <dbCore queriedTable="main">
    <FEED source="//scs#coreDescs"/>
    <condDesc buildFrom="mv"/>
    <condDesc>
      <inputKey original="hipno" required="False"/>
    </condDesc>
  </dbCore>

  <publish render="scs.xml" sets="ivo_managed"/>
  <publish render="form" sets="ivo_managed,local"/>
  <outputTable verbLevel="20"/>
</service>

To understand what's going on here, some basic understanding of DaCHS' service architecture is required; it consists of:

• cores; they actually do the computation or database query
• renderers; these digest the data coming in from the service and (in general) format the result in some way requested by the user. There are renderers for web forms, VO protocols, imges, etc. Frequently – as in the example –, you can use the same core for both a VO protocol and a form-based service by just allowing different renderers.
• The service; it holds together the core and the renderer, can reformat core results, controls the metadata, etc.

The renderers are referenced by name in the service's allowed attribute. What can be given there (concatenated by commas) is listed in the reference documentation's renderer chapter. As you have seen above, the renderer is selected via the URL. If a client tries to retrieve a URL with a renderer that is not in the service's allowed list, DaCHS will respond with a 403 forbidden HTTP code (excepting certain "unchecked" renderers like info that typically expose service metadata). Note that some of the more exotic renderers may pose special requirements on cores, so don't be too disappointed of some exotic renderer won't work with your core.

The most common core for catalog services (and the one you'll typically use for SCS services) is the dbCore, as used here. See cores available in the reference documentation for more predefined cores -- e.g., to run ADQL queries or to upload files. For special functionality, you can even write your own core.

The dbCore generates a (single-table) query from condition descriptors and returns a table that you describe through an output table. Cores are defined as direct children of the resource (as with grammars, you can also have them in resource and then write core="id-of-element", which makes sense when a single core is shared by several services).

dbCores need a queriedTable attribute, the value of which must be a table reference. This is the table the query will run against.

The condition descriptors (or condDescs for short) define input fields (for the form renderer, these will be rendered as form items people can fill in). Most commonly, you will either define them using the original attribute (when inheriting from predefined condDescs) or using buildFrom. The first case is typically used in connection with protocols and on tables having mixins; such condDescs result in zero or more input fields, and they typically inspect the queried table. For example, the //scs#humanScs condDesc locates the "main" positions as identified by UCDs and generates queries against them using two input fields, one it tries to guess a position from, and another for the search radius.

When you define a condDesc using buildFrom, the result is usually one or more input field(s) constraining values in the column named in the buildFrom attribute. The software tries to make some useful input definition from that column, depending on the renderer. Renderers with a parameter style (this is given in each renderer's description the reference documentation) of "form", for example, let users query string-like columns using Vizier-like string expressions, real and double precision columns using Vizier-like float expressions, and so on. The pql style allows a specification like for some SSAP parameters (e.g., "range_min/range_max").

The service element must have an id attribute that is used to select the service run in the access URL. Furthermore, there should be certain pieces of metadata useful in later registration. First, there's shortName, which is typically used by clients in space-restricted displays. It must not be longer than 16 characters, so something like an acronym and a very terse role identifier is the best you can do (hence the "arihip cone" here). Frequently, a title meta is also useful, in particular when an RD contains multiple services, in which case one could be "Cone search for ARI's HIPPARCOS re-reduction" and the other, say, "Autocorrelation on ARI's HIPPARCOS re-reduction".

See data_checklist.html for more information on useful generic metadata and remember that services inherit whatever is defined within resource when not specified within the element.

Many standard VO protocols require additional, protocol-specific metadata. In the case of arihip, we have a Simple Cone Search service, which, as laid down in the section on the scs.xml renderer in the reference documentation, requires the parameters of a test query returning a nonempty result.

General Hints

• If you get error messages, be sure to check our hints on common problems – may commonly encountered problems are explained there with suggestions for how to fix them.

• The dachs command takes a --hints switch. With it, error messages are frequently accompanied with – you guessed it – hints on what might cause the problem and possible solutions. Note that --hints, like the other debugging switches, goes between gavo and the subcommand name, as in dachs --hints serve.

• Validate your RD. This is, in general, a good idea before doing anything with the RD, since it will allow you to more easily catch errors than the in all likelihood even more byzantine error messages that may arise when something goes wrong later. The dachs val subcommand takes one or more RDs. If you don't understand its output, complain to gavo@ari.uni-heidelberg.de – the command is really intended to help you catch errors, and if it doesn't do so, it's a bug.

• Check the logs. They are in /var/gavo/logs by default, and there's dcErrors and dcInfos (you'll want to look at both).

• When hunting bugs, it's usually a good idea to enable the logging of (many) tracebacks by passing the --debug flag to gavo.

• If you're trying to figure out server behaviour, don't run the server daemonized but use dachs serve debug instead; this won't detach and log to stdout.

• With this (or the problem is in normally-running DaCHS code in the first place), the python debugger is your friend. The gavo command has an option --enable-pdb that will dump you into the debugger where an uncaught exception happened (as the server should never let an uncaught exception through, that's not useful with dachs serve). If that doesn't help you, you can set breakpoints (e.g., in your own in-RD procedure defininitions by writing import pdb; pdb.set_trace. If you install the python-ipdb package, and write ipdb instead of pdb, you'll get a nicer debugger commandline with tab completion and similar frills.

• To see what SQL is actually sent to the database, set the GAVO_SQL_DEBUG environment variable to any value. This could look like this:

  env GAVO_SQL_DEBUG=1 dachs imp q create
  

  The first couple of requests are for internal use (like checking that some meta tables are present).

• If dachs serve start doesn't actually cause the server to run, something went wrong after detaching from the controlling terminal. The messages are in the logs directory, in the file serverStderr.

Publication Options

TODO: registry chapter, purx, eschrow

Debugging Services

Debugging services is of course an extra challenge since code runs deep in the bowels of a complex system, with timeouts, threads, and all kinds of nastyness involved. The first adivce is: Pull whatever is mysterious into your development system and run dachs serve debug. You can even do the import pdb;pdb.set_trace() trick then. It's a bit tricky to communicate with the debugger in between the log messages of gavo serve debug, and there's no readline support since pdb doesn't think it's running within a terminal there, but it's definitely doable.

Another challenge is that sometimes problems manifest themselves in a running server. In that case it's sometimes useful to open a manhole into the server. One reasonably convenient way to do this is to but a special RD somewhere (e.g., __tests/manhole.rd) and use some RD-embedded code to introspect the server. We ususally use a datalink service for this since it keeps things nicely self-contained – an obvious alternative with less bending could be a custom renderer.

Here's how something that fiddles out a column property would look like:

<resource schema="test">
  <service id="look" allowed="dlget">
    <datalinkCore>
      <descriptorGenerator>
        <code>
          return ProductDescriptor(None, None, None, 'text/plain', )
        </code>
      </descriptorGenerator>
      <dataFunction>
        <code>
          descriptor.data = "debugging"
        </code>
      </dataFunction>
      <dataFormatter>
        <code>
          class DebugResult(Page):
            def renderHTTP(self, ctx):
              request = IRequest(ctx)
              request.setHeader("content-type", "text/plain")
              return "%s"%base.caches.getRD("maidanak/res/rawframes"
                ).getById("rawframes").getColumnByName("accref").displayHint
          return DebugResult()
        </code>
      </dataFormatter>
    </datalinkCore>
  </service>
</resource>

All but the data formatter is just blind code for hiding our true intentions from the datalink machinery. In the data formatter, you can return arbitrary text. You can now access http://localhost:8080/__tests/manhole/look?ID=0 and see whatever gets returned from renderHTTP (the value of ID obviously is arbitrary here, although you could use it to transmit information into your debugging code; not that we think that's a good idea).

Note that you can edit manhole.rd, save it, and reload the page; the server will notice your changes and display the new result without a restart.

Case Studies

In this section we will damage the arihip RD in various ways, show you how the errors manifest themselves, and how you could try and figure them out. It's highly recommended to play the scenarios; it's very well possible that the actual messages will look differently for you if we've changed and hopefully improved the software. We're thankful if you point us to outdated samples.

arihip > dachs imp q
*** Error: In /home/msdemlei/gavo/inputs/arihip/q.rd: mismatched tag:
line 674, column 2

/arihip > xmlstarlet val -e q.rd
q.rd:674.12: Opening and ending tag mismatch: meta line 72 and resource
</resource>
           ^
q.rd - invalid

<table id="main" onDisk="True" adql="True" mixin="//scs#q3cindex"
     primary="hipno">
   <wonder>foo</wonder>

msdemlei@victor:/home/msdemlei/gavo/inputs/arihip > dachs imp q
*** Error: At /home/msdemlei/gavo/inputs/arihip/q.rd, (112, 4): table
elements have no wonder attributes or children.

<meta name="note" tag="tabflags"><![CDATA[

arihip > dachs imp q
*** Error: In /home/msdemlei/gavo/inputs/arihip/q.rd: not well-formed
(invalid token): line 411, column 24

<meta name="note" tag="src">
  The srcSel field indicates which catalogue the astrometric solution
was taken from using the following codes:

arihip > dachs --debug imp q
*** Error: At /home/msdemlei/gavo/inputs/arihip/q.rd, (317, 4): Bad
text in meta value (Bad indent in line u'    was taken from using the
following codes:')

<map dest="kbin">parsWithNull(@kbin, str, "9")</map>

arihip > dachs imp q
Making data import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
Failed /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
*** Error: Row {u'pmdeLTP': None, u'srcSel': 'T2H', u'err_raHIP':
[abridged]
.. .... ... ...', u'ddeLTP': '-     2.37', u'pmra_mas': '-    4.85',
u'raHIP': None} Field kbin: While building kbin in None: name
'parsWithNull' is not defined

<make table="main">
  <rowmaker idmaps="*" id="make_main_row">

Field kbin: While building kbin in make_main_row: name
'parsWithNull' is not defined

<map dest="vrad">parseWithNull(
   @vrad, lambda a:float(killBlanks(a)), "")</map>

/arihip > dachs imp -M 12 q
[...]
Field vrad: While building vrad in None: could not
convert string to float: +   8.3

vrad:      188-195

arihip > dachs imp -M 12 q
Making data import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
  Source hit import limit, import aborted.
Done /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz, read 13
Shipped 13/13
Create index Primary key on arihip.main
Create index main_mv
Create index main_q3c_main
Rows affected: 13

arihip > dachs info arihip/q#main
 col           min         avg     max       hasnulls
 [...]
 vrad          None        None    None      True
 [...]

<column name="vrad" ucd="phys.veloc;pos.heliocentric"
  required="True"
  tablehead="v_rad" verbLevel="20" unit="km/s"
  description="Radial velocity as used in calculating the foreshortening
  effect."/>

arihip > dachs imp -M 12 q
Making data import
Starting /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz
  Source hit import limit, import aborted.
Done /home/msdemlei/gavo/inputs/arihip/data/data.txt.gz, read 1
*** Error: Row {u'pmdeLTP': None, u'srcSel': 'T2H', u'err_raHIP':
[...]
u'ddeLTP': '-     2.37', u'pmra_mas': '-    4.85', u'raHIP': None}
Field vrad: While building vrad in None: Key 'vrad' not found in a
mapping.

 <apply>
  <code>
    ddt
  </code>
</apply>

arihip > dachs imp q
[...]
u'raHIP': None} Field proc98577cc: While executing proc98577cc in
None: global name 'ddt' is not defined

<apply>
   <code>
     a = 1
     b = 2/a
     c = 3/(a+b)
     d = 4/(a+b+c+1)
     e = 5/d
   </code>
 </apply>

While executing proc985706c in None: integer division or modulo by zero

2013-09-23 15:57:37,304 [INFO 24430] Swallowed the exception below, re-raising Field proc985706c: While executing proc985706c in None: integer division or modulo by zero
Traceback (most recent call last):
  File ".../gavo/rscdef/rmkdef.py", line 583, in __call__
    exec self.code in self.globals, locals
  File "generated mapper code", line 13, in <module>
  File "<string>", line 9, in proc985706c
ZeroDivisionError: integer division or modulo by zero

<apply>
   <code>
     import pdb; pdb.set_trace()
     ...

Automatic and manual control

However, most condDescs correspond to one input key, and the input key is mostly derived from a table column. This is effected by the standard idiom:

<condDesc buildFrom="somecol"/>

where somecol is a column in the table queried by the core. This construct will cause the an input key to be built from somecol. While doing this, the type will be mapped automatically. The primary rules are:

• Numeric types will get mapped to numeric vizier-like expressions
• Datetimes will get mapped to date vizier-like expressions
• text and chars will get mapped to string vizier-like expressions
• enumerated values (i.e., columns with value elements giving options) will not become vizier-like expressions but input keys that yield selection widgets.

To have more control (e.g., if you do not want to allow vizier-like expressions, give the input key yourself):

<condDesc>
  <inputKey original="primaryId" required="False"/>
</condDesc>

(which would make a column required in the table optional in the query), or:

<condDesc>
  <inputKey name="specType" tablehead="Spectral Type"
    type="text" description="Spectral type of the target object">
</condDesc>

(which creates an input key matching everything literally), or even:

<condDesc>
  <inputKey name="color" type="text" required="True">
    <values multiOk="True">
      <option title="Red">R</option>
      <option title="Green">G</option>
      <option title="Blue">B</option>
    </values>
  </inputKey>
</condDesc>

-- if the input key is required, queries not giving it will be rejected. The title attribute on option gives the label of an option in the HTML input widget; if it's missing, a string representation of the value will be used.

In all those cases, the SQL generated from the condDesc is a conjunction of the input key's individual SQL expressions. Those, in turn, are simply comparisons for equality for plain types and more or less arbitrary expressions for vizier expression types.

Incidentally, two properties on inputKeys are defined to only show inputs for certain renderers, viz., onlyForRenderer and notForRenderer. Both have single strings as values. This is intended mainly for cases like SIAP and SCS where there are "human-oriented" versions of the input fields available. The built-in SCS and SIAP conditions already to that, so you can give both scs and humanSCS conditions in a core. Here is how you would define an input key that is only used for the form renderer:

<inputKey original="color">
  <property name="onlyForRenderer">form</property>
</inputKey>

Phrase makers

For complete control over what SQL is generated, condDescs may contain code called a phrase maker. This, again, is a procedure application, quite like with rowmaker procs, except that the signature of condDesc code is different.

Phrase maker code has the following names available:

• inputKeys – the list of input keys for the parent CondDesc
• inPars – a dictionary mapping inputKey names to the values provided by the user
• outPars – a dictionary that is later used as the parameter dictionary to the query.

The code should amend the outPars dictionary with the keys mentioned in the conditions. The conditions themselves are yielded. So, a very simple condDesc with generated SQL could look like this:

<condDesc> <!-- don't do it like this, see below -->
  <inputKey name="val"/>
  <phraseMaker>
    <code>
      outPars["xxyy"] = "x"*inPars.get("val", 20)
      yield "someColumn=%(xxyy)s"
    </code>
  </phraseMaker>
</condDesc>

However, using fixed names in outPars is not recommended, if only because condDescs could be used multiple times. The recommended way uses the vizierexprs.getSQLKey function. It takes a name, a value, and the outPars dictionary. It will return a key unique to the query in question and enter the value into the outPars dictionary under that key. While that sounds complicated, it is actually rather harmless, as shown in the following real-world example that lets users input date, time and an interval in split-up form (e.g., when you cannot hope anyone will try to write the equivalent vizier-like expressions):

<condDesc>
  <inputKey name="date" tablehead="Date" type="date"
    multiplicity="single"
    required="True"/>
  <inputKey name="time" tablehead="Time (UTC)" type="time"
    multiplicity="single"
    required="True"/>
  <inputKey name="within" required="True" type="integer"
    multiplicity="single"
    tablehead="plus/minus" unit="minutes"
    description="Give measurements within this many minutes
     of your chosen date and time.  The sampling rate is 20 minutes">
    <values default="11"/>
  </inputKey>
  <phraseMaker>
    <code>
      baseTS = datetime.datetime.combine(inPars["date"], inPars["time"])
      dt = datetime.timedelta(minutes=inPars["within"])
      yield "date BETWEEN %%(%s)s AND %%(%s)s"%(
        vizierexprs.getSQLKey("date", baseTS-dt, outPars),
        vizierexprs.getSQLKey("date", baseTS+dt, outPars))
    </code>
  </phraseMaker>
</condDesc>

Tables

In principle, SCS can expose any table that has a exactly one column each with the UCDs pos.eq.ra;meta.main, pos.eq.dec;meta.main, and meta.id;meta.main, where the coordinates must be real or double precision, and the id must be either some integral type or text; the standard requires the id to be text, but the renderer will automatically convert integral types. The main query is then ran against the position specified in this way.

You almost always want to have a spatial index on these columns. To do that, use the //scs#q3cindex mixin on the tables, like this:

<table id="forSCS" onDisk="true" mixin="//scs#q3cindex"> ...

Finally, note that to have a valid SCS service, you must make sure the output table always contains the three required columns (as defined by the UCDs given above. To ensure that, these columns' verbLevel attribute must be 10 or less (we advise to have it at 1).

Cores

The SCS core simply is a dbCore. You must include the SCS condDesc, like this:

<dbCore queriedTable="main">
  <condDesc original="//scs#protoInput"/>
</dbCore>

There is an alternative condDesc more suitable for humans. They can be used in parallel. The form renderer will then use the human-oriented one, the DAL renderer the protocol one. You'll get this by writing:

<dbCore id="xlcore" queriedTable="main">
  <condDesc original="//scs#humanInput"/>
  <condDesc original="//scs#protoInput"/>
</dbCore>

Although not required by SCS, we recommend to also include a MAXREC argument that lets people change the match limit in the SCS service (for the web service, the database widget already provides this functionality). A usable definition for it is given in the SCS RD in a STREAM with the id coreDescs, together with the two condDescs above. So, here's the recommended way to build a bare-bone SCS service:

<dbCore id="xlcore" queriedTable="main">
  <FEED source="//scs#coreDescs"/>
</dbCore>

SCS allows more query parameters; you can usually use condDesc's buildFrom attribute to directly make one from an input column. If you want to add a larger number of them, you would use an active tag:

<dbCore id="xlcore" queriedTable="main">
  <FEED source="//scs#coreDescs"/>
  <LOOP listItems="ipix bmag rmag jmag pmra pmde">
    <events>
      <condDesc buildFrom="\item"/>
    </events>
  </LOOP>
</dbCore>

Note that most current SCS clients are not good at discovering them, since for SCS this requires going through the registry. In TOPCAT, for example, users would have to manually edit the cone search URL.

Service

To expose that core through a service, just allow the scs.xml renderer on it. As the core is built, you can have a web-based form interface for free:

<service id="cone" allowed="scs.xml,form">
  <meta name="title">Nice Catalog Cone Search</meta>
  <meta name="shortName">NC Cone</meta>
  <meta name="testQuery.ra">10</meta>
  <meta name="testQuery.dec">10</meta>
  <meta name="testQuery.sr">0.01</meta>
  <dbCore id="xlcore" queriedTable="main">
    <FEED source="//scs#coreDescs"/>
    <LOOP listItems="ipix bmag rmag jmag pmra pmde">
      <events>
        <condDesc buildFrom="\item"/>
      </events>
    </LOOP>
  </dbCore>
</service>

The meta information given is used when generating registration records. In particular, you should make sure that a query with the given ra, dec, and sr actually returns some data.

Quick Start

Check out a sample resource directory:

cd `dachs config inputsDir`
svn co http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/emi
cd emi
mkdir data

Now fetch some files to populate the data directory so you have something to import:

cd data
SRCURL=http://dc.g-vo.org/emi/q/s/siap.xml
curl -s $SRCURL"?POS=163.3,57.8&SIZE=20,20&MAXREC=5&weighting=uniform" \
  | tr '<TD>' '\n' \
  | grep "^http://" \
  | sed -e 's/<[^>]*>//g' \
  | xargs -n1 curl -sO

(no, this is not in general the way to operate SIAP services; use a proper client for real work, and we didn't show you this).

Run the import:

cd ..
dachs imp q

Now start the server as necessary (see above), and start TOPCAT and Aladin. In TOPCAT, open VO/SIA Query, enter your new service's access URL (it's http://localhost:8080/emi/q/s/siap.xml unless you did something cunning and should know better yourself) under "SIA URL" pretty far down in the dialog.

Then have "Lockman Hole" as Object Name and resolve it, or manually enter 161.25 and 58.0 as RA and Dec, respectively, and have 2 as Angular Size. Send off the request. You'll get back a table that you can send to Aladin (Interop/Send to/Aladin), which will respond by presenting a load dialog. Doubleclick and load as you like (yes, the images look a bit like static noise; that's all right here; but do combine these images with, say, DSS colored optical imagery and marvel at the wonders of modern VLBI interferometry).

Indicentally, we made the detour through TOPCAT since there's no nice UI to query non-registred SIAP services in Aladin.

Tables

SIAP-capable tables should mix in //siap#pgs. This mixin provides all the columns necessary for valid SIAP responses.

So, in the simplest case, a table that's going to be published through SIAP would look like this:

<table id="images" onDisk="True" mixin="//siap#pgs"/>

(of course, you can add more columns if you need them, and you might need metadata and all that, but this is all it really takes).

In the case of the Lockman Hole RD, things are a bit more verbose:

<table id="main" onDisk="True">
  <mixin>//siap#pgs</mixin>
  <mixin
    calibLevel="3"
    collectionName="'VLBA LH sources'"
    facilityName="'VLBA'"
    oUCD="'phot.flux.density;em:radio.750-1500MHz;phys.polarisation.Stokes.I'"
    polStates="'/I/'"
    targetName="object"
    tResolution="5000000"
    targetClass="'AGN'"
  >//obscore#publishSIAP</mixin>

  <column name="object" type="text"
    ucd="meta.id"
    tablehead="Object"
    description="Source name as in
      2009MNRAS.397..281I (VizieR J/MNRAS/397/281)"
    verbLevel="1"/>
  <column name="obsra"
    ucd="pos.eq.ra" unit="deg"
    description="Antenna pointing, RA"
    verbLevel="18"/>
  <column name="obsdec"
    ucd="pos.eq.dec" unit="deg"
    description="Antenna pointing, Dec"
    verbLevel="18"/>
  <column name="weighting" type="text"
    ucd="meta.code"
    description="Natural or uniform, according to weighting method."
    verbLevel="18">
    <values><option>natural</option><option>uniform</option></values>
  </column>
</table>

More on the obscore mixin below. otherwise, you see that additional, non-siap columns are simply added as usual. Anything with a verbLevel lower or equal 20 will be included in standard SIAP replies.

To fill such tables, the normal //products#define rowfilter is necessary, and there are two procDefs from //siap that come in handy:

<data id="import_main">
  <sources recurse="True">
    <pattern>data/*.fits</pattern>
  </sources>
  <fitsProdGrammar qnd="True">
    <maxHeaderBlocks>80</maxHeaderBlocks>
    <mapKeys>
      <map key="object">OBJECT</map>
      <map key="obsdec">OBSDEC</map>
      <map key="obsra">OBSRA</map>
    </mapKeys>
    <rowfilter procDef="//products#define">
      <bind key="table">"emi.main"</bind>
    </rowfilter>
  </fitsProdGrammar>

  <make table="main" >
    <rowmaker id="gen_rmk" idmaps="object, obsra, obsdec">
      <apply procDef="//siap#computePGS"/>
      <apply procDef="//siap#setMeta">
        <bind name="bandpassLo">0.207</bind>
        <bind name="bandpassHi">0.228</bind>
        <bind name="bandpassId">"1.4 GHz"</bind>
        <bind name="bandpassRefval">0.214</bind>
        <bind name="bandpassUnit">"m"</bind>
        <!-- since the images are fairly complex mosaics, there's no way we
        can have sensible dates; this one here "plays a special role
        in the calibration" (Middelberg) -->
        <bind name="dateObs"
          >dateTimeToMJD(datetime.datetime(2010, 7, 4))</bind>
        <bind name="instrument">@INSTRUME</bind>
        <bind name="title">"VLBA 1.4 GHz "+@object</bind>
      </apply>

      <apply name="fixObjectName">
        <setup>
          <code>
            import csv
            with open(rd.getAbsPath(
                "res/namemap.csv")) as f:
              nameMap = dict(csv.reader(f))
          </code>
        </setup>
        <code>
          @object = nameMap[@object]
        </code>
      </apply>

      <map key="weighting">\inputRelativePath.split("_")[-1][:-5]</map>
    </rowmaker>
  </make>
</data>

This does, step by step:

• The sources element is as always – with image collections, the recurse attribute usually comes in particularly handy.
• When ingesting images, you will (at least for a while, still) almost always read from FITS images, i.e., FITS primary headers. A fitsProdGrammar delivers the key-value-pairs from a header as a rawdict.
• The qnd attribute of the grammar is recommended. It makes some (weak) assumptions to yield significant speedups with large images, just so long as you can make do with the primary header.
• The fitsProdGrammar will map keys with hyphens to names with underscores, which allows for smoother action with them in rowmakers. The mapKey` element can produce additional mappings; in this case, we abuse it a bit to let us have idmaps (rather than simplemaps) in the rowmaker. And, actually, to illustrate the feature, as this data does not need that facility, really.
• The grammar further needs a rowfilter. The products#define rowfilter lets you add keys on owners and embargo in case you want password protection for images, but most importantly it defines what table the data is destined for. This is crucial information, and if you ever get it wrong, you need to manually connect to the database and issue a command like DELETE FROM products WHERE sourcetable='<your wrong table>'. So, always bind table. Make sure to include the quotes, this is supposed to be a valid python expression yielding a string. Also, this is where you'd define the media type for your products if they're not FITS files.
• You then need to define a rowmaker that must apply two procs. For one, you need //siap#computePGS (if you mixed in //siap#pgsSIAP). No bindings are required here.
• The second proc application required is //siap#setMeta . Try to give all its keys somewhat sensible values, you will make your users' lives much easier.
• Typically, many values coming in the FITS headers will be messy and fouled up. You'll spend some quality time fixing these values in the typical case. Here, we translate somewhat broken object names using a simple mapping file that was provided by the author. In other circumstances there's the procs#mapValue procApp helping you.
• As is usual in procApps, you can access the embedding RD as rd. Here, we use that to let DaCHS find the input file independently of where the program was started.

Warning: Do not use idmaps="*" with SIAP, since the auto-generated mappings will clobber the work of the xSIAP procs.

Cores

There are two cores you may want for SIAP services:

• dbCore, to which you add the necessary condDescs manually as below, for "normal" SIAP services.
• siapCutoutCore, which speaks SIAP but returns cutouts rather than full images; the size of these cutouts is determined by the SIZE argument (i.e., the region of interest).

To furnish these cores with the parameters required by the standard, use the //siap#protoInput condDesc. If you want to re-use the core for a form-based service, use the //siap#humanInput condDesc as well. Both are written in a way that they'll sense if they run under a SIAP renderer or not.

So, a basic core with a couple of additional fields would look like this:

<dbCore id="query_images" queriedTable="main">
  <condDesc original="//siap#protoInput"/>
  <condDesc original="//siap#humanInput"/>
  <condDesc buildFrom="dateObs"/>
  <condDesc buildFrom="bandpassId" />
  <condDesc>
    <inputKey name="object" type="text"
        tablehead="Target Object"
        description="Object being observed, Simbad-resolvable form"
        ucd="meta.name" verbLevel="5" required="True">
        <values fromdb="object FROM lensunion.main"/>
    </inputKey>
  </condDesc>
</dbCore>

Service

If you wrote the core to work for both SIAP and form as described above, there's little more to say except you'll want to use the siap.xml renderer, and you need some additional metadata for VO registration. The latter is described in the siap.xml reference.

With this, the service definition would look like this:

<service id="im" allowed="form,siap.xml" core="query_image">
  <meta name="shortName">sample images</meta>
  <meta name="title">Sample Image Archive</meta>
  <meta name="sia.type">Pointed</meta>

  <meta name="testQuery.pos.ra">230.444</meta>
  <meta name="testQuery.pos.dec">52.929</meta>
  <meta name="testQuery.size.ra">0.1</meta>
  <meta name="testQuery.size.dec">0.1</meta>

  <publish render="siap.xml" sets="ivo_managed"/>
  <publish render="form" sets="local,ivo_managed"/>
</service>

(where again you can just write the above core inline rather than referencing it; that's the style we usually recommend).

SSAP Quick Start

Check out a sample resource directory into your inputs directory:

cd `dachs config inputsDir`
svn co http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/zcosmos
cd zcosmos
mkdir data

The checkout does not contain actual data, so let's fetch a file:

cd data
curl -O \
  http://dc.g-vo.org/getproduct/zcosmos/data/1D/zCOSMOS_BRIGHT_DR2_000826802_ZCMRa26_M1_Q4_13_1.fits
cd ..

This dataset needs obscore, and you should obscore-publish your spectra as well. So, make sure you have the obscore table:

dachs imp //obscore

Run the input and the regression tests:

dachs imp q
dachs test q

One regression test should fail since you've not yet pre-generated the previews (which are optional but recommended for your datasets, too):

python bin/makepreviews.py
dachs test q

If the regressions tests don't pass now, please complain to the authors of this tutorial.

From here on, you can point your favourite spectral client (fallback: TOPCAT's SSA client; note that TOPCAT itself cannot open this service's native format and you'll have to go through datalink, which TOPCAT knows how to do since about version 4.6) to http://localhost:8080/zcosmos/q/ssa/ssap.xml and do your queries (if you don't know anything else, do a positional query for 149.734, 2.28216).

Please drop the dataset again when you're done playing:

dachs drop zcosmos/q

Since the dataset is in the obscore table, it would otherwise be globally discoverable, and that'd be bad.

The remainder of this chapter sometimes refers to this RD. It is probably a good idea to have it open while reading this.

SSAP Local Tables

Tables suitable for SSAP querying need a certain structure. Contrary to what DaCHS does with the relatively small SCS, SIAP, and SLAP models, for SSAP we now always go through a database view on top of a table the structure of which is controlled by you; you're saving quite a bit of work if you keep your own table's columns as close to their SSA form as possible, though.

Another special situation is that most spectra are delivered in fairly crazy formats, which means that it's usually a good idea to have a datalink service that serves halfway standard files – in DaCHS, these comply to the VO's spectral data model, which is a VOTable with a bit of standard metadata. It's certainly not beautiful or very sensible, but it sure beats the 1D images people still routinely push around.

So, to start a spectral service, use the ssap+datalink template:

$ mkdir myspectra; cd myspectra
$ dachs start ssap+datalink

This will result in the usual q.rd file as per starting from scratch; see there for how to efficiently edit this and for explanations on the common metadata items.

SSAP-specific material starts at the meta definitions for ssap.dataSource and ssap.creationType. These are actually used in service discovery, so you should be careful to select the right words. Realistically, using survey/archival for observational data and theory/archival for theoretical spectra should be the right thing most of the time.

Next, you define the raw_data table; this should contain all metadata “unpredictably” varying between datasets (or, for large data collections, anything that needs to be indexed for good performance). For instance, for observational data, the observed location is going to change from row to row. The start and the end of the spectrum is probably going to be fixed by the instrument, and so may not belong to the table.

To conveniently define the table, it is recommended to pull the SSA columns for raw_data by name from DaCHS' SSAP definitions and use SSAP conventions (e.g., units). The generated RD is set up for this by giving namePath="//ssap#instance", which essentially means “if someone requests an element by id, first look in the instance table of the RD. This is then used in the following LOOP (cf. Active Tags). As generated, this will look somewhat like:

<LOOP listItems="ssa_dateObs ssa_dstitle ssa_targname ssa_length
  ssa_specres ssa_timeExt">

– this will pull in the enumerated columns as if you had defined them literally in the table. Depending on the nature of your data, you may want to pull in more columns if they vary for your datasets (or throw out ones you don't need, as ssa_dateObs for theoretical data).

To see what is available, refer to the reference documentation of the the //ssap#view mixin. Any parameter that starts with ssa_ can be pulled in as a column.

The template RD then mixes in //products#table (which you pretty certainly want; see above), //ssap#plainlocation (which you want if you have positions on observational data) and //ssap#simpleCoverage (which you want if you want to publish your spectra through obscore.

You can, of course, define further columns here, both for later use in the view and for local management. SSAP lets you return arbitrary local columns, and in particular for theory services, you will have to (to define the physics of your model). As a DaCHS convention, please don't use the ssa_ prefix on your custom columns. See theossa/q for an example of that.

The SSA template then goes on with a data item filling the raw_data table. The template assumes you're parsing from IRAF-style 1D images. You will have to use a different grammar if that is not what you have, in particular you cannot use the specAx view defined in the rowmaker.

The first thing you will notice in the template is <recreateAfter>make_view</recreateAfter>; this simply makes sure that the SSA view will be regenerated after you import the table itself.

The rowfilter in the grammar is fairly complex here because we will completely hide the original; if you simply want to serve your upstream format, just cut it down to just giving table, mime, preview and preview_mime. If you do that, use the following strings in mime:

• image/fits for IRAF-style 1D image spectra
• application/fits for spectra in FITS tables
• application/x-votable+xml for spectra in VOTables

Let's hope you're not struggling with anything else.

However, for most cases (including also ASCII spectra), here's what we recommend as the product definition rowfilter (it's roughly what's in the template) to isolate your clients from the odd upstream formats:

<rowfilter procDef="//products#define">
  <bind name="table">"\\schema.main"</bind>
  <bind name="path">\\fullDLURL{"dl"}</bind>
  <bind name="fsize">%typical size of an SDM VOTable%</bind>
  <bind name="datalink">"\\rdId#dl"</bind>
  <bind name="mime">"application/x-votable+xml"</bind>
  <bind name="preview">\\standardPreviewPath</bind>
  <bind name="preview_mime">"image/png"</bind>
</rowfilter>

This is pointing the path accessed to a datalink URL using the fullDLURL macro, which expands to a URL retrieving the full dataset; the “dl” argument to the macro references the datalink service defined further down. Since the data returned is generated on the fly, you'll have to give an estimate of how large the VOTable will be (overriding DaCHS' default of the size of the source file). Don't sweat this too much, just don't claim something is 1e9 bytes when you're really just returning 1e4.

The bindings already prepare for making and serving previews, which is discussed in more detail in Product Previews in the DaCHS reference; see there for everything mentioning “preview”:

SSAP has a feature that lets users request certain formats, and for clients that don't know Datalink, this may be a good idea. In that scheme, you use a rowfilter to return a description of your native data and the processed SDM-compliant dataset as used here. See theossa/q for an example how that would look like. Our recommenation: don't bother, it's a misfeature confusing your users.

The rowmaker is fairly standard; we should perhaps mention the elements:

<var name="specAx">%use getWCSAxis(@header_, 1) for IRAF FITSes (delete otherwise)%</var>
<map key="ssa_specstart">%typically, @specAx.pixToPhys(1)*1e-10%</map>
<map name="ssa_length">%typically, @specAx.axisLength%</map>

getWCSAxis is a function that looks at a FITS image's WCS information to let you transform pixel to physical coordinates. This currently uses a simplified DaCHS implemenation that only does a small part of WCS (but we may change that, keeping the interface stable). The var, anyway, binds the resulting object to specAx. You can use that later to find out the limits of the spectrum. The way it's written here, you'll still have convert the value to metres manually.

The SSA View

The template goes on defining the data table that will serve as the basis of the service. It starts with the declaration:

<meta name="_associatedDatalinkService">
  <meta name="serviceId">sdl</meta>
  <meta name="idColumn">ssa_pubDID</meta>
</meta>

This is lets DaCHS add a link to the datalink service to results generated from this (both SSA and TAP). There's nothing you need to change here (unless you chuck datalink).

The main body of the table definition is the //ssap#view mixin. This lets you write the SSA parameters as SQL literals (i.e., strings need single quotes around them) or expressions involving column references (it's recommended to have SSA-ready columns in the source table, so you'll usually have column references only here). Most of these items default to NULL, so if you don't have a piece of metadata, it's reasonably safe to just remove the attribute.

A few of the mixin's parameters deserve extra discussion:

• sourcetable – this is a reference, i.e., this must resolve to the id of some table element. It can be cross-RD if really necessary. It is not the SQL table reference (that would include a schema reference).
• copiedcolumns – this lets you copy over columns from the source table, i.e., the one you just defined using (comma-separated) shell patterns of their names (yes, that's just like idmaps in rowmakers). The * given in the template should work in most cases, but if you have private columns in the source table, you can suppress them in the view; a useful convention might be to start all private columns with a p; you'd then say copiedcolumns="[^p]*". Note that copied columns are automatically added in the view as 1:1 maps, and you cannot use view arguments to override them. Use different column names in the source table and the view if you (think you) have to do view-level processing of your values.
• customcode – use this if you have extra material to enter in the view definition generated by the mixin. We hope you won't need that and would be interested in your use case if you find yourself using this.
• ssa_spectralunit, ssa_fluxunit – these are the only mandatory parameters starting with ssa_ (but their values are still overwritten if they are in copiedcolumns). There really is no point in having them vary from row to row because their values are metadata for the corresponding error columns (which is a real problem in the SSA spec).
• ssa_spectralucd, ssa_fluxucd – these are like the unit parameters in that they contain dataset-level metadata. The only reason they're not mandatory is that there are defaults that seem sensible for a large number of cases. Check them, and again, you can't really let them vary from row to row.
• ssa_fluxSI, ssa_spectralSI, ssa_timeSI – these were an attempt to work around a missing specification for unit strings in the VO. Since we now have VOUnit, just ignore them.

The data item making this table is trivial – you should set it to auto="False" (i.e., don't build this on a unadorned dachs imp). The building of this data will normally be triggered by the recreateAfter of the source table import.

SSAP Services

Use the element ssapCore for SSAP services. You must manually feed in the condition descriptors for the SSAP parameters; the simplest way to do that is to FEED the ssap#hcd_condDescs stream. The hcd_condDescs STREAM includes condition descriptors for all mandatory and optional parameters that we can remotely see in use.

Some of them may not be relevant to your service because your table never has values for them. For example, theoretical spectra will typically not give information on positions. The SSAP spec says that such a service should ignore POS rather than returning the empty set.

If you think you must ignore certain conditions, you can use the PRUNE active tag. This looks like this:

<ssapCore queriedTable="newdata">
  <FEED source="//ssap#hcd_condDescs">
    <PRUNE id="coneCond"/>
    <PRUNE id="bandCond"/>
  </FEED>
</ssapCore>

Do not do this just because you don't have position information – this would mean that you would dump your complete archive for (typical) queries with a position, and that is neither required by the spec (even if you might think so at first reading) nor desirable.

Here is a table of parameter names and ids; you can always check them by inspecting the output of dachs adm dumpDF //ssap:

For APERTURE, SNR, REDSHIFT, TARGETNAME, TARGETCLASS, PUBDID, CREATORDID, and MTIME, the condDesc id simply is <keyname>_cond, e.g., APERTURE_cond.

To have custom parameters, simply add condDesc elements as usual:

<ssapCore queriedTable="newdata">
  <FEED source="//ssap#hcd_condDescs"/>
  <condDesc buildFrom="t_eff"/>
</ssapCore>

For SSAP cores, buildFrom will enable “PQL”-like query syntax such that users can post arguments like 20000/30000,35000 to t_eff. This is in keeping with the general SSAP parameter style, while more modern VO services use 2-arrays for intervals.

To expose SSAP cores, use the ssap.xml renderer. Using the form renderer on SSAP cores is not terribly useful, because the core returns XML directly, and there are far too many parameters no human will ever be interested in anyway. So, you'll typically define extra browser-based services. The example RD shows a compact way to do that.

Form-based Spectral services

The SSAP interface is fairly ugly on both input and output if used with a form renderer. Hence, we recommend to build an extra form-based service with just minimal metadata. The template has something like this:

<service id="web" defaultRenderer="form">
  <meta name="shortName">\\schema Web</meta>

  <dbCore queriedTable="main">
    <condDesc buildFrom="ssa_location"/>
    <condDesc buildFrom="ssa_dateObs"/>
    <condDesc>
      <inputKey original="data.ssa_targname" tablehead="Star">
        <values fromdb="ssa_targname from theossa.data
          order by ssa_targname"/>
      </inputKey>
    </condDesc>
  </dbCore>

  <outputTable>
    <autoCols>accref, mime, ssa_targname,
      ssa_aperture, ssa_dateObs, datalink</autoCols>
    <FEED source="//ssap#atomicCoords"/>
    <outputField original="ssa_specstart" displayHint="displayUnit=Angstrom"/>
    <outputField original="ssa_specend" displayHint="displayUnit=Angstrom"/>
  </outputTable>
</service>

Essentially, we only select a few fields people might want to query against, and we directly build them out of the query fields; the SSA condDescs are bound to the funny and insufficiently defined SSA input syntax and probably not very useful in interactive applications.

The extra selector for object names with the names actually present in the database is a nice service as long as you only have a few hundred objects or so.

In the output table, we only give a few of the many dozen SSAP output fields; and we change the units of the spectral limits to Angstroms, which will look nicer of optical spectra.

In the template, this form-based service is published as a capability of the SSA service. This is done using the service attribute in:

<publish render="form" sets="ivo_managed,local" service="web"/>

SSAP Obscore Publication

The SSA metadata is not far from the obscore metadata, and so it's fairly easy to also publish the spectra through ObsTAP. Just mix in //obscore#publishSSAPMIXC and set calibLevel. The template does a bit more for one reason or another:

<mixin
  calibLevel="%likely one of 1 for uncalibrated or 2 for uncalibrated data%"
  coverage="%ssa_region -- or remove this if you have no ssa_region%"
  sResolution="ssa_spaceres"
  oUCD="ssa_fluxucd"
  emUCD="ssa_spectralucd"
  >//obscore#publishSSAPMIXC</mixin>

– if you use one of the old hcd or mixc mixins, you certainly don't want oUCD and emUCD.

SSAP and Datalink

Given that you'll usually get fairly bizarre inputs and will probably want to publish “repaired” spectra, using Datalink to provide both native and SDM (“Spectral Data Model”) compliant spectra without having to resort to SSAP's ill-thought-out FORMAT feature is a fairly natural thing to do. In DaCHS, that's not too hard; essentially, you have to write an embedded (or custom) grammar to parse the spectra. Other than that, it's just a few formalities.

So, you first define the table that will later hold your spectrum. Use the //ssap#sdm-instance mixin for that (this continues material from the template):

<table id="instance" onDisk="False">
  <mixin ssaTable="main"
    spectralDescription="%something like 'Wavelength' or so%"
    fluxDescription="%something like 'Flux density' or so%"
    >//ssap#sdm-instance</mixin>
  <meta name="description">%a few words what a spectrum represents%</meta>
</table>

The descriptions you need to enter here typically end up being labels on axes of spectral plots, so it's particularly important to be concise and precise here.

If your spectrum has additional columns (e.g., errors, noise estimates, bin widths), just put more columns in here. The mixin pulls in all the various params that SDM wants to see from the row of the spectrum in the SSAP table.

Note that the table does not have onDisk="True"; these tables are only made for the brief moment it takes to serialise them into what the user receives.

As usual, to fill tables, you want a data element. In the zcosmos example, it looks like this:

<data id="build_sdm_data" auto="False">
  <embeddedGrammar>
    <iterator>
      <setup>
        <code>
          from gavo.protocols import products
          from gavo.utils import pyfits
        </code>
      </setup>
      <code>

      fitsPath = products.RAccref.fromString(
          self.sourceToken["accref"]).localpath
      hdus = pyfits.open(fitsPath)
      ax = utils.getWCSAxis(hdus[0].header, 1)

      for spec, flux in enumerate(hdus[0].data[0]):
        yield {"spectral": ax.pix0ToPhys(spec), "flux": flux}
      hdus.close()
      </code>
    </iterator>
  </embeddedGrammar>
  <make table="spectrum">
    <parmaker>
      <apply procDef="//ssap#feedSSAToSDM"/>
    </parmaker>
  </make>
</data>

You'll notice that the rowmaker in the example is missing. DaCHS will then fill in the default one, which essentially is idmaps="*". Since you're writing the grammar from scratch, just use the names of the columns defined in the instance table and be done with it. The predefined column names are spectral and flux, so make sure you always have keys for them in the dictionaries you yield those from your grammars.

Then there's a Element parmaker; this is stereotypical, just always use the procDef as here. It copies values from the SSA input row to the params in the instance table.

The thing you have to vary is the code in the embedded grammar. It must, from its input (self.sourceToken, which here is the SSA row) figure out where the data file is (in this case, we can get it from the products table, which is the code with products.RAccref. More typically, just make a convention how to get from accref to the local file and use that. An example for that is in feros/q (the complication there is that the datalink products actually have accrefs themselves, and for those the path in the product table is the datalink URL; that, of course, wouldn't bring the grammar any closer to finding the file.

But you can do arbitrary things here; see the califa/q3 RD for an example for how you can decode the accref to database rows. A more complex scenario with ASCII data stored externally and cached is in theossa/q, a case where multiple orders of echelle spectra are being processed in flashheros/q.

Once you have that just build dictionaries with at least spectral and flux keys; at least with the default rowmaker you need to make sure that the values match the units you define in the instance table.

Finally, write the service; you should at least have skimmed the endless chapter on Datalink and SODA in the reference documentation to get an idea of how descriptor generators, data functions, and meta makers play together:

<service id="sdl" allowed="dlget,dlmeta">
  <meta name="title">zCosmos Datalink Service</meta>

  <datalinkCore>
    <descriptorGenerator procDef="//soda#sdm_genDesc">
      <bind name="ssaTD">"\rdId#data"</bind>
    </descriptorGenerator>
    <dataFunction procDef="//soda#sdm_genData">
      <bind name="builder">"\rdId#build_sdm_data"</bind>
    </dataFunction>
    <FEED source="//soda#sdm_plainfluxcalib"/>
    <FEED source="//soda#sdm_cutout"/>
    <FEED source="//soda#sdm_format"/>
  </datalinkCore>
</service>

Datalink services (almost) always need to be enabled for both dlget and dlmeta. The //soda#sdm_genData data function needs a reference to the data element filling the instance table. The actual operations you can simply pull from the soda RD; sdm_plainfluxcalib offers a maximum=1 calibration of the spectral values (which probably is not terribly useful in all cases), sdm_cutout does cutouts, which may come handy when you have long spectra, and sdm_format finally is the format translation that's frequently useful with spectra.

In particular as long as only few clients evaluate datalink blocks in DAL responses, it may be a good idea to include a custom column with a datalink URL in the SSAP table. The ssap+datalink template already has that in its source table:

<column name="datalink" type="text"
  ucd="meta.ref.url;meta.data.datalink"
  tablehead="Datalink"
  description="A link to a datalink document for this spectrum."
  verbLevel="15" displayHint="type=url"/>

(we're cheating a bit with the UCD; it's actually not legal right now, but we're trying). To compute the datalink URL, there's again a helpful macro taking the datalink service id:

<map name="datalink">\dlMetaURI{sdl}</map>

DaCHS for now delivers XSLT with datalink (or does the XSLT translation itself if it suspects its client is actually a web brower), so there's no harm (but potentially quite a bit benefit) to feed this URL to tables displayed by web browsers, as we do here in the web service.

SSAP for Time Series

As long as there's no anointed successor to SSAP for time series, you can use SSAP itself to publish time series. It's a bit of a hack, but clients like SPLAT do about the right thing with them.

If you're looking for examples, check out k2c9vst/q (which is pretty much only time series parsed from ASCII time series) and gaia/q2 (which stores the acutal time series in the database, a technique we believe is a very good idea).

Obscore Derived from Typed Service Tables

ObsTAP is basically a single table, ivoa.ObsCore. In DaCHS, this is a view generated from input tables. To include the products within a table, you must use one of the mixins from the //obscore RD and fill out some of the mixin's parameters. There is some documentation on what to put where in the mixin documentation, but frankly, as a publisher, you should have at least passing knowledge of the obscore data model as laid down in Tody et al (2011).

In the simplest case, a SIAP table, you could get by simply adding:

mixin="//obscore#publishSIAP"

to the table definition's start tag. You do not have to re-import a table to publish it to obscore after the fact – dachs imp -m <rd id> && dachs imp //obscore create will include an existing table to the obscore view.

Even for SIAP, you will usually want to add metadata not contained in DaCHS' SIAP meta. To do this, add a mixin element to the table definition's body:

<mixin
  sResolution="0.5"
  calibLevel="2"
  >//obscore#publishSIAP</mixin>

To find out what parameters the mixin takes, see //obscore#publishSIAP in the reference documentation.

On a table import, the obscore table will automatically be recreated to include the data. If you retrofit ObsCore support to large tables, you can avoid having to re-import everything by adding the mixin clause and then updating the metadata. In that case, you must manually remake the obscore table:

dachs imp -m path/to/my/rd
dachs imp //obscore create

For SSAP tables, there is an //obscore#publishSSAPMIXC mixin that works like its SIAP cousin (see the reference documentation of details).

Pure Obscore Tables

You can also have "pure" Obscore tables which do not build on protocol mixins. A live example is the cubes table in the califa/q3 RD within the GAVO data center. Here's a brief explanation of how this works.

For the non-constant parts of your data, re-use the metadata given in the global obscore table – to make that convenient, tell DaCHS to resolve original references in there:

<table id="cubes" onDisk="True" namePath="//obscore#ObsCore">

adql="True" is absent here as the obscore mixin set it. It wouldn't hurt, though.

You will almost always want to have DaCHS manage your products. This works even when all your files are external (i.e., you're entering http URLs in accessURL), so it's a good idea to always mix in products:

<mixin>//products#table</mixin>

Then, you mix in //obscore#publish, which is like the protocol-specific mixins except it doesn't pre-set parameters based on what's already in protocol-specific tables:

<mixin
    accessURL="dlurl"
    size="10"
    mime="'application/x-votable+xml;content=datalink'"
    calibLevel="3"
    collectionName="'CALIFA'"
    coverage="s_region"
    dec="s_dec"
    emMax="7e-7"
    emMin="3.7e-7"
    emResPower="4000/red_disp_mean"
    expTime="t_exptime"
    facilityName="'Calar Alto'"
    fov="0.01"
    instrumentName="'PMAS/PPAK at 3.5m Calar Alto'"
    oUCD="'phot.flux;em.opt'"
    productType="'cube'"
    ra="s_ra"
    sResolution="0.0002778"
    title="obs_title"
    tMax="t_min"
    tMin="t_max"
    targetClass="'Galaxy'"
    targetName="target_name"
  >//obscore#publish</mixin>

Essentially, what's constant is given in literals, what's variable is given as a column reference. It is a bit unfortunate that you have to enter quite a few identity mappings in here, so we might provide a mixin presetting those if this turns out to be a common use case. Tell us if you're annoyed.

You can then add your custom columns (which might be useful if people directly query your table). The central part is copying over obscore columns that are not constant for your data collection. For califa, this looks like this:

<LOOP listItems="obs_id obs_title obs_publisher_did
    target_name t_exptime t_min t_max s_region
    t_exptime">
    <events>
      <column original="\item"/>
    </events>
  </LOOP>

– you'll obviously have to adapt the listItems. To see what column names are available, see the obscore table description.

That's about it for defining the table. To fill the table, just have a normal rowmaker; since the table contains products, don't forget the //products#define rowfilter in the grammar.

Datalinks in columns

The simplest way to let people discover datalink documents is by including links in normal tables (as for DAL tables) and keep the normal access URL in the accref column. To do that, define a column like this:

<column name="datalink" type="text"
  ucd="meta.ref.url"
  tablehead="DL"
  description="URL of a datalink document for this dataset"
  verbLevel="1" displayHint="type=url"/>

and then fill it in the corresponding rowmaker. For the normal case where a product corresponds to a disk file, there is a macro to help you:

<map key="datalink">\dlMetaURI{dl}</map>

Here, the “dl” in the macro argument must be the id of the datalink service as discussed below.

In such situations, it is still usually advisable to use the datalink URL rather than the naked accref in the obscore table – for instance, when you are using a SIAP cutout on large image (obscore cannot do cutouts) or to allow format conversions. To do that, you need to advise the obscore to use the datalink response; besides changing the access URL, you will also need to adjust the access_format (so clients know they will have to deal with a datalink) and the access_estsize (because they will not have to download the entire dataset in one go). For the latter, it is hard to predict how large the datalink stream will end up being. Guessing “about 10 kB” should be good enough, though.

In total, the obscore mixin would have to look like this:

 <mixin
  mime="'application/x-votable;content=datalink'"
  accessURL="datalink"
  size="10"
  ...
>//obscore#publish(WHATEVER)</mixin>

(in case you're wondering: the datalink given here for accessURL is just the column name as given above).

Datalinks as accrefs

In particular for large datasets, it is probably a good idea to keep people from blindly pulling the data without first having been made aware that what they're accessing is not just a little CCD frame. Having the datalink document as the primary document retrieved is a fairly good way to do that; of course, without a datalink-enabled client people might be locked out from the dataset entirely. On the other hand, DaCHS comes with a stylesheet that enables datalink operation from a common web brower, so that's perhaps not too bad.

To have datalinks rather than the plain dataset as what the accref points to, you need to change what DaCHS thinks of your dataset; this is what the //products#define rowfilter in your grammar is for:

<fitsProdGrammar qnd="True">
  <rowfilter procDef="//products#define">
    <bind key="path">\dlMetaURI{dl}</bind>
    <bind key="mime">'application/x-votable+xml;content=datalink'</bind>
    [...]
  </rowfilter>
  [...]
</fitsProdGrammar>

When you do this, you must use a datalink-aware descriptor generator. When you use the recommended setup, where the accref is the inputsDir-relative path to the main file, and you're dealing with FITS, you can use the DLFITSProductDescriptor class (in other cases, its implementation (in gavo.protocols.datalink) will show you how to handle these cases). Thus, the base functionality of a FITS cutout service with datalink products would be:

 <service id="dl" allowed="dlget,dlmeta">
  <meta name="title">My Cutout Service</meta>
  <datalinkCore>
    <descriptorGenerator procDef="//soda#fits_genDesc"
      name="genFITSDesc">
      <bind key="accrefPrefix">'mysvcs/data'</bind>
      <bind key="descClass">DLFITSProductDescriptor</bind>
    </descriptorGenerator>
    <FEED source="//soda#fits_standardDLFuncs"/>
  </datalinkCore>
</service>

Also note that DaCHS will not produce automatic previews in this situation. You will have to generate precomputed previews as discussed in Product Previews.

Simple FITS Datalink Services

A fairly typical case for a datalink service is a file that has a few related datasets and perhaps a cutout. To define such a thing, write something like (cf. lswscans/res/positions.rd):

<service id="dl" allowed="dlget,dlmeta">
  <meta name="title">HDAP Datalink</meta>
  <meta name="description">This service lets you access cutouts
    from HDAP plates and retrieve scaled versions.</meta>
  <datalinkCore>
    <descriptorGenerator procDef="//soda#fits_genDesc">
      <bind key="accrefPrefix">lswscans</bind>
    </descriptorGenerator>
     <FEED source="//soda#fits_standardDLFuncs"/>

    <metaMaker>
      <code>
        yield descriptor.makeLink(
          makeProductLink(descriptor.accref+"?scale=4"),
          contentType="image/fits",
          description="FITS, scaled by 1/4",
          semantics="#science",
          contentLength=descriptor.estimateSize()/16.)
      </code>
    </metaMaker>
  </datalinkCore>
</service>

TODO: Explain what's going on here.

When you attach the datalink functionality (rather than having datalink links as access URLs), the following is the recommended pattern:

1. Add a pub_did column to the SIAP table. Make sure you're giving it a UCD of meta.id;meta.main and that you have no other such column in your table:

   <table id="data" onDisk="true" mixin="//siap#pgs">
     <index columns="pub_did"/>
     <column name="pub_did" type="text"
       ucd="meta.id;meta.main"
       tablehead="P. DID"
       description="Dataset identifier assigned by the publisher"
       verbLevel="15"/>
   

2. Populate the pub_did column in the row maker:

   <map key="pub_did">\standardPubDID</map>
   

3. In the service that has the siap.xml renderer, declare datalink support by setting the corresponding property to the id of the datalink service (here assumed to be dl):

   <service id="ssa" core="siacore" allowed="siap.xml">
     <property name="datalink">dl</property>
   

4. If you have a form-based publication, hide the pub_did column and rather make datalinks:

   <service id="web" core="siacore">
     ...
     <outputTable>
        <outputField name="dlurl" select="accref"
         tablehead="Datalink Access"
         description="URL of a datalink document for the dataset
           (cutouts, different formats, etc)">
         <formatter>
           yield T.a(href=getDatalinkMetaLink(
             rd.getById("dl"), data)
             )["Datalink"]
         </formatter>
       </outputField>
   

Quick Start

Install PyPDS if you don't have it anyway:

curl -LO https://github.com/RyanBalfanz/PyPDS/archive/master.zip
unzip master.zip
cd PyPDS
python setup.py build
sudo python setup.py install

Get the sample data:

cd `dachs config inputsDir`
curl -O http://docs.g-vo.org/epntap-example.tar.gz
tar -xvzf epntap-example.tar.gz
cd lutetia

Import it and build the previews from the PDS images:

dachs imp q
python bin/makePreview.py

Start the server as necessary. If you go to your local ADQL endpoint (something like http://localhost:8080/adql) and execute queries like:

SELECT * FROM lutetia.epn_core

there.

For access through a standard protocol, start TOPCAT, select VO/TAP Query, and at the bottom of the dialog enter http://localhost:8080/tap (or whatever you configured) in “TAP URL”. Hit “Use Service”, wait until the table metadata is in and then again query something like:

SELECT * FROM lutetia.epn_core

Hit "Run Query",open the table and play with it. As a little visual treat, in TOPCAT's main window hit "Activation Action", and configure the preview_url column under “View URL as Image”. Then click on the table rows.

To get into Vespa's query interface, you will have to register your table. Do not do this with the sample data.

Tables

In essence, EPN-TAP is just a set of columns, some mandatory, some optional. The mandatory ones are pulled into a table by using the //epntap2#table-2_0 mixin, which needs the spatial_frame_type parameter (see reference for what's supported there) since that determines the metadata on the spatial columns.

This treatment distinguishes the two cases mentioned above: Files served by someone else first, files served by DaCHS next.

Externally Served Products

The mixin will already open the table for ADQL querying, and to make that work, it is also declared onDisk. What's not declared is the table name, although it's fixed at epn_core. Hence, the minimal definition of an EPN-TAP table is:

<table id="epn_core">
  <mixin
    spatial_frame_type="body"
  >//epntap2#table-2_0</mixin>
</table>

If you want to use optional columns from the EPN-TAP specification, give them in optional_columns. In the typcial case that you publish data products, you'll at least want the access columns, which would translate into this:

<table id="epn_core">
  <mixin
    optional_columns="access_url access_format access_estsize"
    spatial_frame_type="body"
  >//epntap2#table-2_0</mixin>
</table>

You can of course add further custom columns as necessary using the normal Element column.

Filling the table from what comes from the grammar happens through idmaps="*" on the rowmaker (which you want as the apply doesn't actually map anything into the results and there's too many columns to make enumerating fun) and an application of //epntap2#populate-2_0 . The mixin's documentation is intended as a checklist; go through the parameters one by one and see which ones correspond to something in your input and then say:

<par key="param-name">@source-key</par>,

possibly using python functions and expressions to adapt your inputs to what should end up in the database.

You may need to refer to the EPN-TAP proposed specification now and then while filling things out. Note again that these are python expressions, and so you have to use quotes when specifying literal strings.

More complex operations should probably go to var declarations rather than the bind bodies; again, see the examples to see how.

The real work is populating the table. Let's first assume you data products and use DaCHS to publish them. You will then use a grammar, and as EPN-TAP then deals with data products, the grammar must use the rowfilter products#define rowfilter as discussed above in SIAP; the lutetia example shows how this can be done while also providing for DaCHS-generated previews.

Externally Served Products

When you can store the data products you describe locally and have DaCHS hand them out, that's usually preferable. It also lets you do things like enforcing embargoes, make previews, etc.

To be able to manage the products, you need to add the //epntap2#localfile-2_0 mixin, as well as the optional columns starting with access_ in the table definition, so minimally:

<table id="epn_core">
  <mixin
    spatial_frame_type="celestial"
    optional_columns="access_url access_format access_estsize"
    >//epntap2#table-2_0</mixin>
  <mixin>//epntap2#localfile-2_0</mixin>
</table>

To let DaCHS manage the files, you need the //products#define rowfilter in your grammar. You will minimally have to tell it the table you're writing to (this is so DaCHS can remove the entries when you drop the table); since it by default assumes you're serving FITS files, which in planetary sciences is probably untrue, you should also give the media type explicitly, for instance:

<sources pattern="data/*.pds" recurse="True"/>
<pdsGrammar>
  <rowfilter procDef="//products#define">
    <bind name="table">"\schema.epn_core"</bind>
    <bind name="mime">"image/x-pds"</bind>
  </rowfilter>
</pdsGrammar>

(the \schema will just put in the current schema name defined at the top of your RD). This is also where you'd configure previews, embargos, etc.

Finally, in the rowmaker you need to make sure that the new columns are filled. That's done using a paramterless apply:

<rowmaker>
  ...
  <apply procDef="//epntap2#populate-localfile-2_0"/>
</rowmaker>

About s_region

The s_region parameter (see //epntap2#populate-2_0) is essentially a footprint describing the area covered by 2D spatially extended data products. It uses pgshpere types such as spoly, scircle, sbox or spoint (We advise against the use of spoint as a s_region type: only spatially extended types should be used). The default type is spoly, the others must be specified using the regiontype mixin parameter (Please refer to //epntap2#table-2_0).

Please note that the spoly type has some limitations that can trigger unclear errors when not respected: the segments can not be crossed and, more important, the maximum dimension of a polygon must be less than 180°.

Here are some example that shows how to implement it with a sbox:

First the regiontype:

<table id ="epn_core">
...
  <mixin ... regiontype="sbox">//epntap2#table-2_0</mixin>
</table>

Then the s_region parameter, defined in the rowmaker (see Mapping data):

<rowmaker idmaps="*">
  <map key = "s_region">
    pgsphere.SBox(
      pgsphere.SPoint.fromDegrees(float(@c1_min),float(@c2_min)),
      pgsphere.SPoint.fromDegrees(float(@c1_max),float(@c2_max)))
  </map>
</rowmaker>

If you want an “aperture” (point and radius), you'd write something like:

<table id ="epn_core">
...
  <mixin ... regiontype="scircle">//epntap2#table-2_0</mixin>
</table>

<rowmaker idmaps="*">
  <map key = "s_region">
    pgsphere.SCircle(
      pgsphere.SPoint.fromDegrees(float(@c1_min),float(@c2_min)),
      float(@aperture)*DEG)
  </map>
</rowmaker>

The radius of the circle must be given in radians here; if it's in degrees, just multiply with DaCHS' internal DEG symbol (which is pi/180).

Service

EPN-TAP tables are queried through the data center's TAP service. If you have registred that, there is nothing else you need to do to access your data.

For registration, just add:

<publish/>

to your table body and run dachs pub <rd-id>.