gavo.protocols.products module

Products and a core turning accrefs into lists of products.

There is a substantial overlap between what’s going on there and datalink (and datalink uses some of the products mentioned here). The cutouts and scale things here shouldn’t be developed on, all this should move towards datalink. Meanwhile, we still have siapCutoutCore and friends that relies on the mess here, so all this is going to remain for the foreseeable future. Just don’t extend it.

The “user-visible” part are just accrefs, as modelled by the RAccref – they can contain instructions for cutouts or scaling, hence the additional structure.

Using the product table and some logic in this module, such accrefs are turned into subclasses of ProductBase.

These have mime types and know how to generate their data through their synchronous iterData methods. They must also work as t.w resources and thus have implement asynchronuous render(request) methods. It’s a bit unfortunate that we thus depend on t.w here, but we’d have to reimplement quite a bit of it if we don’t, and for now it doesn’t seem we’ll support a different framework in the foreseeable future.

class gavo.protocols.products.CutoutProduct(rAccref)

Bases: ProductBase

A class representing cutouts from FITS files.

This only works for local FITS files with two axes. For everything else, use datalink.

We assume the cutouts are smallish – they are, right now, not streamed, but accumulated in memory.

classmethod fromRAccref(rAccref, authGroups=None)

iterData()

makePreview()

render(request)

class gavo.protocols.products.FileProduct(rAccref)

Bases: ProductBase

A product corresponding to a local file.

As long as the accessPath in the RAccref’s productsRow corresponds to a real file and no params are in the RAccref, this will return a product.

close()

forSaving = True

classmethod fromRAccref(rAccref, authGroups=None)

getFile()

iterData()

read(size=None)

render(request)

class gavo.protocols.products.InvalidProduct(rAccref)

Bases: NonExistingProduct

An invalid file.

This is returned by getProductForRAccref if all else fails. This usually happens when a file known to the products table is deleted, but it could also be an attempt to use unsupported combinations of files and parameters.

Since any situation leading here is a bit weird, we probably should be doing something else but just return a 404. Hm…

This class always returns an instance from fromRAccref; this means any resolution chain ends with it. But it shouldn’t be in PRODUCT_CLASSES in the first place since the fallback is hardcoded into getProductForRAccref.

classmethod fromRAccref(rAccref, authGroups=None)

iterData()

class gavo.protocols.products.NonExistingProduct(rAccref)

Bases: ProductBase

A local file that went away.

iterData here raises an IOError, render an UnknownURI.

These should normally yield 404s.

We don’t immediately raise some error here as archive generation shouldn’t fail just because a single part of it is missing.

classmethod fromRAccref(rAccref, authGroups=None)

iterData()

render(request)

class gavo.protocols.products.PreviewCacheManager

Bases: object

is a class that manages the preview cache.

It’s really the class that manages it, so don’t bother creating instances.

The normal operation is that you pass the product you want a preview to getPreviewFor. If a cached preview already exists, you get back its content (the mime type must be taken from the products table).

If the file does not exist yet, some internal magic tries to come up with a preview and determines whether it should be cached, in which case it does so provided a preview has been generated successfully.

A cache file is touched when it is used, so you can clean up rarely used cache files by deleting all files in the preview cache older than some limit.

cachePath = '/home/msdemlei/gavo/web/previewcache'

classmethod getCacheName(accref)

returns the full path a preview for accref is be stored under.

classmethod getCachedPreviewPath(accref)

returns the path to a cached preview if it exists, None otherwise.

classmethod getPreviewFor(product)

returns a deferred firing the data for a preview.

This will raise a DataError if it doesn’t know how to make the preview.

classmethod saveToCache(data, cacheName)

class gavo.protocols.products.ProductBase(rAccref)

Bases: object

A base class for products returned by the product core.

See the module docstring for the big picture.

The constructor arguments of RAccrefs depend on what they are. The common interface is the the class method fromRAccref(rAccref, authGroups=None). It returns None if the RAccref is not for a product of the respective sort, the product otherwise.

authGroups is a set of groups authorised for the user when controlling access to embargoed products. This is the main reason you should never hand out products yourself but always expose the to the user through the product core.

The actual constructor requires a RAccref, which is exposed as the rAccref attribute. Do not use the productsRow attribute from rAccref, though, as constructors may want to manipulate the content of the product row (e.g., in NonExistingProduct). Access the product row as self.pr in product classes.

In addition to those, all Products have a name attribute, which must be something suitable as a file name; the default constructor calls a _makeName method to come up with one, and you should simply override it.

The iterData method has to yield reasonably-sized chunks of data (self.chunkSize should be a good choice). It must be synchronous.

Products usually are used as t.w resources. Therefore, they must have a render method. This must be asynchronuous, i.e., it should not block for extended periods of time.

Products also work as rudimentary files via read and close methods; by default, these are implemented on top of iterData. Clients must never mix calls to the file interface and to iterData. Derived classes that are based on actual files should set up optimized read and close methods using the setupRealFile class method (look for the getFile method on the instance to see if there’s a real file). Again, the assumption is made there that clients use either iterData or read, but never both.

If a product knows how to (quickly) generate a preview for itself, it can define a makePreview() method. This must return content for a mime type conventional for that kind of product (which is laid down in the products table).

chunkSize = 65536

close()

classmethod fromRAccref(accref, authGroups=None)

iterData()

read(size=None)

render(request)

classmethod setupRealFile(openMethod)

changes cls such that read and close work an an actual file-like object rather than the inefficient iterData.

openMethod has to be an instance method of the class returning an opened input file.

class gavo.protocols.products.ProductCore(parent, **kwargs)

Bases: DBCore

A core retrieving paths and/or data from the product table.

You will not usually mention this core in your RDs. It is mainly used internally to serve /getproduct queries.

It is instantiated from within //products.rd and relies on tables within that RD.

The input data consists of accref; you can use the string form of RAccrefs, and if you renderer wants, it can pass in ready-made RAccrefs. You can pass accrefs in through both an accref param and table rows.

The accref param is the normal way if you just want to retrieve a single image, the table case is for building tar files and such. There is one core instance in //products for each case.

The core returns a list of instances of a subclass of ProductBase above.

This core and its supporting machinery handles all the fancy product functionality (user authorization, cutouts, …).

attrSeq = [<gavo.base.complexattrs.StructListAttribute object>, <gavo.base.attrdef.BooleanAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>, <gavo.base.parsecontext.IdAttribute object>, <gavo.base.complexattrs.StructAttribute object>, <gavo.base.attrdef.IntAttribute object>, <gavo.rscdef.common.NamePathAttribute object>, <gavo.base.parsecontext.OriginalAttribute object>, <gavo.base.complexattrs.StructAttribute object>, <gavo.base.complexattrs.PropertyAttribute object>, <gavo.base.parsecontext.ReferenceAttribute object>, <gavo.rscdef.common.RDAttribute object>, <gavo.base.attrdef.UnicodeAttribute object>]

clearProperty(name)

completedCallbacks = []

getFullId()

getProperty(name, default=<Undefined>)

hasProperty(name)

managedAttrs = {'condDesc': <gavo.base.complexattrs.StructListAttribute object>, 'condDescs': <gavo.base.complexattrs.StructListAttribute object>, 'distinct': <gavo.base.attrdef.BooleanAttribute object>, 'groupBy': <gavo.base.attrdef.UnicodeAttribute object>, 'id': <gavo.base.parsecontext.IdAttribute object>, 'inputTable': <gavo.base.complexattrs.StructAttribute object>, 'limit': <gavo.base.attrdef.IntAttribute object>, 'namePath': <gavo.rscdef.common.NamePathAttribute object>, 'original': <gavo.base.parsecontext.OriginalAttribute object>, 'outputTable': <gavo.base.complexattrs.StructAttribute object>, 'properties': <gavo.base.complexattrs.PropertyAttribute object>, 'property': <gavo.base.complexattrs.PropertyAttribute object>, 'queriedTable': <gavo.base.parsecontext.ReferenceAttribute object>, 'rd': <gavo.rscdef.common.RDAttribute object>, 'sortKey': <gavo.base.attrdef.UnicodeAttribute object>}

name_ = 'productCore'

property rd

resolveName(context, id)

run(service, inputTable, queryMeta)

returns a list of {“source”: product} dicts for products matching the inputTable.

setProperty(name, value)

class gavo.protocols.products.RAccref(accref, inputDict={})

Bases: object

A product key including possible modifiers.

The product key is in the accref attribute.

The modifiers come in the params dictionary. It contains (typed) values, the possible keys of which are given in _buildKeys. The values in passed in the inputDict constructor argument are parsed, anything not in _buildKeys is discarded.

In principle, these modifiers are just the query part of a URL, and they generally come from the arguments of a web request. However, we don’t want to carry around all request args, just those meant for product generation.

One major reason for having this class is serialization into URL-parts. Basically, stringifying a RAccref yields something that can be pasted to <server root>/getproduct to yield the product URL. For the path part, this means just percent-escaping blanks, plusses and percents in the file name. The parameters are urlencoded and appended with a question mark. This representation is be parsed by the fromString function.

RAccrefs have a (read only) property productsRow attribute – that’s a dictionary containing the row for accres from //products#products if that exists. If it doesn’t, accessing the property will raise an NotFoundError.

classmethod fromPathAndArgs(path, args)

returns a rich accref from a path and a parse_qs-dictionary args.

(it’s mainly a helper for fromRequest and fromString).

classmethod fromRequest(path, request)

returns a rich accref from a t.w request.

Basically, it raises an error if there’s no key at all, it will return a (string) accref if no processing is desired, and it will return a RAccref if any processing is requested.

classmethod fromString(keyString)

returns a fat product key from a string representation.

As a convenience, if keyString already is a RAccref, it is returned unchanged.

property localpath

previewIsCacheable()

returns True if the a preview generated for this rAccref is representative for all representative rAccrefs.

Basically, scaled versions all have the same preview, cutouts do not.

property productsRow

returns the row in dc.products corresponding to this RAccref’s accref, or raises a NotFoundError.

class gavo.protocols.products.RemotePreview(rAccref)

Bases: RemoteProduct

A preview that’s on a remote server.

classmethod fromRAccref(rAccref, authGroups=None)

class gavo.protocols.products.RemoteProduct(rAccref)

Bases: ProductBase

A class for products at remote sites, given by their URL.

classmethod fromRAccref(rAccref, authGroups=None)

classmethod fromURL(url)

iterData()

render(request)

class gavo.protocols.products.ScaledFITSProduct(rAccref)

Bases: ProductBase

A class representing a scaled FITS file.

Right now, this only works for local FITS files. Still, the class is constructed with a full rAccref.

classmethod fromRAccref(rAccref, authGroups=None)

iterData()

render(request)

class gavo.protocols.products.StaticPreview(rAccref)

Bases: FileProduct

A product that’s a cached or pre-computed preview.

forSaving = False

classmethod fromRAccref(rAccref, authGroups=None)

class gavo.protocols.products.UnauthorizedProduct(rAccref)

Bases: FileProduct

A local file that is not delivered to the current client.

iterData returns the data for the benefit of preview making. However, there is a render method, so the product renderer will not use it; it will, instead, raise an Authenticate exception.

forSaving = False

classmethod fromRAccref(rAccref, authGroups=None)

render(request)

gavo.protocols.products.computePreviewFor(product)

returns a deferred returning image/jpeg bytes containing a preview of product.

This only works for a select subset of products. You’re usually better off using static previews.

This will raise a DataError if it doesn’t know how to make a preview.

gavo.protocols.products.formatProductLink(val, useHost)

turns a string val into a product link.

This is faily ad-hoc: if val looks like a URL, it is left alone, else turn it into a link into our product service (which means val must be an accref into the product table).

gavo.protocols.products.getProductColumns(colSeq)

returns the columns within colSeq that contain product links of some sort.

gavo.protocols.products.getProductForRAccref(rAccref, authGroups=None)

returns a product for a RAccref.

This tries, in sequence, to make a product using each element of PRODUCT_CLASSES’ fromRAccref method. If nothing succeeds, it will return an InvalidProduct.

If rAccref is a string, the function makes a real RAccref through RAccref’s fromString method from it.

gavo.protocols.products.makePreviewFromFITS(product)

returns image/jpeg bytes for a preview of a product spitting out a 2D FITS.

gavo.protocols.products.makePreviewWithPIL(product)

returns image/jpeg bytes for a preview of the PIL-readable product.

gavo.protocols.products.makeProductLink(key, withHost=True, useHost=None)

returns the URL at which a product can be retrieved.

key can be an accref string or an RAccref.

Note that this is using the preferred host as the basic URL. If you are running dual-protocol http/https and you ingest results of this function into the database, it is advisable to cut off the scheme part of the URI (e.g., split(":", 1)[-1]). In data products served, DaCHS will then put in the scheme used for the query.

DaCHS (almost always) also allows full http URIs as accrefs. These will be returned unchanged.

gavo.protocols.products.quoteProductKey(key)

returns key as getproduct URL-part.

If key is a string, it is quoted as a naked accref so it’s usable as the path part of an URL. If it’s an RAccref, it is just stringified. The result is something that can be used after getproduct in URLs in any case.

gavo.protocols.products.unquoteProductKey(key)

reverses quoteProductKey.