gavo.utils.misctricks module

Various helpers that didn’t fit into any other xTricks.

class gavo.utils.misctricks.CaseSemisensitiveDict(*args, **kwargs)

Bases: dict

A dictionary allowing case-insensitive access to its content.

This is used for DAL renderers which, unfortunately, are supposed to be case insensitive. Since case insensitivity is at least undesirable for service-specific keys, we go a semi-insenstitve approach here: First, we try literal matches, if that does not work, we try matching against an all-uppercase version.

Name clashes resulting from different names being mapped to the same normalized version are handled in some random way. Don’t do this. And don’t rely on case normalization if at all possible.

Only strings are allowed as keys here. This class is not concerned with the values. >>> d = CaseSemisensitiveDict({“a”: 1, “A”: 2, “b”: 3}) >>> d[“a”], d[“A”], d[“b”], d[“B”] (1, 2, 3, 3) >>> d[“B”] = 9; d[“b”], d[“B”] (3, 9) >>> del d[“b”]; d[“b”], d[“B”] (9, 9) >>> “B” in d, “b” in d, “u” in d (True, True, False) >>> d.pop(“a”), list(d.keys()) (1, [‘A’, ‘B’])

copy() → a shallow copy of D

classmethod fromDict(aDict: Union[dict, CaseSemisensitiveDict]) → CaseSemisensitiveDict

get(key: str, default: Any = None) → Any

Return the value for key if key is in the dictionary, else default.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

class gavo.utils.misctricks.QuotedName(name: str)

Bases: object

A string-like thing basically representing SQL delimited identifiers.

This has some features that make handling these relatively painless in ADQL code.

The most horrible feature is that these hash and compare as their embedded names, except to other QuotedNamess.

SQL-92, in 5.2, roughly says:

delimited identifiers compare literally with each other, delimited identifiers compare with regular identifiers after the latter are all turned to upper case. But since postgres turns everything to lower case, we do so here, too.

>>> n1, n2, n3 = QuotedName("foo"), QuotedName('foo"l'), QuotedName("foo")
>>> n1==n2,n1==n3,hash(n1)==hash("foo")
(False, True, True)
>>> print(n1, n2)
"foo" "foo""l"
>>> "Foo"<n1, n1>"bar"
(False, True)
>>> QuotedName('7oh-no"+rob').makeIdentifier()
'id7oh2dno222brob'

capitalize() → str

flatten() → str

isRegularLower() → bool

lower()

makeIdentifier() → str

returns self as something usable as a SQL regular identifier.

This will be rather unreadable if there’s a substantial number of non-letters in there, and of course there’s no absolute guarantee that doesn’t clash with actual identifiers.

This is not for SQL serialisation but mainly for generating sqlKey, where this kind of thing ends up in %(name)s patterns.

class gavo.utils.misctricks.RSTExtensions

Bases: object

a register for local RST extensions.

This is for both directives and interpreted text roles.

We need these as additional markup in examples; these always introduce local rst interpreted text roles, which always add some class to the node in question (modifications are possible).

These classes are then changed to properties as the HTML fragments from RST translation are processed by the _Example nevow data factory.

To add a new text role, say:

RSTExtensions.addRole(roleName, roleFunc=None)

You can pass in a full role function as discussed in …docs/howto/rst-roles.html#define-the-role-function It must, however, add a dachs-ex-<roleName> class to the node. The default function produces a nodes.emphasis item with the proper class.

In a pinch, you can pass a propertyName argument to addRole if the desired property name is distinct from the role name in the RST. This is used by tapquery and taprole since we didn’t want to change our examples when the standard changed.

To add a directive, say:

RSTExtensions.addDirective(dirName, dirClass)

In HTML, these classes become properties named like the role name (except you can again use propertyName in a pinch).

classmethod addDirective(name: str, implementingClass: type, propertyName: Optional[str] = None) → None

classToProperty: Dict[str, str] = {'dachs-ex-bibcode': 'bibcode', 'dachs-ex-dachsdoc': 'dachsdoc', 'dachs-ex-dachsref': 'dachsref', 'dachs-ex-dl-id': 'dl-id', 'dachs-ex-genparam': 'genparam', 'dachs-ex-samplerd': 'samplerd', 'dachs-ex-tapquery': 'query', 'dachs-ex-taptable': 'table'}

classmethod makeTextRole(roleName: str, roleFunc: Optional[Callable[[str, str, str, int, Inliner, Optional[dict], Optional[List[str]]], Tuple[List[Node], List[str]]]] = None, propertyName=None) → None

creates a new text role for roleName.

See class docstring.

class gavo.utils.misctricks.RateLimiter(timeout: float)

Bases: object

A class that helps limit rates of events.

You construct it with a timeout (in seconds) and then protect things you want to rate-limit with “if rl.inDeadtime(key): skip”. The key is an identifier for what it is that you want to limit (e.g., the sort of an event, so that different events can share a rate limiter).

If you have many events that usually need rate limiting, you’d have to revisit this implementation – this is really for when rate limiting is the exception.

inDeadtime(key: str)

class gavo.utils.misctricks.StreamBuffer(chunkSize: Optional[int] = None, binary: bool = True)

Bases: Generic[_StreamData]

a buffer that takes data in arbitrary chunks and returns them in chops of chunkSize bytes.

There’s a lock in place so you can access add and get from different threads.

When everything is written, you must all doneWriting.

add(data: _StreamData) → None

chunkSize = 50000

doneWriting() → None

get(numBytes: Optional[int] = None) → Optional[_StreamData]

getRest() → _StreamData

returns the entire buffer as far as it is left over.

getToChar(char: _StreamData) → Optional[_StreamData]

returns the the buffer up to the first occurrence of char.

If char is not present in the buffer, the function returns None.

class gavo.utils.misctricks.Undefined

Bases: object

a sentinel for all kinds of undefined values.

Do not instantiate.

>>> Undefined()
Traceback (most recent call last):
TypeError: Undefined cannot be instantiated.
>>> bool(Undefined)
False
>>> repr(Undefined)
'<Undefined>'
>>> str(Undefined)
Traceback (most recent call last):
gavo.utils.excs.StructureError: Undefined cannot be stringified.

gavo.utils.misctricks.couldBeABibcode(s: str) → bool

returns true if we think that the string s is a bibcode.

This is based on matching against BIBCODE_PATTERN.

gavo.utils.misctricks.getCleanBytes(b: Union[str, bytes]) → bytes

returns the bytes b in an ASCII representation.

This is zlib-compressed base64 stuff. b can be a string, too, in which case it’s utf-8 encoded before marshalling.

gavo.utils.misctricks.getDirtyBytes(b: bytes) → bytes

returns b decoded and uncompressed.

This is the inverse operation of getCleanBytes. b must be bytes, and bytes is what you get back.

gavo.utils.misctricks.getFortranRec(f: BinaryIO) → Optional[bytes]

reads a “fortran record” from f and returns the payload.

A “fortran record” comes from an unformatted file and has a 4-byte payload length before and after the payload. Native endianness is assumed here.

If the two length specs do not match, a ValueError is raised.

Of course, f must be open in binary mode.

gavo.utils.misctricks.getWithCache(url: str, cacheDir: str, extraHeaders: dict = {}) → bytes

returns the content of url, from a cache if possible.

Of course, you only want to use this if there’s some external guarantee that the resource behind url doesn’t change. No expiry mechanism is present here.

gavo.utils.misctricks.getfirst(args: ~typing.Dict[str, ~gavo.utils.misctricks._T], key: str, default: ~gavo.utils.misctricks._T = <Undefined>) → _T

returns the first value of key in the web argument-like object args.

args is a dictionary mapping keys to lists of values. If key is present, the first element of the list is returned; else, or if the list is empty, default if given. If not, a Validation error for the requested column is raised.

Finally, if args[key] is neither list nor tuple (in an ininstance sense), it is returned unchanged.

>>> getfirst({'x': [1,2,3]}, 'x')
1
>>> getfirst({'x': []}, 'x')
Traceback (most recent call last):
gavo.utils.excs.ValidationError: Field x: Missing mandatory parameter x
>>> getfirst({'x': []}, 'y')
Traceback (most recent call last):
gavo.utils.excs.ValidationError: Field y: Missing mandatory parameter y
>>> print(getfirst({'x': []}, 'y', None))
None
>>> getfirst({'x': 'abc'}, 'x')
'abc'

gavo.utils.misctricks.grouped(n: int, seq: Sequence[_T]) → List[List[_T]]

yields items of seq in groups n elements.

If len(seq)%n!=0, the last elements are discarded.

>>> list(grouped(2, range(5)))
[(0, 1), (2, 3)]
>>> list(grouped(3, range(9)))
[(0, 1, 2), (3, 4, 5), (6, 7, 8)]

gavo.utils.misctricks.iterFortranRecs(f: BinaryIO, skip: int = 0) → Generator[bytes, None, None]

iterates over the fortran records in f.

For details, see getFortranRec.

gavo.utils.misctricks.logOldExc(exc: Exception) → Exception

logs the mutation of the currently handled exception to exc.

This just does a notifyExceptionMutation using sendUIEvent; it should only be used by code at or below base.

gavo.utils.misctricks.makeKVLine(aDict: Dict[str, str]) → str

serializes a dictionary to a key-value line.

See parseKVLine for details.

gavo.utils.misctricks.parseKVLine(aString: str)

returns a dictionary for a “key-value line”.

key-value lines represent string-valued dictionaries following postgres libpq/dsn (see PQconnectdb docs; it’s keyword=value, whitespace-separated, with whitespace allowed in values through single quoting, and backslash-escaping

gavo.utils.misctricks.rstxToHTML(source: str, **userOverrides) → str

returns HTML for a piece of ReStructured text.

source can be a unicode string or a byte string in utf-8.

userOverrides will be added to the overrides argument of docutils’ core.publish_parts.

gavo.utils.misctricks.rstxToHTMLWithWarning(source: str, **userOverrides) → Tuple[str, str]

returns HTML and a string with warnings for a piece of ReStructured text.

source can be a unicode string or a byte string in utf-8.

userOverrides will be added to the overrides argument of docutils’ core.publish_parts.

gavo.utils.misctricks.sendUIEvent(eventName: str, *args) → None

sends an eventName to the DC event dispatcher.

If no event dispatcher is available, do nothing.

The base.ui object that DaCHS uses for event dispatching is only available to sub-packages above base. Other code should not use or need it under normal circumstances, but if it does, it can use this.

All other code should use base.ui.notify<eventName>(*args) directly.