Classes and decorator

Class definitions supporting the computation graphs.

class paragraph.types.Variable(name=None, op=None, args=NOTHING, dependencies=NOTHING)

Bases: object

A generic variable.

This class is the return type of all operations in a computation graph. Independent variables can be instantiated directly as follows:

>>> my_var = Variable(name="my_var")

Note

Type annotations are missing for attributes op and args, for Sphinx does not seem to cope with forward references.

name

a string used to represent the variable. The attribute is mandatory for independent variables and None (the default) otherwise.

op

the operation producing the variable, of type Op.

args

a dictionary mapping arguments of the above op onto their values, of type Dict[Variable, Any].

dependencies

a dictionary mapping arguments of the above callable onto other variables, of type Dict[str, Variable].

isinput()
Return type:bool
isdependent()
Return type:bool
class paragraph.types.Requirement

Bases: abc.ABC

Base class for defining a requirement mixin

A Requirement class defines one or several attributes storing the actual requirement being expressed, and takes complete responsibility of these attributes. Therefore, a concrete requirement class should:

  • define default values or factory callbacks for all the attributes it introduces,
  • redefine the merge() method below appropriately, respecting the prescribed constraints,
  • expose convenience methods to manipulate their attributes, these methods should operate in place to avoid unwanted interference with other Requirement mixins,
  • provide an implementation of __deepcopy__() whenever relevant .

Deriving a compound requirements class boils down to putting together requirement mixins:

>>> @attr.s
>>> class MyRequirements(DateRangeRequirement, DatasetContentsRequirement):
...     pass

assuming DateRangeRequirement and DatasetContentsRequirement derive from the present class.

merge(other)

Merges other into self in-place, must be implemented by all cooperating mixin classes.

The requirement mixins should define the merge() method for the attributes they define (and therefore take responsibility for). The computation implemented must be both:

commutative
the result of [self.merge(usg) for usg in usages[self]] does not depend on the order of the usages in the list
idempotent
if usg1 = usg2, invoking self.update(usg2) after self.update(usg1) leaves self unchanged

Only under these conditions is it guaranteed that all backward traversals of the computation graph result in the same input requirements.

In addition, every implementation of this method must be cooperative and end with the following line:

super().merge(other)
Parameters:other – the requirement to merge into self, must be of the same concrete type as self.
new()

Return an empty requirement of the same type as self.

class paragraph.types.Op(*, thread_safe=True)

Bases: object

Class of all computation graph operations.

A concrete Op class should redefine the _run() method, which fully specifies its behavior. Calling an Operation instance results in the following:

  • if no argument is of Variable type, the return value of the _run() method is returned,
  • otherwise, a Variable is returned, which represents the result of the operation applied to its arguments.

Additional requirements can be introduced for variable arguments, globally or individually, by redefining the _arg_requirements() method appropriately.

thread_safe

If False, the op is always executed in the main thread. Defaults to True.

static split_args(args)

Separate positional from keyword arguments in a dict.

This method extracts args entries with a key of type integer, and collates them into a list

Return type:Tuple[List[Any], Dict[str, Any]]
arg_requirements(req, arg=None)

Compute the requirements on the input value for argument arg from the requirements req bearing on the output variable.

The base implementation below returns an empty requirement for every argument, preserving the type. Concrete classes should redefine this method whenever applicable.

Warning

This method should never update requirements in-place, as this would result in adversary side-effects. The recommended practice is for this method to: - instantiate a new Requirement instance or deep-copy the passed in req if information should be retained, - update the new instance using in-place methods exposed by the requirement class, and return it.

Should the output requirement be returned unmodified, it is however safe to just return its reference as-is.

Return type:Requirement
op(*args, **kwargs)

Define a variable as the result of applying the op to the arguments provided.

While this method always returns a Variable instance, it accepts:
  • _concrete_ arguments of the type expected by _run at the same position/for the same keyword,
  • Variable instances that resolve to a value of the expected type.
Return type:Variable
paragraph.types.op(func)

Wraps a function within an Op object.

The returned function accepts arguments of type Variable everywhere in its signature, in addition to the types accepted by the decorated function. In presence of such arguments, it returns a Variable object symbolizing the result of the operation with the arguments passed in. In absence of variable arguments, the function returned is just equivalent to the function passed in, in particular it returns a value of the pristine return type.

Warning

Operations returned by this decorator are marked thread-safe by default. It is the user’s responsibility to set Op.thread_safe to False where appropriate.

Parameters:func (Callable) – the function to transform into an Op
Return type:Op