External File Spec¶

Specify metadata that’s in a separate, external file from the standard format files

Classes:

`BaseExternalFileSpec`(path, key, field, …)	Parameters self
`JSON`(hook, args, *kwargs)	Load a field from a .json file.
`Mat`(simplified, args, *kwargs)	Parameters simplified (bool) – Whether we attempt to simplify the matlab struct into lists
`YAML`(path, key, field, Tuple[str, …]], …)	Parameters self

Functions:

load_clean_mat(filename)

Load a matlab .mat file as python lists, dictionaries, and numpy arrays rather than the sort-of hard to work with numpy record arrays.

class onice_conversion.spec.external_file.BaseExternalFileSpec(path: pathlib.Path, key: str, field: Union[str, Tuple[str, …]], cache: bool = True, *args, **kwargs)¶

Bases: onice_conversion.spec.base_spec.BaseSpec

Parameters

self
path (path relative to base_dir that is passed in _parse())
key
field
cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.
kwargs

Attributes:

`loaded_files`
`parent`
`specifies`	Which metadata variables are specified by this Spec object and its children

Methods:

`__init__`(path, key, field[, cache])	Parameters self
`_load_file`(path)	Load the file and return it as a nested dictionary of dictionaries or tuples
`_sub_select`(loaded_file)	Use `field` to select from the loaded_file
`_expand_named_fields`(named_fields)	Convert nested key specs like key[key2] into nested dicts
`_full_name`()	Returns the full module and class name of an object, eg.
`_get_init_args`()	introspect object and get all arguments passed on __init__
`children`()	Generator for iterating over children (added)
`parse`(base_path[, metadata])	Parse all parameters from self and child `_parse()` methods, combining into single dictionary
`to_dict`()	Get a dictionary description of this spec object, of the form.

loaded_files = {}¶

__init__(path: pathlib.Path, key: str, field: Union[str, Tuple[str, …]], cache: bool = True, *args, **kwargs)¶

Parameters

self
path (path relative to base_dir that is passed in _parse())
key
field
cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.
kwargs

abstract _load_file(path: pathlib.Path) → dict ¶

Load the file and return it as a nested dictionary of dictionaries or tuples

such that it can be indexed by successively slicing with field

Parameters

self
key (str) – name of the property that will be returned
path

_sub_select(loaded_file: dict) → Any¶

Use field to select from the loaded_file

Parameters: loaded_file

_expand_named_fields(named_fields)¶

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()¶

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Returns
Return type: str

_get_init_args()¶

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Returns
Return type: dict of argument names and params

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]¶: Generator for iterating over children (added)

property parent¶

parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict ¶

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

base_path (Path) – The base path we compute the spec’d value from!
metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies¶

Which metadata variables are specified by this Spec object and its children

Returns
Return type: tuple of strings

to_dict() → dict ¶

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Returns
Return type: dict of initialization parameters, as described above

class onice_conversion.spec.external_file.JSON(hook: Optional[Callable] = None, *args, **kwargs)¶

Bases: onice_conversion.spec.external_file.BaseExternalFileSpec

Load a field from a .json file. see base class for docs

Parameters

hook (Optionally, include some callable function to use as the fallback) – object loader hook (see object_hook argument in json.load for more information)
args (passed to BaseExternalFileSpec)
kwargs

Methods:

`__init__`([hook])	Load a field from a .json file.
`_expand_named_fields`(named_fields)	Convert nested key specs like key[key2] into nested dicts
`_full_name`()	Returns the full module and class name of an object, eg.
`_get_init_args`()	introspect object and get all arguments passed on __init__
`_sub_select`(loaded_file)	Use `field` to select from the loaded_file
`children`()	Generator for iterating over children (added)
`parse`(base_path[, metadata])	Parse all parameters from self and child `_parse()` methods, combining into single dictionary
`to_dict`()	Get a dictionary description of this spec object, of the form.

Attributes:

`loaded_files`
`parent`
`specifies`	Which metadata variables are specified by this Spec object and its children

__init__(hook: Optional[Callable] = None, *args, **kwargs)¶

Load a field from a .json file. see base class for docs

Parameters

hook (Optionally, include some callable function to use as the fallback) – object loader hook (see object_hook argument in json.load for more information)
args (passed to BaseExternalFileSpec)
kwargs

_expand_named_fields(named_fields)¶

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()¶

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Returns
Return type: str

_get_init_args()¶

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Returns
Return type: dict of argument names and params

_sub_select(loaded_file: dict) → Any¶

Use field to select from the loaded_file

Parameters: loaded_file

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]¶: Generator for iterating over children (added)

loaded_files = {}¶

property parent¶

parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict ¶

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

base_path (Path) – The base path we compute the spec’d value from!
metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies¶

Which metadata variables are specified by this Spec object and its children

Returns
Return type: tuple of strings

to_dict() → dict ¶

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Returns
Return type: dict of initialization parameters, as described above

class onice_conversion.spec.external_file.Mat(simplified: bool = True, *args, **kwargs)¶

Bases: onice_conversion.spec.external_file.BaseExternalFileSpec

Parameters

simplified (bool) – Whether we attempt to simplify the matlab struct into lists and dicts, or just take the base output from scipy.io.loadmat()
*args () – Passed to superclass
**kwargs () – Passed to superclass

Methods:

`__init__`([simplified])	Parameters simplified (bool) – Whether we attempt to simplify the matlab struct into lists
`_sub_select`(loaded_file)	Calls `BaseExternalFileSpec._sub_select()`, but then unstacks all len == 1 numpy arrays so that the field arg can be like (‘sessionInfo’, ‘session’) rather than (‘sessionInfo’, ‘session’, 0, 0, 0, 0, 0, 0)
`_expand_named_fields`(named_fields)	Convert nested key specs like key[key2] into nested dicts
`_full_name`()	Returns the full module and class name of an object, eg.
`_get_init_args`()	introspect object and get all arguments passed on __init__
`children`()	Generator for iterating over children (added)
`parse`(base_path[, metadata])	Parse all parameters from self and child `_parse()` methods, combining into single dictionary
`to_dict`()	Get a dictionary description of this spec object, of the form.

Attributes:

`loaded_files`
`parent`
`specifies`	Which metadata variables are specified by this Spec object and its children

__init__(simplified: bool = True, *args, **kwargs)¶

Parameters

simplified (bool) – Whether we attempt to simplify the matlab struct into lists and dicts, or just take the base output from scipy.io.loadmat()
*args () – Passed to superclass
**kwargs () – Passed to superclass

_sub_select(loaded_file: dict) → Any¶

Calls BaseExternalFileSpec._sub_select(), but then unstacks all len == 1 numpy arrays so that the field arg can be like (‘sessionInfo’, ‘session’) rather than (‘sessionInfo’, ‘session’, 0, 0, 0, 0, 0, 0)

Parameters: loaded_file

_expand_named_fields(named_fields)¶

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()¶

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Returns
Return type: str

_get_init_args()¶

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Returns
Return type: dict of argument names and params

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]¶: Generator for iterating over children (added)

loaded_files = {}¶

property parent¶

parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict ¶

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

base_path (Path) – The base path we compute the spec’d value from!
metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies¶

Which metadata variables are specified by this Spec object and its children

Returns
Return type: tuple of strings

to_dict() → dict ¶

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Returns
Return type: dict of initialization parameters, as described above

class onice_conversion.spec.external_file.YAML(path: pathlib.Path, key: str, field: Union[str, Tuple[str, …]], cache: bool = True, *args, **kwargs)¶

Bases: onice_conversion.spec.external_file.BaseExternalFileSpec

Parameters

self
path (path relative to base_dir that is passed in _parse())
key
field
cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.
kwargs

Methods:

`__init__`(path, key, field[, cache])	Parameters self
`_expand_named_fields`(named_fields)	Convert nested key specs like key[key2] into nested dicts
`_full_name`()	Returns the full module and class name of an object, eg.
`_get_init_args`()	introspect object and get all arguments passed on __init__
`_sub_select`(loaded_file)	Use `field` to select from the loaded_file
`children`()	Generator for iterating over children (added)
`parse`(base_path[, metadata])	Parse all parameters from self and child `_parse()` methods, combining into single dictionary
`to_dict`()	Get a dictionary description of this spec object, of the form.

Attributes:

`loaded_files`
`parent`
`specifies`	Which metadata variables are specified by this Spec object and its children

__init__(path: pathlib.Path, key: str, field: Union[str, Tuple[str, …]], cache: bool = True, *args, **kwargs)¶

Parameters

self
path (path relative to base_dir that is passed in _parse())
key
field
cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.
kwargs

_expand_named_fields(named_fields)¶

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()¶

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Returns
Return type: str

_get_init_args()¶

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Returns
Return type: dict of argument names and params

_sub_select(loaded_file: dict) → Any¶

Use field to select from the loaded_file

Parameters: loaded_file

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]¶: Generator for iterating over children (added)

loaded_files = {}¶

property parent¶

parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict ¶

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

base_path (Path) – The base path we compute the spec’d value from!
metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies¶

Which metadata variables are specified by this Spec object and its children

Returns
Return type: tuple of strings

to_dict() → dict ¶

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Returns
Return type: dict of initialization parameters, as described above

onice_conversion.spec.external_file.load_clean_mat(filename: str) → dict ¶

Load a matlab .mat file as python lists, dictionaries, and numpy arrays rather than the sort-of hard to work with numpy record arrays.

Credit to https://stackoverflow.com/a/29126361/13113166

Parameters: filename (str) – filename of .mat to load
Returns: dict