External File Spec#

Specify metadata that’s in a separate, external file from the standard format files

Classes:

BaseExternalFileSpec(path, key, field[, cache])

Parameters

  • self

JSON([hook])

Load a field from a .json file.

Mat([simplified])

Parameters

  • simplified (bool) -- Whether we attempt to simplify the matlab struct into lists

YAML(path, key, field[, cache])

Parameters

  • self

Functions:

load_clean_mat(filename)

Load a matlab .mat file as python lists, dictionaries, and numpy arrays rather than the sort-of hard to work with numpy record arrays.
class onice_conversion.spec.external_file.BaseExternalFileSpec(path: pathlib.Path, key: str, field: Union[str, Tuple[str, ...]], cache: bool = True, *args, **kwargs)#

Bases: onice_conversion.spec.base_spec.BaseSpec

Parameters

  • self

  • path (path relative to base_dir that is passed in _parse())

  • key

  • field

  • cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.

  • kwargs

Attributes:

loaded_files

parent

specifies

Which metadata variables are specified by this Spec object and its children

Methods:

__init__(path, key, field[, cache])

Parameters

  • self

_load_file(path)

Load the file and return it as a nested dictionary of dictionaries or tuples

_sub_select(loaded_file)

Use field to select from the loaded_file

_expand_named_fields(named_fields)

Convert nested key specs like key[key2] into nested dicts

_full_name()

Returns the full module and class name of an object, eg.

_get_init_args()

introspect object and get all arguments passed on __init__

children()

Generator for iterating over children (added)

parse(base_path[, metadata])

Parse all parameters from self and child _parse() methods, combining into single dictionary

to_dict()

Get a dictionary description of this spec object, of the form.
loaded_files = {}#
__init__(path: pathlib.Path, key: str, field: Union[str, Tuple[str, ...]], cache: bool = True, *args, **kwargs)#
Parameters

  • self

  • path (path relative to base_dir that is passed in _parse())

  • key

  • field

  • cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.

  • kwargs

abstract _load_file(path: pathlib.Path) → dict#

Load the file and return it as a nested dictionary of dictionaries or tuples

such that it can be indexed by successively slicing with field

Parameters

  • self

  • key (str) – name of the property that will be returned

  • path

_sub_select(loaded_file: dict) → Any#

Use field to select from the loaded_file

Parameters

loaded_file

_expand_named_fields(named_fields)#

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()#

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Return type

str

_get_init_args()#

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Return type

dict of argument names and params

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]#

Generator for iterating over children (added)

property parent: onice_conversion.spec.base_spec.BaseSpec#
parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict#

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

  • base_path (Path) – The base path we compute the spec’d value from!

  • metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies: Tuple[str, ...]#

Which metadata variables are specified by this Spec object and its children

Return type

tuple of strings

to_dict() → dict#

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Return type

dict of initialization parameters, as described above

class onice_conversion.spec.external_file.JSON(hook: Optional[Callable] = None, *args, **kwargs)#

Bases: onice_conversion.spec.external_file.BaseExternalFileSpec

Load a field from a .json file. see base class for docs

Parameters

  • hook (Optionally, include some callable function to use as the fallback) – object loader hook (see object_hook argument in json.load for more information)

  • args (passed to BaseExternalFileSpec)

  • kwargs

Methods:

__init__([hook])

Load a field from a .json file.

_expand_named_fields(named_fields)

Convert nested key specs like key[key2] into nested dicts

_full_name()

Returns the full module and class name of an object, eg.

_get_init_args()

introspect object and get all arguments passed on __init__

_sub_select(loaded_file)

Use field to select from the loaded_file

children()

Generator for iterating over children (added)

parse(base_path[, metadata])

Parse all parameters from self and child _parse() methods, combining into single dictionary

to_dict()

Get a dictionary description of this spec object, of the form.

Attributes:

loaded_files

parent

specifies

Which metadata variables are specified by this Spec object and its children
__init__(hook: Optional[Callable] = None, *args, **kwargs)#

Load a field from a .json file. see base class for docs

Parameters

  • hook (Optionally, include some callable function to use as the fallback) – object loader hook (see object_hook argument in json.load for more information)

  • args (passed to BaseExternalFileSpec)

  • kwargs

_expand_named_fields(named_fields)#

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()#

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Return type

str

_get_init_args()#

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Return type

dict of argument names and params

_sub_select(loaded_file: dict) → Any#

Use field to select from the loaded_file

Parameters

loaded_file

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]#

Generator for iterating over children (added)

loaded_files = {}#
property parent: onice_conversion.spec.base_spec.BaseSpec#
parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict#

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

  • base_path (Path) – The base path we compute the spec’d value from!

  • metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies: Tuple[str, ...]#

Which metadata variables are specified by this Spec object and its children

Return type

tuple of strings

to_dict() → dict#

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Return type

dict of initialization parameters, as described above

class onice_conversion.spec.external_file.Mat(simplified: bool = True, *args, **kwargs)#

Bases: onice_conversion.spec.external_file.BaseExternalFileSpec

Parameters

  • simplified (bool) – Whether we attempt to simplify the matlab struct into lists and dicts, or just take the base output from scipy.io.loadmat()

  • *args () – Passed to superclass

  • **kwargs () – Passed to superclass

Methods:

__init__([simplified])

Parameters

  • simplified (bool) -- Whether we attempt to simplify the matlab struct into lists

_sub_select(loaded_file)

Calls BaseExternalFileSpec._sub_select(), but then unstacks all len == 1 numpy arrays so that the field arg can be like ('sessionInfo', 'session') rather than ('sessionInfo', 'session', 0, 0, 0, 0, 0, 0)

_expand_named_fields(named_fields)

Convert nested key specs like key[key2] into nested dicts

_full_name()

Returns the full module and class name of an object, eg.

_get_init_args()

introspect object and get all arguments passed on __init__

children()

Generator for iterating over children (added)

parse(base_path[, metadata])

Parse all parameters from self and child _parse() methods, combining into single dictionary

to_dict()

Get a dictionary description of this spec object, of the form.

Attributes:

loaded_files

parent

specifies

Which metadata variables are specified by this Spec object and its children
__init__(simplified: bool = True, *args, **kwargs)#
Parameters

  • simplified (bool) – Whether we attempt to simplify the matlab struct into lists and dicts, or just take the base output from scipy.io.loadmat()

  • *args () – Passed to superclass

  • **kwargs () – Passed to superclass

_sub_select(loaded_file: dict) → Any#

Calls BaseExternalFileSpec._sub_select(), but then unstacks all len == 1 numpy arrays so that the field arg can be like (‘sessionInfo’, ‘session’) rather than (‘sessionInfo’, ‘session’, 0, 0, 0, 0, 0, 0)

Parameters

loaded_file

_expand_named_fields(named_fields)#

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()#

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Return type

str

_get_init_args()#

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Return type

dict of argument names and params

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]#

Generator for iterating over children (added)

loaded_files = {}#
property parent: onice_conversion.spec.base_spec.BaseSpec#
parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict#

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

  • base_path (Path) – The base path we compute the spec’d value from!

  • metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies: Tuple[str, ...]#

Which metadata variables are specified by this Spec object and its children

Return type

tuple of strings

to_dict() → dict#

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Return type

dict of initialization parameters, as described above

class onice_conversion.spec.external_file.YAML(path: pathlib.Path, key: str, field: Union[str, Tuple[str, ...]], cache: bool = True, *args, **kwargs)#

Bases: onice_conversion.spec.external_file.BaseExternalFileSpec

Parameters

  • self

  • path (path relative to base_dir that is passed in _parse())

  • key

  • field

  • cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.

  • kwargs

Methods:

__init__(path, key, field[, cache])

Parameters

  • self

_expand_named_fields(named_fields)

Convert nested key specs like key[key2] into nested dicts

_full_name()

Returns the full module and class name of an object, eg.

_get_init_args()

introspect object and get all arguments passed on __init__

_sub_select(loaded_file)

Use field to select from the loaded_file

children()

Generator for iterating over children (added)

parse(base_path[, metadata])

Parse all parameters from self and child _parse() methods, combining into single dictionary

to_dict()

Get a dictionary description of this spec object, of the form.

Attributes:

loaded_files

parent

specifies

Which metadata variables are specified by this Spec object and its children
__init__(path: pathlib.Path, key: str, field: Union[str, Tuple[str, ...]], cache: bool = True, *args, **kwargs)#
Parameters

  • self

  • path (path relative to base_dir that is passed in _parse())

  • key

  • field

  • cache (bool) – if True, store loaded file in loaded_files dictionary to prevent re-load if another spec needs it.

  • kwargs

_expand_named_fields(named_fields)#

Convert nested key specs like key[key2] into nested dicts

borroed from: https://github.com/r1chardj0n3s/parse/blob/0477aa58673cd957c19d377e029347ce72c08b1b/parse.py#L944

_full_name()#

Returns the full module and class name of an object, eg. nwb_conversion_tools.spec.external_file.JSON

Return type

str

_get_init_args()#

introspect object and get all arguments passed on __init__

depends on introspecting up frames so should only be called during the top-level __init__ of the base class :)

Return type

dict of argument names and params

_sub_select(loaded_file: dict) → Any#

Use field to select from the loaded_file

Parameters

loaded_file

children() → Iterable[onice_conversion.spec.base_spec.BaseSpec]#

Generator for iterating over children (added)

loaded_files = {}#
property parent: onice_conversion.spec.base_spec.BaseSpec#
parse(base_path: pathlib.Path, metadata: Optional[dict] = None) → dict#

Parse all parameters from self and child _parse() methods, combining into single dictionary

Parameters

  • base_path (Path) – The base path we compute the spec’d value from!

  • metadata (dict) – other metadata used by the parsing function, usually passed in NWBConverter.run_conversion()

property specifies: Tuple[str, ...]#

Which metadata variables are specified by this Spec object and its children

Return type

tuple of strings

to_dict() → dict#

Get a dictionary description of this spec object, of the form:

{
    'module': self.__module__,
    'class': type(self).__name__,
    'kwargs': self._init_args,
    'children': [ ... same structure as top-level without children list ...]
}

That allows a spec to be reconstituted with from_dict()

Return type

dict of initialization parameters, as described above

onice_conversion.spec.external_file.load_clean_mat(filename: str) → dict#

Load a matlab .mat file as python lists, dictionaries, and numpy arrays rather than the sort-of hard to work with numpy record arrays.

Credit to https://stackoverflow.com/a/29126361/13113166

Parameters

filename (str) – filename of .mat to load

Returns

dict