Download Jupyter Notebook

Datasource¶

A datasource is an external data that is needed for the toolkit.

The data is usually loaded by some repository of data. Note, that the loaded data is specific for a project. Hence, it is needed to load it seperately for each project.

Internally, datasource is implemented as a data item saved as a measurement document. The type and the resource is determined by the user that added them.

First, Let's add a toolkit. for this example we use GIS_DEMOGRAPHY toolkit.

In [1]:

from hera import toolkitHome

toolkitName = toolkitHome.GIS_DEMOGRAPHY
projectName = "The-Project-Name"
toolkit_specific_parameters = dict() # empty for this presentation. 
tk  = toolkitHome.getToolkit(toolkitName=toolkitName,
                             projectName=projectName,
                             **toolkit_specific_parameters)

from hera import toolkitHome toolkitName = toolkitHome.GIS_DEMOGRAPHY projectName = "The-Project-Name" toolkit_specific_parameters = dict() # empty for this presentation. tk = toolkitHome.getToolkit(toolkitName=toolkitName, projectName=projectName, **toolkit_specific_parameters)

Adding a datasource¶

Adding a datasource to a toolkit is performed by addDataSource property.

The parameters are:

• dataSourceName : str
  The name of the data source

• resource : str
  The path to the data file of the datasource.

• dataFormat : str
  The format of the data.

• version: tuple, default (0,0,1)
  The version of the datasouce. This allows you to add different version of the datasource and access the currect one.

• overwrite: bool, default: False
  If True, overwrite the existing datasource (of the input version). If False, raise an exception if the datasource exists (with the input version).

• Additional parameters:
  additional parameters name and their values for the datasource.

In [2]:

tk.addDataSource(dataSourceName="thedata",
                 resource="path-to-data",
                 dataFormat=tk.datatypes.STRING,
                 version=(0,0,1),overwrite=True)

tk.addDataSource(dataSourceName="thedata", resource="path-to-data", dataFormat=tk.datatypes.STRING, version=(0,0,1),overwrite=True)

Out[2]:

<Measurements: {
    "_cls": "Metadata.Measurements",
    "projectName": "The-Project-Name",
    "desc": {
        "toolkit": "Demography",
        "datasourceName": "thedata",
        "version": [
            0,
            0,
            1
        ]
    },
    "type": "ToolkitDataSource",
    "resource": "path-to-data",
    "dataFormat": "string"
}>

Adding another version of the datasource

In [3]:

tk.addDataSource(dataSourceName="thedata",
                 resource="path-to-data-2",
                 dataFormat=tk.datatypes.STRING,
                 overwrite=True,
                 version=(0,0,2),key="value")

tk.addDataSource(dataSourceName="thedata", resource="path-to-data-2", dataFormat=tk.datatypes.STRING, overwrite=True, version=(0,0,2),key="value")

Out[3]:

<Measurements: {
    "_cls": "Metadata.Measurements",
    "projectName": "The-Project-Name",
    "desc": {
        "key": "value",
        "toolkit": "Demography",
        "datasourceName": "thedata",
        "version": [
            0,
            0,
            2
        ]
    },
    "type": "ToolkitDataSource",
    "resource": "path-to-data-2",
    "dataFormat": "string"
}>

Listing the datasources¶

List the datasources that were added to the project is performed by

In [4]:

tk.getDataSourceTable()

tk.getDataSourceTable()

Out[4]:

	dataFormat	resource	toolkit	datasourceName	version	key
0	string	path-to-data	Demography	thedata	[0, 0, 1]	NaN
1	string	path-to-data-2	Demography	thedata	[0, 0, 2]	value

It is possible to filter the datasources using the key/value

In [5]:

tk.getDataSourceTable(key="value")

tk.getDataSourceTable(key="value")

Out[5]:

	dataFormat	resource	key	toolkit	datasourceName	version
0	string	path-to-data-2	value	Demography	thedata	[0, 0, 2]

Alternativle, it is possible to get the datasource as a list of dictionaryes.

In [6]:

tk.getDataSourceMap()

tk.getDataSourceMap()

Out[6]:

[{'dataFormat': 'string',
  'resource': 'path-to-data',
  'toolkit': 'Demography',
  'datasourceName': 'thedata',
  'version': [0, 0, 1]},
 {'dataFormat': 'string',
  'resource': 'path-to-data-2',
  'key': 'value',
  'toolkit': 'Demography',
  'datasourceName': 'thedata',
  'version': [0, 0, 2]}]

Getting the datasource¶

It is possible to retrieve either the metadata document (datasource document) or the data itself.

To get the data document we use

In [7]:

import json 

datasourceName  = "thedata"
doc = tk.getDataSourceDocument(datasourceName=datasourceName)
print(json.dumps(doc.desc,indent=4))

import json datasourceName = "thedata" doc = tk.getDataSourceDocument(datasourceName=datasourceName) print(json.dumps(doc.desc,indent=4))

{
    "key": "value",
    "toolkit": "Demography",
    "datasourceName": "thedata",
    "version": [
        0,
        0,
        2
    ]
}

If the version is not specified, the function will return the highest version.

It is possible to set the version to get the relevant datasource

In [8]:

doc = tk.getDataSourceDocument(datasourceName=datasourceName,version=(0,0,1))
print(json.dumps(doc.desc,indent=4))

doc = tk.getDataSourceDocument(datasourceName=datasourceName,version=(0,0,1)) print(json.dumps(doc.desc,indent=4))

{
    "toolkit": "Demography",
    "datasourceName": "thedata",
    "version": [
        0,
        0,
        1
    ]
}

Getting the data is possible by using the getData

In [9]:

doc.getData()

doc.getData()

Out[9]:

'path-to-data'

It is possible to get a list of all the datasources

In [10]:

tk.getDataSourceDocumentsList()

tk.getDataSourceDocumentsList()

Out[10]:

[<Measurements: {
    "_cls": "Metadata.Measurements",
    "projectName": "The-Project-Name",
    "desc": {
        "toolkit": "Demography",
        "datasourceName": "thedata",
        "version": [
            0,
            0,
            1
        ]
    },
    "type": "ToolkitDataSource",
    "resource": "path-to-data",
    "dataFormat": "string"
}>, <Measurements: {
    "_cls": "Metadata.Measurements",
    "projectName": "The-Project-Name",
    "desc": {
        "key": "value",
        "toolkit": "Demography",
        "datasourceName": "thedata",
        "version": [
            0,
            0,
            2
        ]
    },
    "type": "ToolkitDataSource",
    "resource": "path-to-data-2",
    "dataFormat": "string"
}>]

It is also possible to get the datasource data directly

In [11]:

tk.getDataSourceData(datasourceName=datasourceName)

tk.getDataSourceData(datasourceName=datasourceName)

Out[11]:

'path-to-data-2'

Delete datasource¶

Deleting the datasource is achieved by

In [12]:

tk.deleteDataSource(datasourceName=datasourceName,version=(0,0,1))

tk.deleteDataSource(datasourceName=datasourceName,version=(0,0,1))

Out[12]:

<Measurements: {
    "_cls": "Metadata.Measurements",
    "projectName": "The-Project-Name",
    "desc": {
        "toolkit": "Demography",
        "datasourceName": "thedata",
        "version": [
            0,
            0,
            1
        ]
    },
    "type": "ToolkitDataSource",
    "resource": "path-to-data",
    "dataFormat": "string"
}>

In [13]:

tk.getDataSourceTable()

tk.getDataSourceTable()

Out[13]:

	dataFormat	resource	key	toolkit	datasourceName	version
0	string	path-to-data-2	value	Demography	thedata	[0, 0, 2]