h1. Contents
{toc:maxLevel=2|printable=false}
h1. Task List
{tasklist:SCAPE components TODO list|sortAscending=false|sortBy=priority}
||Completed||Priority||Locked||CreatedDate||CompletedDate||Assignee||Name||
|T|H|F|1386074420265|1386074456386|mferreira|Support annotation extraction and query in myExperiment (MAN)|
|T|H|F|1382444187590|1385029411826|dkf|Implement, test and release extended myExperiment API -- working on component plugin, myE API to go live this week|
|F|H|F|1386074415709| |mferreira|Generate Components for existings tools and publish to MyExperiment (PC developers)|
|F|H|F|1385031044015| |duretec|Finalize component profile specification|
|F|M|F|1385029988018| |dkf|Validate workflow according to profile|
|T|M|F|1382444244284|1390914168462|pmay|Review component end-2-end workflow whitepaper comments and update|
|T|M|F|1385030153502|1390914198825|pmay|Report status of support for complex annotations to Markus P|
|F|M|F|1386074590967| |mferreira|Extract and validate dependencies on the Platform (AIT)|
|F|M|F|1382444161234| |pmay|Enhance toolwrapper (KEEPS, started)|
|F|M|F|1386074793810| |mferreira|Component Validation on Taverna (MAN)|
|T|M|F|1382444210218|1382444218307|mferreira|Clarification from Per about external post-processing step for command line tool components -- Miguel talked to Per and point was taken. Post-processing will happen within the component.|
|T|M|F|1382444230959|1386074886525|pmay|Check status of tasks in Preservation Components, Workflows, Planning and Platform integration -- outdated page|
|F|M|F|1382447500453| |all|[All] Update "developments" as per Toolwrapper example|
{tasklist}
h1. {color:#000000}*"Things" under development{*}{color}
h2. Components specs
h3.
Examples of SCAPE components: [https://github.com/openplanets/scape-component-profiles/tree/master/examples]
h2. Component profiles spec
h3.
The component profiles are available in the [component profile repository|https://github.com/openplanets/scape-component-profiles/tree/master/profiles].
The used ontology is also developed in this repository and published on [http://purl.org/DP/components] alongside with documentation.
h2. MyExperiment API
h3.
(supports creating, searching and managing components)
This is anticipated to go live _this week_ (by 25/10/13). ([Documentation link.|http://wiki.myexperiment.org/index.php/Developer:Components])
h2. Component plugin for taverna
h3.
Currently testing version 1.1.0 (will release once it works\!) ([Browse source.|https://code.google.com/p/taverna/source/browse/#svn%2Funsorted%2Fcomponent%2Fbranches%2Fcomponent-1.1.0])
h2. Toolwrapper - Wrapped tools from PC
| *Module name* | Toolwrapper |
| *Description* | The toolwrapper is a tool that, based on a small specification file, is able to produce the following outputs:
* Bash scripts that standardizes the invocation of Preservation tools
* Debian packages for easy installation \\
\\
The toolwrapper will be enhanced to also support the following features: \\
\\
* Generate SCAPE components according to a Component Profile (Characterization, QA, and Actions)
* Publish generated components to MyExperiment |
| *WP* | PC.WP.2 - Action Services |
| *Responsible* | Miguel Ferreira (WP lead) |
h4.
h3. Tasks
|| Task || Depends of || Status || Responsible || Due date ||
| Development of new version of the toolwrapper | {color:#000000}Component examples{color} | {color:#339966}{*}DONE (see github)*{color} | Hélder Silva (KEEPS) | December |
| Development of new tool that publishes SCAPE components to MyExperiment | MyExperiment API | {color:#339966}{*}UNDER DEVELOPMENT{*}{color} \\ | Hélder Silva (KEEPS) \\ | December \\ |
h3. References
|| Useful information || Description || URL ||
| SCAPE Component examples | Examples of how components should look like. These will be used as templates \\
by the new version of the ToolWrapper so that Components are produced automatically \\
by this tool | +[+Github: SCAPE Component Profiles examples+|https://github.com/openplanets/scape-component-profiles/tree/master/examples]+ \\ |
| SCAPE Component profiles | Profiles that validate that a SCAPE component is according to the right specification. \\
There are 3 types of profiles. | +[+Github: SCAPE Component Profiles+|https://github.com/openplanets/scape-component-profiles/tree/master/profiles]+ \\ |
| MyExperiment API documentation | Documentation of the API provided by MyExperiment to support the publication of \\
Components by the new tool to be developed | +[+MyExperiment: Components API+|http://wiki.myexperiment.org/index.php/Developer:Components]+ \\ |
h3. Issues
|| Issue description || Reported by || Caused by || Solution owner ||
| | | | |
| | | | |
| | | | |
h2. Execution module to run components on the platform
h3.
We need feedback from Rainer.
Main question: Will we be able to submit a SCAPE component to the Job execution service and have it run on the platform.
h2. Integration of Plato and Components
| *Module name* | Plato |
| *Description* | Preservation planning currently does
* Query myExperiment for migration components (based on samples, user input)
* Configure migration components based on parameter ports
* Run selected migration components
* Query CC/QA components defined on decision tree and run them \\
\\
Enhancements \\
\\
* Allow better selection of CC/QA components
* Create an executable plan template containing input/output ports and migration components
* Allow configuration of CC/QA components
* Add CC/QA components to template |
| *WP* | PW.WP.3 Automated Planning |
| *Responsible* | Kresimir Duretec (WP lead) |
h3. Tasks
|| Task || Depends on || Status || Responsible || Due date ||
| Executable Plan Template | {color:#000000}MyExperiment API, Component Profiles{color} | {color:#ff6600}{*}PENDING{*}{color} | Markus Plangg (TUW) | December |
h3. References
|| Useful information || Description || URL ||
| | | |
h3. Issues
|| Issue description || Reported by || Caused by || Solution owner ||
| | | | |
| | | | |
| | | | |
h1. Integration issues
h2. PC Requirements
On a previous PC meeting held in Paris the following whitepaper has been drafted according to PC members perspective. [^SCAPE_Preservation Compoments Profiles_KEEPS_V0.1.docx]
The document has been shared with PW at the time.
h2. PC/PT
How to handle non-1-to-1 migrations? _(Current_ _[migration component profile|https://github.com/openplanets/scape-component-profiles/blob/master/profiles/MigrationActionComponentProfile.xml]_ _handles this correctly; output files are presented component outputs, and both inputs and outputs can be singletons (=filename) or lists of singletons, so 1-many, many-1 and many-many transformations can be represented.)_
h2. PT/PW
How to express measures so that they can be assessed ("do we write this object back to the repository?")
* *Proposal:* Produce a common format to contain the measures. This common container format can then be combined (using some sort of standardised combiner component) so that a single document is presented to an assessment component; the boolean output of that assessment component is then used to guide what happens next (i.e., “write back to repository” or “complain to user about a failed migration”.)
** Needs basic worker components for container document construction, container document combination, container document assessment
** Container format needs to be logically a collection of triples: subject, property type, measured value
*** Having an explicit subject allows reference to the input object, the output object, the comparison of the two, or anything else (e.g., intermediate transformations)
*** Property types _should_ be mentioned in relevant ontology; not sure that subjects can be in general (too many possibilities, too dependent on the structure of the overall migration workflow)
*** Measured values should be simple strings; meaning determined by property type
** *Container format/assessment suggestion:* make it be XML and use something like Schematron to do the assessment.
* Specific format example ([proposed schema|https://github.com/dkfellows/scape-components/blob/master/ComponentMeasures.xsd]):
** {code:xml}<measures>
<measure subject="inputFile" type="compressionRatio">95</measure>
<measure subject="inputFile" type="width">1400</measure>
<measure subject="inputFile" type="height">1800</measure>
<measure subject="outputFile" type="compressionRatio">0</measure>
<measure subject="outputFile" type="width">1400</measure>
<measure subject="outputFile" type="height">1800</measure>
<measure subject="comparison" type="similarity">0.99</measure>
</measures>
{code}
** Note: no caution was taken over the {{subject}} and {{type}} attribute values.
* Specific suggested operations:
** [Workflow Entry: SCAPE Metric Document Builder|http://www.myexperiment.org/workflows/3702.html]
** [Workflow Entry: SCAPE Metric Document Joiner|http://www.myexperiment.org/workflows/3703.html]
** [Workflow Entry: SCAPE Assess Metrics|http://www.myexperiment.org/workflows/3704.html] (note that this workflow is currently _known_ to not work)
* *Consequence:* The characterisation and QA components have to change to (also?) produce a metric document output.
How to get it so that PW can ask the platform to perform a migration?
* *Plan, part A:* It is currently planned (check with [~repomatthias]) to have a standardised migration execution service interface that will allow a migration to be requested (probably by POSTing a description document to a service endpoint). The document will contain references to any subordinate resources required, such as:
** the migration plan to enact,
** the collection of digital objects to apply the migration plan to, and
** the acceptance criteria for judging that the plan successfully applied to a particular digital object.
* *Plan, part B:* It is currently planned to enhance Taverna Server to support that standardised migration execution service interface (in addition to the current service interfaces it has).
** Note that it's not currently decided what the relationship of the requested plan is to the actual workflow to be executed. Does the plan in/referred-to the request include the staging and assessment steps?
h2. PC/PW
How to handle similarity measures?
{color:#000000}A question is if the two ontologies/vocabularies that being developed in SCAPE are compatible or even the same:{color}
{color:#000000}1. The Component measures vocabulary ({color}[http://ifs.tuwien.ac.at/dp/vocabulary/quality/measures|http://ifs.tuwien.ac.at/dp/vocabulary/quality/measures]{color:#000000}){color}
{color:#000000}2. The Policies representation ontology developed in WP13.{color}
{color:#000000}The question is... are both of these ontology structures compatible? Let me put it in another way, are the measures defined by the first ontology compatible with the lower nodes of the second ontology (i.e. the nodes that represent operational measures of the policy representation)?{color}
{toc:maxLevel=2|printable=false}
h1. Task List
{tasklist:SCAPE components TODO list|sortAscending=false|sortBy=priority}
||Completed||Priority||Locked||CreatedDate||CompletedDate||Assignee||Name||
|T|H|F|1386074420265|1386074456386|mferreira|Support annotation extraction and query in myExperiment (MAN)|
|T|H|F|1382444187590|1385029411826|dkf|Implement, test and release extended myExperiment API -- working on component plugin, myE API to go live this week|
|F|H|F|1386074415709| |mferreira|Generate Components for existings tools and publish to MyExperiment (PC developers)|
|F|H|F|1385031044015| |duretec|Finalize component profile specification|
|F|M|F|1385029988018| |dkf|Validate workflow according to profile|
|T|M|F|1382444244284|1390914168462|pmay|Review component end-2-end workflow whitepaper comments and update|
|T|M|F|1385030153502|1390914198825|pmay|Report status of support for complex annotations to Markus P|
|F|M|F|1386074590967| |mferreira|Extract and validate dependencies on the Platform (AIT)|
|F|M|F|1382444161234| |pmay|Enhance toolwrapper (KEEPS, started)|
|F|M|F|1386074793810| |mferreira|Component Validation on Taverna (MAN)|
|T|M|F|1382444210218|1382444218307|mferreira|Clarification from Per about external post-processing step for command line tool components -- Miguel talked to Per and point was taken. Post-processing will happen within the component.|
|T|M|F|1382444230959|1386074886525|pmay|Check status of tasks in Preservation Components, Workflows, Planning and Platform integration -- outdated page|
|F|M|F|1382447500453| |all|[All] Update "developments" as per Toolwrapper example|
{tasklist}
h1. {color:#000000}*"Things" under development{*}{color}
h2. Components specs
h3.
Examples of SCAPE components: [https://github.com/openplanets/scape-component-profiles/tree/master/examples]
h2. Component profiles spec
h3.
The component profiles are available in the [component profile repository|https://github.com/openplanets/scape-component-profiles/tree/master/profiles].
The used ontology is also developed in this repository and published on [http://purl.org/DP/components] alongside with documentation.
h2. MyExperiment API
h3.
(supports creating, searching and managing components)
This is anticipated to go live _this week_ (by 25/10/13). ([Documentation link.|http://wiki.myexperiment.org/index.php/Developer:Components])
h2. Component plugin for taverna
h3.
Currently testing version 1.1.0 (will release once it works\!) ([Browse source.|https://code.google.com/p/taverna/source/browse/#svn%2Funsorted%2Fcomponent%2Fbranches%2Fcomponent-1.1.0])
h2. Toolwrapper - Wrapped tools from PC
| *Module name* | Toolwrapper |
| *Description* | The toolwrapper is a tool that, based on a small specification file, is able to produce the following outputs:
* Bash scripts that standardizes the invocation of Preservation tools
* Debian packages for easy installation \\
\\
The toolwrapper will be enhanced to also support the following features: \\
\\
* Generate SCAPE components according to a Component Profile (Characterization, QA, and Actions)
* Publish generated components to MyExperiment |
| *WP* | PC.WP.2 - Action Services |
| *Responsible* | Miguel Ferreira (WP lead) |
h4.
h3. Tasks
|| Task || Depends of || Status || Responsible || Due date ||
| Development of new version of the toolwrapper | {color:#000000}Component examples{color} | {color:#339966}{*}DONE (see github)*{color} | Hélder Silva (KEEPS) | December |
| Development of new tool that publishes SCAPE components to MyExperiment | MyExperiment API | {color:#339966}{*}UNDER DEVELOPMENT{*}{color} \\ | Hélder Silva (KEEPS) \\ | December \\ |
h3. References
|| Useful information || Description || URL ||
| SCAPE Component examples | Examples of how components should look like. These will be used as templates \\
by the new version of the ToolWrapper so that Components are produced automatically \\
by this tool | +[+Github: SCAPE Component Profiles examples+|https://github.com/openplanets/scape-component-profiles/tree/master/examples]+ \\ |
| SCAPE Component profiles | Profiles that validate that a SCAPE component is according to the right specification. \\
There are 3 types of profiles. | +[+Github: SCAPE Component Profiles+|https://github.com/openplanets/scape-component-profiles/tree/master/profiles]+ \\ |
| MyExperiment API documentation | Documentation of the API provided by MyExperiment to support the publication of \\
Components by the new tool to be developed | +[+MyExperiment: Components API+|http://wiki.myexperiment.org/index.php/Developer:Components]+ \\ |
h3. Issues
|| Issue description || Reported by || Caused by || Solution owner ||
| | | | |
| | | | |
| | | | |
h2. Execution module to run components on the platform
h3.
We need feedback from Rainer.
Main question: Will we be able to submit a SCAPE component to the Job execution service and have it run on the platform.
h2. Integration of Plato and Components
| *Module name* | Plato |
| *Description* | Preservation planning currently does
* Query myExperiment for migration components (based on samples, user input)
* Configure migration components based on parameter ports
* Run selected migration components
* Query CC/QA components defined on decision tree and run them \\
\\
Enhancements \\
\\
* Allow better selection of CC/QA components
* Create an executable plan template containing input/output ports and migration components
* Allow configuration of CC/QA components
* Add CC/QA components to template |
| *WP* | PW.WP.3 Automated Planning |
| *Responsible* | Kresimir Duretec (WP lead) |
h3. Tasks
|| Task || Depends on || Status || Responsible || Due date ||
| Executable Plan Template | {color:#000000}MyExperiment API, Component Profiles{color} | {color:#ff6600}{*}PENDING{*}{color} | Markus Plangg (TUW) | December |
h3. References
|| Useful information || Description || URL ||
| | | |
h3. Issues
|| Issue description || Reported by || Caused by || Solution owner ||
| | | | |
| | | | |
| | | | |
h1. Integration issues
h2. PC Requirements
On a previous PC meeting held in Paris the following whitepaper has been drafted according to PC members perspective. [^SCAPE_Preservation Compoments Profiles_KEEPS_V0.1.docx]
The document has been shared with PW at the time.
h2. PC/PT
How to handle non-1-to-1 migrations? _(Current_ _[migration component profile|https://github.com/openplanets/scape-component-profiles/blob/master/profiles/MigrationActionComponentProfile.xml]_ _handles this correctly; output files are presented component outputs, and both inputs and outputs can be singletons (=filename) or lists of singletons, so 1-many, many-1 and many-many transformations can be represented.)_
h2. PT/PW
How to express measures so that they can be assessed ("do we write this object back to the repository?")
* *Proposal:* Produce a common format to contain the measures. This common container format can then be combined (using some sort of standardised combiner component) so that a single document is presented to an assessment component; the boolean output of that assessment component is then used to guide what happens next (i.e., “write back to repository” or “complain to user about a failed migration”.)
** Needs basic worker components for container document construction, container document combination, container document assessment
** Container format needs to be logically a collection of triples: subject, property type, measured value
*** Having an explicit subject allows reference to the input object, the output object, the comparison of the two, or anything else (e.g., intermediate transformations)
*** Property types _should_ be mentioned in relevant ontology; not sure that subjects can be in general (too many possibilities, too dependent on the structure of the overall migration workflow)
*** Measured values should be simple strings; meaning determined by property type
** *Container format/assessment suggestion:* make it be XML and use something like Schematron to do the assessment.
* Specific format example ([proposed schema|https://github.com/dkfellows/scape-components/blob/master/ComponentMeasures.xsd]):
** {code:xml}<measures>
<measure subject="inputFile" type="compressionRatio">95</measure>
<measure subject="inputFile" type="width">1400</measure>
<measure subject="inputFile" type="height">1800</measure>
<measure subject="outputFile" type="compressionRatio">0</measure>
<measure subject="outputFile" type="width">1400</measure>
<measure subject="outputFile" type="height">1800</measure>
<measure subject="comparison" type="similarity">0.99</measure>
</measures>
{code}
** Note: no caution was taken over the {{subject}} and {{type}} attribute values.
* Specific suggested operations:
** [Workflow Entry: SCAPE Metric Document Builder|http://www.myexperiment.org/workflows/3702.html]
** [Workflow Entry: SCAPE Metric Document Joiner|http://www.myexperiment.org/workflows/3703.html]
** [Workflow Entry: SCAPE Assess Metrics|http://www.myexperiment.org/workflows/3704.html] (note that this workflow is currently _known_ to not work)
* *Consequence:* The characterisation and QA components have to change to (also?) produce a metric document output.
How to get it so that PW can ask the platform to perform a migration?
* *Plan, part A:* It is currently planned (check with [~repomatthias]) to have a standardised migration execution service interface that will allow a migration to be requested (probably by POSTing a description document to a service endpoint). The document will contain references to any subordinate resources required, such as:
** the migration plan to enact,
** the collection of digital objects to apply the migration plan to, and
** the acceptance criteria for judging that the plan successfully applied to a particular digital object.
* *Plan, part B:* It is currently planned to enhance Taverna Server to support that standardised migration execution service interface (in addition to the current service interfaces it has).
** Note that it's not currently decided what the relationship of the requested plan is to the actual workflow to be executed. Does the plan in/referred-to the request include the staging and assessment steps?
h2. PC/PW
How to handle similarity measures?
{color:#000000}A question is if the two ontologies/vocabularies that being developed in SCAPE are compatible or even the same:{color}
{color:#000000}1. The Component measures vocabulary ({color}[http://ifs.tuwien.ac.at/dp/vocabulary/quality/measures|http://ifs.tuwien.ac.at/dp/vocabulary/quality/measures]{color:#000000}){color}
{color:#000000}2. The Policies representation ontology developed in WP13.{color}
{color:#000000}The question is... are both of these ontology structures compatible? Let me put it in another way, are the measures defined by the first ontology compatible with the lower nodes of the second ontology (i.e. the nodes that represent operational measures of the policy representation)?{color}