Introduction

The aims of of the SCAPE functional review process are to:

Define software development best practise for the SCAPE project.
Promote defined best practises to SCAPE developers.
Ensure that projects and developers follow defined best practises.
Improve code quality, using both automated tools and peer review.
Test that SCAPE software meets the functional requirements it was intended to address.

This page presents an overview and of the process, and a schedule with accompanying goals.

Overview

What is a Functional Review?

Overall, the aim is to audit all of the software produced by the SCAPE project, with emphasis on examining the following criteria:

Development Process
- Is the code in a publicly available repository (i.e. Open Planets GitHub site)? (MUST)
- Is there a continuous build and automated unit testing process in place? (SHOULD)
- Does the project have an active issue tracking system, e.g. GitHub issues? (SHOULD)
Code Quality
- Does the project adhere to coding standards? (MUST)
- Do unit tests exist? What is the unit test coverage, and is it appropriate? (MUST)
- Does the project build and run? (MUST)
Documentation
- Does a basic functional specification exist? (MUST)
- Does a README file exist? (MUST)
- Is there adequate software deliverable documentation, covering building, installation, administration and usage? (MUST)
- Are the development team members documented? (MUST)
- Are dependencies/licenses clearly identifiable? (MUST)
Functional Evaluation
- Does the software satisfy its functional specification?
- How has this been established?
Installation and Deployment
- Is a downloadable binary package available?
- Is there an official SCAPE supported installation (e.g. on a central instance)?
- Have Debian Packages been created of the software?

Functional Review Criteria

The following section outlines the criteria used for functional reviews. In the first instance, this will be an initial set of criteria; it is fully expected that, with input from other developers, these criteria will evolve and expand to suit the project’s needs in future functional reviews. Not all criteria have to be met, and the maturity of the component will be taken into account, for example, new projects will not be expected to have produced installation documentation.

Development Process

Code Availability in Publicly Available Repositories

All project source code MUST be stored in the publicly available Open Planets Foundation's GitHub repository to provide a central repository of code and to ensure its sustainability beyond the project. See the GitHub Project Structure page for details about the appropriate place to store code in the repositories.

Continuous Build/Integration and Automated Unit Testing

Each project SHOULD have a continuous build process, i.e. all new check-ins should trigger a compilation and execution of unit tests.

The SCAPE and Preservation Watch projects currently use the OPF-Labs Bamboo server for continuous build and reporting. This has suffered from a number of problems surrounding integration with GitHub, often resulting in periodic builds failing. OPF-Labs is not tied in to using Bamboo and the suggestion is to move to using Jenkins (or another CI service). Until an appropriate replacement plan has been defined and agreed, SCAPE will continue to use the OPF Bamboo server.

Issue Tracking

Each project SHOULD have a means to track bugs and other development issues.

If a project does have such a process in place, it is RECOMMENDED that it use the GitHub issues tracking facility associated with the project’s GitHub repository. Utilising this mechanism will ensure the long-term association between the code and the issues log.

Code Quality

Coding Standards

Specific coding standards are described for each language used. As described in the Technical Implementation Guidelines, Java SHOULD be used for new developments, and is the preferred platform language.

Documentation

Functional Specification

Every project MUST have a basic functional specification describing the functional intention of the code. This MAY be described in a README file for young projects.

README/Getting Started

Every project MUST have a README file describing how to get started with, or use, the software. This MAY include a functional specification.

Software Deliverable Documentation

Official project software deliverables MUST provide documentation covering building, installation, administration and usage of the software.

SCAPE Deployment

If there is an official SCAPE supported installation, e.g. the SCAPE central instance, or a web server provided by one of the partners, then it MUST clearly be documented where it is hosted, who can use it, and what are the support / sustainability arrangements?

Documented Development Team Members

A primary developer contact (at least name and email) MUST be listed in the relevant Maven POM <developers> element. This developer will be the primary point of contact for the software.

Additional developers SHOULD also be listed in the <developers> element of the POM.

Dependencies/Licences

Dependencies and Licensing information MUST be clearly documented and published as described in Technical Implementation Guidelines.

Installation/Deployment

Downloadable Binary Package

All released components MUST provide a downloadable binary package, for example a GitHub binary download that works on at least one OS.

Debian Package/Windows Installer

SCAPE software SHOULD be packaged as Debian packages. This is important for sustainability and for encouraging SCAPE software to be adopted by the wider DP community.

SCAPE Debian packaging instructions are available - Debian Packaging

Schedule

The functional review schedule is presented as a set of SCAPE project milestones, taken from the SCAPE description of work. Each milestone is accompanied by a date and a set of goals which should be met for the milestone.

MS17 - Initial Review

The initial review will be lighter than subsequent ones reflecting the effects of the backlog of effort caused by the changes of Technical Coordinator, the fact that many of the software products are at an early stage of development, and the need to establish the review process itself.

Of importance to the review process will be the involvement of other project members, particularly those from the TCC group, but ultimately anyone responsible for developing project software. This review will serve as a starting point for encouraging that involvement, providing an initial set of guidelines and infrastructure which can be developed as appropriate by project members.

The next Functional Review (MS18) is scheduled for January 2013 (M24), with the intervening time seen as a target for engaging other developers in the process. It is hoped that they will be able to shape and direct the process and guidelines to suit, in particular for developers working with programming languages other than Java.

A selection of projects will be reviewed in January, based on the criteria identified in these wiki pages, with the 2nd Year Review Ramp-Up/Developers workshop providing an opportune time to discuss the process with developers and to review software.

First Review Goals

The primary goal of this first functional review is to establish the initial Functional Review process and criteria, in particular to identify:

A catalogue of SCAPE software components;
Initial automated build and deployment practices;
Initial coding standards and guidelines;
What Continuous Integration platform should be used (remain with Bamboo or move to Jenkins/other?);
How to restructure the SCAPE Github repository.

The above goals form the initial foundations upon which subsequent Functional Reviews will be based. The aim is to put in place initial principles, guidelines and infrastructure to help deliver fully functional and maintainable software. Future reviews will evolve these processes and criteria as appropriate.

MS18 - Functional Review Milestone

The second functional review builds upon the groundwork established by the first review by promoting, encouraging and helping the adoption of the coding guidelines. Specifically:

Promoting the functional review process at the developers and all-staff events
Identifying volunteers to form an initial peer-review team
Restructuring of the SCAPE GitHub repository
Actively encouraging and helping developers to adhere to the guidelines
1. getting code moved to the OPF/SCAPE GitHub account
2. ensuring code builds successfully
3. ensuring that developer's create readme's and specify licenses
Reviewing and encouraging developers to add their project to the SCAPE software components catalogue
Creation of an OPF Jenkins Continuous Integration platform for larger, more mature projects (moving away from Bamboo) and the use of Travis-CI for smaller projects
Identifying targets for the following Milestone MS19, see below.

MS19 - Functional Review Milestone

The following task lists constitute the MS19 goals and the progress made towards them captured on the milestone close.

Many of the tasks are unfinished, although that is because many are ongoing until the project end. Status and where possible a metric for all tasks was added when MS19 was closed. All unfinished tasks were copied into MS20 so that progress can be tracked through to the next milestone.

Continuous Integration for ALL SCAPE software projects

Use of OPF GitHub site
- [~] All SCAPE software projects on openplanets GitHub site.
  NOT Marking this as complete as nine the projects are forks, ongoing maintenance required.
- [x] Fork any SCAPE GitHub projects still using their own repos.
  Complete to my knowledge now that MarcAlizer and browser-snapshots tools forked, ongoing maintenance required.
- [~] Apache 2.0 License file for all software projects on GitHub site.
  80% of the SCAPE software projects with LICENSE, though some unlicensed projects may be dormant / dead, ongoing maintenance required.
- [x] CC License for all other projects on GitHub site.
  I believe all non software projects have had a CC licensed added, ongoing maintenance required.
- [x] Populated OPF YAML Metadata file for all projects.
  Currently up to date, ongoing maintenance required
- [~] README with software description and dev level install instructions for software projects.
  Not complete for all projects but all major deliverables now have a satisfactory initial README. New README goal for MS20.
Use of Travis CI service
- [x] Pair work for Travis admin with FR volunteers.
  Ongoing task.
- [~] All SCAPE software projects registered with Travis-CI.
  This isn't complete but nearly all (80%+) of the main deliverables have working Travis builds.
- [x] Identify any non-building projects and contact project owner.
  Done.
- [~] All SCAPE software projects passing Travis CI build.
  Most projects have software builds but not all are passing consistently. The changing dependencies on the upcoming Fedora 4 release have been fluid and not all of the Fedora jars are widely available.

Scheduled Code Quality Builds for Software Deliverables

Set up Jenkins CI Server
OPF Jenkins server available at http://jenkins.opf-labs.org.
- [x] Register main SCAPE software deliverable project builds with the Jenkins server.
  Done although new projects, ongoing maintenance required.
- [x] Install Sonar code quality software with Jenkins integration.
  OPF Sonar instance available at http://sonar.opf-labs.org.
- [~] Add Sonar QA stage to all scheduled Jenkins builds.
  Ongoing although the majority (18) of SCAPE projects now have Sonar QA builds scheduled.
- [x] Pair work with initial FR volunteers for Jenkins and Sonar administration.
  Ongoing.
Continuous QA & Delivery for Java Projects
- [~] Add Maven site builds for all Jenkins Java projects, published under projects.opf-labs.org/mvn/project
  Ongoing, getting the correct package names and Maven Group IDs for Java projects has taken priority.
- [~] Java projects subject to automated code QA
  Ongoing but the majority (75%+) of the Java projects are subject to nightly Sonar QA builds.
- [x] Trial use of Sonar to spot missing JavaDoc for public methods and unit test coverage.
  Completed, public API documentation and Unit Test coverage available here http://sonar.opf-labs.org/dashboard/?did=6 .
- [x] Identify minimum requirements for publication to Maven Snapshot repo (e.g. correct Maven ids and namespace).
  Completed, and minimal requirements in place for all Java projects.
- [~] Jenkins built Java projects that pass QA published to Maven Snapshot repo.
  Ongoing, sorting out the package names and group Ids took longer than anticipated.

MS20 - Functional Review Milestone

The task list and goals for MS20 are still been created but will be available for 20/12/2013/

The following lists a set of goals that together constitute MS20.

- [~] README with software description and dev level install instructions for software projects.
  Not complete for all projects but all major deliverables now have a satisfactory initial README. New README goal for MS20.
- [~] All SCAPE software projects passing Travis CI build.
  Most projects have software builds but not all are passing consistently. The changing dependencies on the upcoming Fedora 4 release have been fluid and not all of the Fedora jars are widely available.
- [x] Pair work with initial FR volunteers for Jenkins and Sonar administration.
  Ongoing.
Continuous QA & Delivery for Java Projects
- [~] Add Maven site builds for all Jenkins Java projects, published under projects.opf-labs.org/mvn/project
  Ongoing, getting the correct package names and Maven Group IDs for Java projects has taken priority.
- [~] Java projects subject to automated code QA
  Ongoing but the majority (75%+) of the Java projects are subject to nightly Sonar QA builds.
- [x] Trial use of Sonar to spot missing JavaDoc for public methods and unit test coverage.
  Completed, public API documentation and Unit Test coverage available here http://sonar.opf-labs.org/dashboard/?did=6 .
- [x] Identify minimum requirements for publication to Maven Snapshot repo (e.g. correct Maven ids and namespace).
  Completed, and minimal requirements in place for all Java projects.
- [~] Jenkins built Java projects that pass QA published to Maven Snapshot repo.
  Ongoing, sorting out the package names and group Ids took longer than anticipated.

Ongoing Maintenance

The following tasks were brought up to date for MS19 but now require ongoing checking to make sure new projects comply with project guidelines.

All SCAPE software projects on openplanets GitHub site.
Fork any SCAPE GitHub projects still using their own repos.
Populated OPF YAML Metadata file for all projects.
Apache 2.0 License file for all software projects on GitHub site.
CC License for all other projects on GitHub site.
All SCAPE software projects registered with Travis-CI.
Register main SCAPE software deliverable project builds with the Jenkins server.
Add Sonar QA stage to all scheduled Jenkins builds.

Subsequent Reviews

Subsequent Functional Reviews will need to consider:

Re-evaluating coding standards and guidelines
Re-evaluating the review criteria
Kick off a Peer review system for performing functional reviews
Assessing documentation and software in accordance with the defined criteria

The SCAPE Functional Review Process