arc-unpack

Version 15 by Asger Askov Blekinge
on Apr 23, 2012 12:44.

<< Changes from 14 to 15

compared with

Current by Asger Askov Blekinge
on Apr 23, 2012 14:12.

Key

This line was removed.

This word was removed. This word was added.

This line was added.

Changes (62)

View Page History

...

First, we need to define a few properties in the pom file

{code:language=xml|title=Pom file properties}<properties> properties}<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.deb.name>${groupId}.${artifactId}</maven.deb.name>
<maven.deb.libfolder>/usr/share/${maven.deb.name}</maven.deb.libfolder>
<maven.deb.docfolder>/usr/share/doc/${maven.deb.name}</maven.deb.docfolder>
<maven.deb.manfolder>/usr/share/man</maven.deb.manfolder>
<maven.deb.binfolder>/usr/bin</maven.deb.binfolder>
<maven.deb.assembly>debian-prepare</maven.deb.assembly>

</properties> <maven.deb.name>${groupId}.${artifactId}</maven.deb.name>

<maven.deb.libfolder>/usr/share/${maven.deb.name}</maven.deb.libfolder>
<maven.deb.docfolder>/usr/share/doc/${maven.deb.name}</maven.deb.docfolder>
<maven.deb.manfolder>/usr/share/man</maven.deb.manfolder>
<maven.deb.binfolder>/usr/bin</maven.deb.binfolder>
<maven.deb.assembly>debian-prepare</maven.deb.assembly>
<maven.deb.description>unpacker application for arc and warc files, based on the heritrix crawler.</maven.deb.description>
<maven.deb.extendedDescription>This is the extended Description</maven.deb.extendedDescription>
<maven.deb.maintainer><![CDATA[Asger Askov Blekinge <[email protected]>]]></maven.deb.maintainer>
<maven.deb.copyright>Copyright (C) 2012 by State and University Library, Denmark</maven.deb.copyright>
<maven.deb.license>Apache-2.0</maven.deb.license>

{code}
Basically, these just allows us to use consistent names in the plugin configs that follow.

...

Then we come to the <build> section. The first thing to do, is to enable filtering on the resources. This is done such
{code:language=xml}<build>

<resource> <directory>src/main/resources</directory>

<filtering>true</filtering>
<directory>src/main/resources</directory>

</resource>

</resources>{code}

The effect of this is that you can use $\{maven.deb.name\}

...

</excludes>
</configuration>

</plugin>{code}

In the manifest, I declare the main class, and the classpath. Notice that I declare the classpathPrefix to be the maven.deb.libfolder, in order for the system to be able to resolve the dependencies when installed as a debian package.

...

</execution>
</executions>

</plugin>

{code}
This plugin uses the deb assembly. So, what will the deb.assembly do? Lets look at it. 

...

</mapper>
</data>

</dataSet>
</configuration>
</execution>
</executions>

</plugin>

{code}
Note that it is important that this plugin is placed below the assembly plugin, as they both execute in the package phase, and jdeb depends on the debian assembly having been run.

...

Make a file "control" in resources/debian/control, with this content

{code:language=none}Package: [[groupId]].[[artifactId]]
Version: [[version]]

{code:language=none}Package: ${groupId}.${artifactId}
Version: ${version}

Section: java
Priority: optional
Architecture: all
Depends: default-jre (>= 1.5)

Maintainer: Asger Askov Blekinge <[email protected]>
Description: unpacker app for arc and warc files, based on the heritrix crawler
This is the extended description

Maintainer: ${maven.deb.maintainer}
Description: ${maven.deb.description}

{code}skfjklsdj ${maven.deb.extendedDescription}

{code}Now we should generate the deb, to check that what we have done so far works. Run mvn package.

Now we should generate the deb, to check that what we have done so far works. Run mvn package.

Then we should check the deb. CD to the target directory, and use the lintian tool (lintian \*.deb). Install it if you do not have it (sudo apt-get install lintian)

...

Files: *
Copyright:

${maven.deb.copyright}

License: Apache-2.0 ${maven.deb.license}

/usr/share/common-licenses/Apache-2.0

/usr/share/common-licenses/${maven.deb.license}

{code}Now the jdeb plugin needs to be told to include this file. Add the following section to the plugin's list of datasets

{code}<data>
<src>${project.build.directory}/classes/debian/copyright</src>

...

<prefix>${maven.deb.docfolder}</prefix>
</mapper>

</data>

{code}mvn clean package to generate the package once more, and lintian it again. The error should now be
{code}E: dk.statsbiblioteket.scape.arc-unpacker: changelog-file-missing-in-native-package{code}
Okay, so we need to make a changelog file for debian. 

Here is some sample content for at changelog file, that we can use 

{code}${maven.deb.name} (0.8.10) (0.3) unstable; urgency=low

* Added the archive.org maven repository, so we can build without local caches.
* Added the debian package functionality

* Minor bugfix releases:

-- Asger Askov Blekinge <[email protected]> Mon, 23 Apr 2011 15:00:00 +0100

o manual page confused --dbname and --dbsystem (Closes: #573235)

${maven.deb.name} (0.2) unstable; urgency=low
* Added the command line interface
* Added the min/max response code restriction
* Added the multiple naming of extracted resources

o applied patch by Warren Thompson to use the registered default
currency rather than a hard-code 'USD'

-- Asger Askov Blekinge <[email protected]> Fri, 2 Dec 2011 17:34:00 +0100

* debian/control: Standards-Version: increased to 3.8.4
* debian/source/format: Added with "3.0 (native)"
* debian/rules: Updated Perl invocation

${maven.deb.name} (0.1) unstable; urgency=low
* Updated maven structure for better git support
* Stuff actually works
* First release

-- Dirk Eddelbuettel <[email protected]> Sun, 13 Jun 2010 17:24:30 -0500

-- Asger Askov Blekinge <[email protected]> Mon, 30 Nov 2011 15:21:00 +0100
{code}Put this content in "resources/debian/changelog/changelog

${maven.deb.name} (0.8.9) unstable; urgency=low

Regenerate the package and validate it once more. The error should now be
{code}E: dk.statsbiblioteket.scape.arc-unpacker: changelog-file-not-compressed changelog
{code}
So, the changelog should be compressed. I have found that the antrun plugin seems to be the simplest way of doing this
{code:language=xml}<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<phase>prepare-package</phase>
<configuration>
<target>
<exec executable="gzip">
<arg value="-9"/>
<arg value="-r"/>
<arg value="${project.build.directory}/classes/debian/changelog/"/>
</exec>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
{code}So, after this change, the deb package should validate. But we have forgotten something. The most important thing, actually. The executable file.

* Minor bugfix releases:

Since this is a java program, we use a shell script to start the program. Create the file "resources/scripts/arc-unpack" with the following content
{code:language=bash}#!/bin/sh
java -jar ${maven.deb.libfolder}/${project.build.finalName}.jar $*
{code}Remember, this file is in resources, so maven will replace the keys when compiling. Now, we need to tell jdeb to include this file. This is done by adding the following definition to the set of datasets
{code:language=xml}<data>
<src>${project.build.directory}/classes/scripts</src>
<type>directory</type>
<mapper>
<type>perm</type>
<prefix>${maven.deb.binfolder}</prefix>
<filemode>755</filemode>
</mapper>
</data>
{code}Regenerate and validate the deb file. You should now receive the following warning:
{code}W: dk.statsbiblioteket.scape.arc-unpacker: binary-without-manpage usr/bin/arc-unpack{code}So, we need to add man pages to the project. 

o DateCalc() now requires an error code variable, so supply one

Create the file "arc-unpack.8" in resources/man/man8, and give it this content:
{code:language=none|title=sample manpage}.TH arc-unpack 8 "April 23, 2012" "version ${project.version}" "USER COMMANDS"
.SH NAME
arc-unpack \- ${maven.deb.description}
.SH SYNOPSIS
.B arc-unpack
\-f dataFile [\-o outputDir] [\-minResp number] [\-maxResp number] [\-naming [MD5,OFFSET,URL]]
.SH DESCRIPTION
${maven.deb.extendedDescription}
.SH OPTIONS
.TP
\-f dataFile
Data file to extract. Can be an arc or warc file, and can be compressed.
.TP
\-o outputDir
extracts the resources to this dir. Defaults to the current dir.
.TP
\-minResp number
Ignore resources, if the http return code is lower than minResp. Useful to filter out returns below the 200 range
.TP
\-maxResp number
Ignore resources, if the http return code is higher than maxResp. Useful to filter out returns above the 200 range
.TP
\-naming [MD5,OFFSET,URL]
Naming scheme for the extracted resources. Each resource in the archive is identified by an URL, but URLs do not map
neatly to filenames.
The URL scheme tries to map the resource urls to files, but can fail.
The OFFSET scheme uses offsets into the arc file as filenames
The MD5 scheme md5hash the resource URLs to ensure valid unique filenames
.SH EXIT STATUS
arc-unpack returns zero if the extraction succeeded
.SH AUTHOR
${maven.deb.maintainer}
{code}
It is important that the file starts with ".TH arc-unpack 8" but the rest seems optional

o finally release the OandA fx code (Closes: #532743)

Now we need to tell jdeb to include this file also. Add this section to the plugins list of datasets
{code:language=none}<data>
<src>${project.build.directory}/classes/debian/man</src>
<type>directory</type>
<mapper>
<type>perm</type>
<prefix>${maven.deb.manfolder}</prefix>
</mapper>
</data>
{code}Clean, regenerate and validate the package.

* debian/control: Standards-Version: increased to 3.8.3

The error received now should be
{code}E: dk.statsbiblioteket.scape.arc-unpacker: manpage-not-compressed usr/share/man/man8/arc-unpack.8{code}

* debian/copyright: Updated to newer format

So, we need to compress this file also. Change the antrun plugin to include the man folder also, like this.

-- Dirk Eddelbuettel <[email protected]> Tue, 22 Dec 2009 20:13:59 -0600

{code}<plugin>

<artifactId>maven-antrun-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<phase>prepare-package</phase>
<configuration>
<target>
<exec executable="gzip">
<arg value="-9"/>
<arg value="-r"/>
<arg value="${project.build.directory}/classes/debian/man/"/>
</exec>
<exec executable="gzip">
<arg value="-9"/>
<arg value="-r"/>
<arg value="${project.build.directory}/classes/debian/changelog/"/>
</exec>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
{code}The package should now be complete and validate. Lets try to install it.

The package should install without a problem. Play around, and try "man arc-unpacker" to see the manpage work