Creating a Debian/Ubuntu mirror

reprepro

This guide is written for reprepro 4.2.0+. Reprepro is a nice tool to create Debian/Ubuntu mirrors and your own repositories.

As soon as you understand how the tool works it’s pretty easy to use.

Installation

Install reprepro on Debian Squeeze

sudo apt-get install reprepro

GPG Configuration

All packages repositories should be signed with your GPG key. To make this as painless as possible use gpg-agent.

Install the GNU GPG agent

sudo apt-get install gnupg-agent

List available GPG keys

gpg --list-keys

Generate a new GPG key if there is no key availble

gpg --gen-key

Add the following to ~/.profile so gpg-agent will be invoked automatically when it is not running

vi ~/.profile
if test -f $HOME/.gpg-agent-info && kill -0 `cut -d: -f 2 $HOME/.gpg-agent-info` 2> /dev/null; then
	GPG_AGENT_INFO=`cat $HOME/.gpg-agent-info`
	export GPG_AGENT_INFO
else
	eval `gpg-agent --daemon --write-env-file ~/.gpg-agent-info`
fi

if [ -f "${HOME}/.gpg-agent-info" ]; then
	. "${HOME}/.gpg-agent-info"
	export GPG_AGENT_INFO
	export SSH_AUTH_SOCK
	export SSH_AGENT_PID
fi

Add the following line to your .bash_profile

vi ~/.bash_profile
export GPG_TTY=`tty`

Export the gpg key

gpg --list-keys
gpg --export -a 6A9E1B52 > key.pub

Add the gpg key to the apt keyring

sudo apt-key add key.pub

Gather various GPG keys

Get the GPG key for the Debian repository

cd /tmp
wget http://ftp.us.debian.org/debian/dists/squeeze/Release
wget http://ftp.us.debian.org/debian/dists/squeeze/Release.gpg
gpg Release.gpg # enter: 'Release' as name of data file
gpg --keyserver subkeys.pgp.net --search-keys "55BE302B" # Enter '1'
rm Release.gpg Release

Get the GPG key for the Ubuntu repository

cd /tmp
wget http://mirrors.kernel.org/ubuntu/dists/natty/Release.gpg
wget http://mirrors.kernel.org/ubuntu/dists/natty/Release
gpg Release.gpg # enter: 'Release' as name of data file
gpg --keyserver subkeys.pgp.net --search-keys 437D05B5 # Enter '1'
gpg --keyserver subkeys.pgp.net --search-keys 55BE302B # Enter '1'
rm Release.gpg Release

Run the following command to get the last 16 hex digits of the fingerprint

gpg --with-colons --list-key
pub:-:4096:1:9AA38DCD55BE302B:2009-01-27:2012-12-31::-:Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>::scSC:
pub:-:4096:1:9AA38DCD55BE302B:2009-01-27:2012-12-31::-:Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>::scSC:

In this case 9AA38DCD55BE302B is the value to use for the reprepro VerifyRelease option in the conf/updates file(s).

Import the key to the GPG keyring and add it to the apt keyring

gpg --keyserver subkeys.pgp.net --recv AED4B06F473041FA
gpg --export --armor AED4B06F473041FA | apt-key add -

Partial mirrors

It is possible to create partially mirrors with reprepro.

The trick is the FilterFormula parameter in the conf/updates file.

Example

FilterFormula: Priority (==required)

Create a Debian Squeeze mirror

Create the mirror directory including a conf directory, all mirror data will be stored here, be sure there is enough disk space available since mirrors take quite some disk space.

mkdir -p /opt/repositories/debian/mirror/conf

Create the “conf/distributions” configuration file

vi /opt/repositories/debian/mirror/conf/distributions
Origin: Debian
Codename: squeeze
Description: Official Debian Squeeze mirror
Architectures: i386 amd64 source
Components: main contrib non-free
UDebComponents: main
Contents: .gz
Update: - debian-squeeze
Log: /opt/repositories/debian/mirror/logs/mirror.log
SignWith: <your public GPG key>

Origin: Debian
Codename: squeeze-updates
Description: Official Debian Squeeze mirror
Architectures: i386 amd64 source
Components: main contrib non-free
UDebComponents: main
Contents: .gz
Update: - debian-squeeze-updates
Log: /opt/repositories/debian/mirror/logs/mirror.log
SignWith: <your public GPG key>

Origin: Debian
Codename: squeeze-security
Description: Official Debian Squeeze mirror
Architectures: i386 amd64 source
Components: main contrib non-free
Contents: .gz
Update: - debian-squeeze-security
Log: /opt/repositories/debian/mirror/logs/mirror.log
SignWith: <your public GPG key>

Create the “conf/updates” configuration file

vi /opt/repositories/debian/mirror/conf/updates
Name: debian-squeeze
Method: ftp://ftp.nl.debian.org/debian
Components: main contrib non-free
UDebComponents: main
Architectures: amd64 source
VerifyRelease: AED4B06F473041FA

Name: debian-squeeze-updates
Method: ftp://ftp.nl.debian.org/debian
Suite: squeeze-updates
Components: main contrib non-free
UDebComponents: main
Architectures: amd64 source
VerifyRelease: AED4B06F473041FA

Name: debian-squeeze-security
Method: ftp://ftp.nl.debian.org/debian-security
Suite: squeeze/updates
Components: main contrib non-free
Architectures: amd64 source
VerifyRelease: AED4B06F473041FA

Sync/Update the mirror

cd /opt/repositories/debian/mirror
reprepro -V update

Create a Debian Squeeze proposed updates mirror

Create the mirror directory including a conf directory, all mirror data will be stored here, be sure there is enough disk space available since mirrors take quite some disk space.

mkdir -p /opt/repositories/debian/proposed-updates/conf

Create the “conf/distributions” configuration file

vi /opt/repositories/debian/proposed-updates/conf/distributions
Origin: Debian
Codename: squeeze
Description: Debian Squeeze proposed updates mirror
Architectures: amd64 source
Components: main contrib non-free
UDebComponents: main
Update: - debian-squeeze-proposed-updates
Contents: .gz
Log: /opt/repositories/debian/proposed/logs/proposed-updates.log
SignWith: <your public GPG key>

Create the “conf/updates” configuration file

vi /opt/repositories/debian/proposed-updates/conf/updates
Name: debian-squeeze-proposed-updates
Method: ftp://ftp.nl.debian.org/debian
Architectures: amd64 source
Suite: /squeeze-proposed-updates
VerifyRelease: AED4B06F473041FA

Sync/Update the mirror

cd /opt/repositories/debian/proposed-updates
reprepro -V update

Create a Debian Squeeze backports mirror

Create the mirror directory including a conf directory, all mirror data will be stored here, be sure there is enough disk space available since mirrors take quite some disk space.

mkdir -p /opt/repositories/debian/backports/conf

Create the “conf/distributions” configuration file

vi /opt/repositories/debian/backports/conf/distributions
origin: Debian
Codename: squeeze
Description: Official Debian Squeeze backports
Architectures: amd64 source
Components: main contrib non-free
Contents: .gz
Update: - squeeze-backports
Log: /opt/repositories/debian/backports/logs/backports.log
SignWith: <your GPG key here>

Create the “conf/updates” configuration file

vi /opt/repositories/debian/backports/conf/updates
Name: squeeze-backports
Method: ftp://ftp.be.debian.org/debian-backports
Architectures: amd64 source
Suite: squeeze-backports
Components: main contrib non-free
UDebComponents: main
VerifyRelease: AED4B06F473041FA

Sync/Update the mirror

cd /opt/repositories/debian/backports
reprepro -V update

Create a Ubuntu Natty mirror

Create a directory including a conf directory which will contain the mirror(s)

mkdir -p /opt/repositories/ubuntu/mirror/conf

Create the “conf/distributions” configuration file

vi /opt/repositories/ubuntu/mirror/conf/distributions
Origin: Ubuntu
Codename: natty
Description: Ubuntu Maverick mirror
Architectures: amd64 source
Components: main multiverse restricted universe
UDebComponents: main
Contents: .gz
Update: - ubuntu-natty
Log: /opt/repositories/ubuntu/mirror/logs/mirror.log
#SignWith: 2CA546B9

Origin: Ubuntu
Codename: natty-proposed
Description: Ubuntu Maverick proposed
Architectures: amd64 source
Components: main multiverse restricted universe
Contents: .gz
Update: - ubuntu-natty-proposed
Log: /opt/repositories/ubuntu/mirror/logs/mirror.log
#SignWith: 2CA546B9

Origin: Ubuntu
Codename: natty-updates
Description: Ubuntu Maverick updates
Architectures: amd64 source
Components: main multiverse restricted universe
Contents: .gz
Update: - ubuntu-natty-updates
Log: /opt/repositories/ubuntu/mirror/logs/mirror.log
#SignWith: 2CA546B9

Origin: Ubuntu
Codename: natty-security
Description: Ubuntu Maverick security
Architectures: amd64 source
Components: main multiverse restricted universe
Contents: .gz
Update: - ubuntu-natty-security
Log: /opt/repositories/ubuntu/mirror/logs/mirror.log
#SignWith: 2CA546B9

Create the “conf/updates” configuration file

vi /opt/repositories/ubuntu/mirror/conf/updates
Name: ubuntu-natty
Method: http://nl.archive.ubuntu.com/ubuntu
Components: main multiverse restricted universe
UDebComponents: main
Architectures: amd64 source
#VerifyRelease: 9AA38DCD55BE302B

Name: ubuntu-natty-proposed
Method: http://nl.archive.ubuntu.com/ubuntu
Components: main multiverse restricted universe
Suite: /natty-proposed
UDebComponents: main
Architectures: amd64 source
#VerifyRelease: 9AA38DCD55BE302B

Name: ubuntu-natty-security
Method: http://nl.archive.ubuntu.com/ubuntu
Components: main multiverse restricted universe
Suite: /natty-security
UDebComponents: main
Architectures: amd64 source
#VerifyRelease: 9AA38DCD55BE302B

Name: ubuntu-natty-updates
Method: http://nl.archive.ubuntu.com/ubuntu
Components: main multiverse restricted universe
Suite: /natty-updates
UDebComponents: main
Architectures: amd64 source
#VerifyRelease: 9AA38DCD55BE302B

Sync/Update the mirror

cd /opt/repositories/ubuntu/mirror
reprepro -V update

Create a custom repository

Create the directory structure

sudo mkdir -p /opt/repositories/debian/custom/conf

Create the configuration file

sudo vi /opt/repositories/debian/custom/conf/distributions
Origin: custom
Label: Custom Debian Repository
Codename: squeeze
Architectures: i386 amd64 source
Components: main
Description: This repository contains custom Debian packages
SignWith: your public gpgkey # (use gpg --list-keys to get the key)

Create the options file

vi /opt/repositories/debian/custom/conf/options
basedir /opt/repositories/debian/custom

Add a package to the repository

cd /opt/repositories/debian/custom
reprepro includedeb custom ~/seedbank_0.8.0_all.deb

Various reprepro commands

List all available packages for Debian Squeeze in the custom repository

reprepro -b /opt/repositories/debian/custom list squeeze
cd /opt/repositories/debian/custom
reprepro list squeeze

Add a Debian package to the custom repository

reprepro -Vb /opt/repositories/debian/custom includedeb squeeze ~/seedbank_0.8.0_all.deb

Remove the seedbank package from the custom repository

reprepro -Vb /opt/repositories/debian/custom remove squeeze seedbank

nginx

The repository or repositories need to be accessible, one way to do is is via the very fast an lighweight web server Nginx.

Install Nginx

sudo apt-get install nginx

Make sure you have a CNAME configured in DNS which points to the server_name configuration directive.

Create a virtual host

sudo vi /etc/nginx/sites-available/packages
server {

    listen      80;
    server_name packages.seedbank.local;
    autoindex   on;

    access_log /var/log/nginx/packages-access.log;
    error_log  /var/log/nginx/packages-error.log;

    location / {
        root /opt/repositories;
        index index.html;
    }

}

Enable the virtual host

sudo ln -s /etc/nginx/sites-available/packages /etc/nginx/sites-enabled/
sudo /etc/init.d/nginx restart

Other mirror tools

There is a handful of other mirror tools available, some to create full mirrors, some to create proxying mirrors.

If disk space/bandwith is an issue take a look to “apt-cacher”, a really nice easy to setup proxy based mirror. Unfortunately the last time I’ve checked it didn’t like mixing distributions like Debian and Ubuntu together.

More information about some of the available tools