Making FreeBSD Ports

BSDCan 2013 — Ottawa, Canada

Gábor Páli <pgj@FreeBSD.org>

Introduction

What is a FreeBSD Port?


A bunch of files describing the process of seamless configuration, building, installation, deinstallation and packaging for a given piece of third-party software on a FreeBSD system. It also helps to ensure that all the other software may be required in build time or run time is present on the host.


This approach heavily relies on use of make(1) macros, and builds the software from its source code (well, most of the time).

Why to Bother with FreeBSD Ports?


Some of the possible reasons why one may want to learn more about ports:


  • Understand better how ports work in general.
  • Base a (optionally proprietary) in-house software or product on FreeBSD.
  • The favorite port is not maintained well enough.
  • Give something back to the FreeBSD Project!


(And arbitrary combination any of the above.)

How to Start?


All we need is...


  • A personal "itch" to scratch.
  • Spare time, determination.
  • PMake, aka. make(1) — included in base (soon to be replaced with BSD Make though).
  • A recent ports source, somewhere under /usr/ports/ — has to be installed or checked out via SVN:

    # svn checkout http://svn.freebsd.org/ports/head /usr/ports
  • Knowledge of sh(1), C or any affected language or framework.

Anatomy of a Port

Home of Ports


Ports Collection or Ports Tree: a directory tree with categories and with ports in the categories.


${PORTSDIR}/${CATEGORY}/${PORT}


where

  • PORTSDIR=/usr/ports (usually)
  • CATEGORY: category of the port, e.g. "devel"
  • PORT: directory of the port, e.g. "git"

Layout of Port Files


files
Makefile
distinfo
pkg-descr
pkg-plist

Some of them may be missing or there may be other additional files, but this is the most typical.

Makefile


# Created by: Dmitry Sivachenko <demon@FreeBSD.org>
# $FreeBSD$

PORTNAME=       cld
PORTVERSION=    0.1
CATEGORIES=     devel
MASTER_SITES=   ${MASTER_SITE_GOOGLE_CODE}
PROJECTHOST=    chromium-compact-language-detector
DISTNAME=       compact-language-detector-${PORTVERSION}

MAINTAINER=     demon@FreeBSD.org
COMMENT=        Chromium compact language detector library

GNU_CONFIGURE=  yes
USE_LDCONFIG=   yes

.include <bsd.port.mk>

distinfo


SHA256 (compact-language-detector-0.1.tar.gz) = b1a3b430c0d39c8a8731d7291c30d896dd0f63c3fd093ec6ceb275bb760d1d5c
SIZE (compact-language-detector-0.1.tar.gz) = 2498722

pkg-descr


A port from the CLD (Compact Language Detector) library embedded in
Google's Chromium browser.  The library detects the language from
provided UTF8 text (plain text or HTML).  It's implemented in C++,
with very basic Python bindings.

WWW: https://code.google.com/p/chromium-compact-language-detector/

pkg-plist


include/cld/base/basictypes.h
include/cld/base/build_config.h
include/cld/base/port.h
include/cld/base/string_util.h
include/cld/compact_lang_det.h
include/cld/encodings/compact_lang_det/letterscript_enum.h
include/cld/encodings/compact_lang_det/win/cld_basictypes.h
include/cld/encodings/compact_lang_det/win/cld_utf8statetable.h
include/cld/encodings/proto/encodings.pb.h
include/cld/encodings/public/encodings.h
include/cld/ext_lang_enc.h
include/cld/lang_enc.h
include/cld/languages/proto/languages.pb.h
include/cld/languages/public/languages.h
lib/libcld.la
lib/libcld.so
lib/libcld.so.0
libdata/pkgconfig/cld.pc
@dirrm include/cld/languages/public
@dirrm include/cld/languages/proto
@dirrm include/cld/languages
@dirrm include/cld/encodings/public
@dirrm include/cld/encodings/proto
@dirrm include/cld/encodings/compact_lang_det/win
@dirrm include/cld/encodings/compact_lang_det
@dirrm include/cld/encodings
@dirrm include/cld/base
@dirrm include/cld

files/patch-Makefile.in


--- Makefile.in.orig    2012-05-23 14:35:00.000000000 +0400
+++ Makefile.in         2013-02-08 15:37:27.000000000 +0400
@@ -282,7 +282,7 @@
 
 # autogen.sh and cleanrepo.sh are script for maintainance use. Not for distribution.
 # dist_noinst_SCRIPTS = autogen.sh
-pkgconfigdir = $(libdir)/pkgconfig
+pkgconfigdir = ${exec_prefix}/libdata/pkgconfig
 pkgconfig_DATA = cld.pc
 basic_test_SOURCES = tests/basic_test.cc
 basic_test_CXXFLAGS = -Wall -fPIC -Isrc/ -I../src -O2 -DCLD_WINDOWS

The Little Language Behind Ports

Ports are Declarative


Ports maintain an implicit default behavior:


fetch (the source tarball)

extract (the source tarball)

patch (the extracted sources)

configure (the patched sources)

build (the configured sources)

install (the built program)

package (the installed program, optional)

deinstall (the installed program, optional)

Phases

Each of the "phases" correspond to the a make(1) target. Actually a group of targets (simplified):


phase-depends

pre-phase

pre-phase-script

do-phase

post-phase

post-phase-script


Targets are like points of entries during the process, executed sequentially.

Variables (Makefile)


Variables are also handled by make(1). They are often used as configuration parameters in the Makefiles.


NO_MTREE=    yes
MAKE_ENV+=   SHELL=${SH} NO_LINT=YES
PREFIX?=     ${LOCALBASE}
ARCH!=       ${UNAME} -p
CFLAGS:=     ${CFLAGS:C/ $//}
$ make -V ARCH

Variables (pkg-plist)


Package listings can only work with substitutions, determined by the value of PLIST_SUB in the associated Makefiles.

bin/cppi
%%NLS%%share/locale/de/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/eo/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/fi/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/fr/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/gl/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/hr/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/it/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/ja/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/pl/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/sr/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/sv/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/uk/LC_MESSAGES/cppi.mo
%%NLS%%share/locale/vi/LC_MESSAGES/cppi.mo
$ make -V PLIST_SUB

Shell Commands (Makefile)


When make(1) does not offer enough control, it is possible to use direct (imperative) sh(1) commands. They are attached to targets or hooks.


makepatch:
        @${MKDIR} ${FILESDIR}
        @(cd ${PATCH_WRKSRC}; \
                for i in `find . -type f -name '*.orig'`; do \
                        ORG=$$i; \
                        NEW=$${i%.orig}; \
                        OUT=${FILESDIR}`${ECHO} $${NEW} | \
                                ${SED} -e 's|/|__|g' \
                                        -e 's|^\.__|/patch-|'`; \
                        ${ECHO} ${DIFF} -ud $${ORG} $${NEW} '>' $${OUT}; \
                        ${DIFF} -ud $${ORG} $${NEW} > $${OUT} || ${TRUE}; \
                done \
        )

Shell Commands (pkg-plist)


Shell commands can be also attached to certain events, i.e. installation and deinstall of the port, in the pkg-plist file.


@exec /bin/ln -sf %D/bin/haddock-ghc-%%GHC_VERSION%% %D/bin/haddock || return true

@unexec /bin/rm -f %D/bin/haddock || return true

Conditional Sections


Makefiles can have conditional sections, controlled by variables or result of Boolean expressions.


.if defined(IA32_BINARY_PORT) && ${ARCH} != "i386"
.if ${ARCH} == "amd64" || ${ARCH} == "ia64"
.if !defined(HAVE_COMPAT_IA32_KERN)
IGNORE=         requires a kernel with compiled-in IA32 compatibility
.elif !defined(HAVE_COMPAT_IA32_LIBS)
IGNORE=         requires 32-bit libraries installed under /usr/lib32
.endif
_LDCONFIG_FLAGS=-32
LIB32DIR=       lib32
.else
IGNORE=         requires i386 (or compatible) platform to run
.endif
.else
LIB32DIR=       lib
.endif

Loops


Loops may be used to walk whitespace-separated values stored in variables.


.for cat in ${CATEGORIES}
.       if empty(VALID_CATEGORIES:M${cat})
                @${ECHO_MSG} "${PKGNAME}: Makefile error: category ${cat} not in list of valid categories."; \
                ${FALSE};
.       endif
.endfor

Use of Other Makefiles


Makefiles may include contents of other Makefiles, often with .mk extension.


.if exists(${MASTERDIR}/Makefile.${ARCH}-${OPSYS})
.include "${MASTERDIR}/Makefile.${ARCH}-${OPSYS}"
USE_SUBMAKE=    yes
.elif exists(${MASTERDIR}/Makefile.${OPSYS})
.include "${MASTERDIR}/Makefile.${OPSYS}"
USE_SUBMAKE=    yes
.elif exists(${MASTERDIR}/Makefile.${ARCH})
.include "${MASTERDIR}/Makefile.${ARCH}"
USE_SUBMAKE=    yes
.endif

.include <bsd.port.mk>

Comments


Makefiles may contain comments to help navigation in the sources.


# DO NOT COMMIT CHANGES TO THIS FILE BY YOURSELF, EVEN IF YOU DID NOT GET
# A RESPONSE FROM THE MAINTAINER(S) WITHIN A REASONABLE TIMEFRAME! ALL
# UNAUTHORISED CHANGES WILL BE UNCONDITIONALLY REVERTED!

Note that pkg-plist uses a different format for comments.


@comment $FreeBSD: head/devel/icmake/pkg-plist 308217 2012-12-04 09:13:30Z pgj $

A Step-By-Step Case Study: Creating a Port

Getting the Sources

PORTNAME=       git
PORTVERSION=    1.8.2
PORTREVISION=   0
PORTEPOCH=      0

CATEGORIES=     devel

MASTER_SITES=   ${MASTER_SITE_GOOGLE_CODE}
PROJECTHOST=    git-core
DISTFILES=      ${PORTNAME}-${PORTVERSION}${EXTRACT_SUFX}

MAINTAINER=     ports@FreeBSD.org
COMMENT=        Distributed source code management tool

.include <bsd.port.mk>
# make fetch
===>   git-1.8.2 depends on file: /usr/local/sbin/pkg - found
=> git-1.8.2.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://git-core.googlecode.com/files/git-1.8.2.tar.gz
git-1.8.2.tar.gz                              100% of 4287 kB  420 kBps 00m00s
===> Fetching all distfiles required by git-1.8.2 for building
# make makesum

Compiling the Sources


$ make
===>  Building for git-1.8.2
"Makefile", line 348: Need an operator
"Makefile", line 388: Need an operator
"Makefile", line 406: Need an operator
"Makefile", line 443: Need an operator
"Makefile", line 603: Need an operator
"Makefile", line 605: Need an operator
"Makefile", line 606: Need an operator
"Makefile", line 607: warning: duplicate script for target "ifndef" ignored
"Makefile", line 608: Need an operator
"Makefile", line 609: Need an operator
"Makefile", line 610: warning: duplicate script for target "ifndef" ignored
"Makefile", line 611: Need an operator
"Makefile", line 613: Need an operator
"Makefile", line 614: Need an operator
...
USE_GMAKE= yes
MAKE_ARGS+= prefix="${PREFIX}"

Adding a Long Description


Add a pkg-descr file (mandatory) with a longer, descriptive description of the ported application.


GIT is a "directory content manager" designed to handle absolutely massive
projects with speed and efficiency, and the release of the 2.6.12 (and later)
versions of the Linux kernel as well as more and more other projects switching
to it would indicate that it does this task well.

GIT falls in the category of distributed source code management tools, similar
to e.g. GNU Arch or Monotone (or, in the commercial world, BitKeeper). Every
GIT working directory is a full-fledged repository with full revision tracking
capabilities, not dependent on network access to a central server.

WWW: http://git-scm.com/

Note the WWW: line — some tools or web sites, e.g. FreshPorts may use it.

Creating the "Packing List"

First, the port should be built and installed with a custom PREFIX:

$ make PREFIX=/var/tmp/git
$ make install PREFIX=/var/tmp/git

Sometimes this can be skipped as the port does not install too many files, e.g. audio/adplay. In that case, just add:

PLIST_FILES=    bin/adplay %%DATADIR%%/adplug.db
PLIST_DIRS=     %%DATADIR%%

Patching the Sources


$ make extract
===>   git-1.8.2 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by git-1.8.2 for building
===>  Extracting for git-1.8.2
=> SHA256 Checksum OK for git-1.8.2.tar.gz.

Find the places where the build or the installing process for the port may have gone wrong — and fix it!


Though, before a modifying a file, create a copy of it with an .orig extension. Once finished, use the makepatch target:


$ make makepatch

Getting pkg-plist


Once everything safely arrived to /var/tmp/git, use the utilities in the Ports Collection to automatically generate a pkg-plist file for it:


$ /usr/ports/Tools/scripts/plist -Md -m `make -V MTREE_FILE` \
    /var/tmp/`make -V PORTNAME` > pkg-plist

So far so good...

The True Challenges

Testing


Testing ports in an isolated (chroot(8), jail(8)) environment is essential to find further potential problems.


Both Ports Tinderbox

# cd /usr/ports/ports-mgmt/tinderbox
# make install

and Poudriere can be used for this.

# cd /usr/ports/ports-mgmt/poudriere
# make install

Working with poudriere


Create a jail for testing.

# poudriere jail -c -j 91amd64 -v 9.1-RELEASE -a amd64 -m ftp

Create a ports tree called local. This will contain the ports to test.

# poudriere ports -c -p local -F

But before we would proceed...

Using a Local Overlay (1)


A fancy option for collecting custom ports is creating an overlay for the ports tree, that could be stored and published separately.


# cd /usr/ports/ports-mgmt/portshaker
# make install

Create the custom ports tree containing only the locally created or modified ones. Add them to the configuration file of portshaker.

ports_trees="main local"
main_ports_tree="/usr/ports"
local_ports_tree="/usr/local/poudriere/ports/local/ports"
local_merge_from="ports local"

Using a Local Overlay (2)


Create a file under /usr/local/etc/portshaker.d that describes the location of the local ports tree:


#!/bin/sh

. /usr/local/share/portshaker/portshaker.subr

method="git"
git_clone_uri="git://github.com/ketchup/freebsd-ports.git"

run_portshaker_command $*

Testing the Port


Use portshaker to get the latest merged ports tree with the modifications.

# portshaker -u ports -u local
# portshaker -v -m local

Use poudriere to do a test build in an isolated environment.

# poudriere testport -j 91amd64 -p local -o devel/git

origin: the containing directory for the port. Multiple packages may share origin.

Testing the Port: Notes


  • Try different architectures and base versions. (Earlier base versions can be usually run in jails, so as 32-bit versions on top of 64-bit).

  • Ports may be broken due to mistakes made in the ports tree. (Be careful!)



Check the logs...

Dependencies (1)


In file included from http.c:1:
http.h:6:23: warning: curl/curl.h: No such file or directory
http.h:7:23: warning: curl/easy.h: No such file or directory
In file included from http.c:1:
http.h:46: error: expected specifier-qualifier-list before 'CURLcode'
http.h:51: error: expected specifier-qualifier-list before 'CURL'
http.h:97: error: 'CURL_ERROR_SIZE' undeclared here (not in a function)
...
...
/usr/bin/perl Makefile.PL PREFIX='/prefix/git-1.8.2' INSTALL_BASE='' --localedir='/prefix/git-1.8.2/share/locale'
gmake[1]: /usr/bin/perl: Command not found
gmake[1]: *** [perl.mak] Error 127
gmake: *** [perl/perl.mak] Error 2
*** [do-build] Error code 1

Obviously applications may rely on other applications for building and running.

Dependencies (2)


MAKE_ENV+= CURLDIR=${LOCALBASE}
BUILD_DEPENDS+= curl:${PORTSDIR}/ftp/curl
RUN_DEPENDS+= curl:${PORTSDIR}/ftp/curl
LIB_DEPENDS+= expat.6:${PORTSDIR}/textproc/expat2

USE_PERL5=      yes
MAKE_ENV+=      PERL_PATH=${PERL}
BUILD_DEPENDS+= p5-Error>=0:${PORTSDIR}/lang/p5-Error
RUN_DEPENDS+=   p5-Error>=0:${PORTSDIR}/lang/p5-Error \
                p5-Net-SMTP-SSL>=0:${PORTSDIR}/mail/p5-Net-SMTP-SSL

USE_PYTHON=     yes
PLIST_SUB+=     PYTHON_VER=${PYTHON_VER} PYTHON=""
CONFIGURE_ARGS+=        --with-python=${LOCALBASE}/bin/python

Configurable Options (1)


Let us make the dependency on curl and expat optional and introduce a user-configurable option for it.


OPTIONS_DEFINE= CURL
OPTION_DEFAULT= # intentionally left blank = not a default option

CURL_DESC= Use curl

.include <bsd.port.options.mk>

.if ${PORT_OPTIONS:MCURL}
BUILD_DEPENDS+= curl:${PORTSDIR}/ftp/curl
RUN_DEPENDS+=   curl:${PORTSDIR}/ftp/curl
LIB_DEPENDS+=   expat.6:${PORTSDIR}/textproc/expat2
PLIST_SUB+=     CURL=""
.else
MAKE_ENV+=      NO_CURL=1 \
                NO_EXPAT=1
PLIST_SUB+=     CURL="@comment "
.endif

Configurable Options (2)


Various combinations of options could be set and tested separately with poudriere.


# poudriere options -c -j 91amd64 -p local devel/git

Running a Daemon: Users and Groups


It easy to tell that the application expects presence of certains users on the system. If they are not present, the ports framework will create it automatically. (But it will not remove...)


USERS=  git_daemon
GROUPS= git_daemon

Running a Daemon: Services


USE_RC_SUBR= git_daemon

files/git_daemon.in

#! /bin/sh
# PROVIDE: git_daemon
# REQUIRE: DAEMON
# KEYWORD: shutdown
  
. /etc/rc.subr
    
name="git_daemon"
rcvar="git_daemon_enable"
    
load_rc_config $name
    
: ${git_daemon_user:=git_daemon}
: ${git_daemon_group:=git_daemon}
: ${git_daemon_enable:=NO}
: ${git_daemon_directory:=%%PREFIX%%/git}
: ${git_daemon_flags:=--syslog --reuseaddr --detach}
    
command="%%PREFIX%%/libexec/git-core/git-daemon"
command_args="${git_daemon_directory}"
    
PATH="${PATH}:%%PREFIX%%/libexec/git-core"
    
run_rc_command "$1"

Manual Pages: Installing Files


Some of the files require us to copy them manually to the destination.


DISTFILES+=     ${PORTNAME}-manpages-${PORTVERSION}${EXTRACT_SUFX}
EXTRACT_ONLY=   ${PORTNAME}-${PORTVERSION}${EXTRACT_SUFX} \
                ${PORTNAME}-manpages-${PORTVERSION}${EXTRACT_SUFX}

post-install:
        (cd ${WRKDIR}/man1/ && ${COPYTREE_SHARE} \* ${MANPREFIX}/man/man1)
        (cd ${WRKDIR}/man5/ && ${COPYTREE_SHARE} \* ${MANPREFIX}/man/man5)
        (cd ${WRKDIR}/man7/ && ${COPYTREE_SHARE} \* ${MANPREFIX}/man/man7)

# make makesum

Manual Pages: Packing

Automated installation and removel of manual pages can be invoked by using MAN1, MAN5, MAN7 variables in the Makefile.

MAN5=           gitattributes.5 \
                githooks.5 \
                gitignore.5 \
                gitmodules.5 \
                gitrepository-layout.5 \
                gitweb.conf.5
MAN7=           gitcli.7 \
                gitcredentials.7 \
                gitglossary.7 \
                gittutorial.7 \
                gitcore-tutorial.7 \
                gitdiffcore.7 \
                gittutorial-2.7 \
                gitworkflows.7 \
                gitrevisions.7 \
                gitnamespaces.7

# MAN1 was left out intentionally due to its size.

Slaves


Multiple packages may be created from a single port, and ports may share code via using slave ports.


PKGNAMESUFFIX=  -curl
COMMENT=        Distributed source code management tool with curl enabled
MASTERDIR=      ${.CURDIR}/../git
DESCR=          ${MASTERDIR}/pkg-descr
WITH_CURL=      yes
CONFLICTS?=     git-[0-9]*
DISTINFO_FILE=  ${MASTERDIR}/distinfo

.include "${MASTERDIR}/Makefile"

Conflicts, Restrictions, Breakage


Use of ports may be restricted based on certain circumstances.


CONFLICTS=      git-curl-[0-9]*
BROKEN=         does not build
IGNORE=         not supported
RESTRICTED=     not allowed to redistribute

Restricted ports must be added to the LEGAL file (in the root of the ports tree).

MOVED


Sometimes ports get obsoleted, or they have to be renamed.


misc/git|misc/gnuit|2009-03-14|Renamed

Note: portshaker supports this.

UPDATING


For changes visible to the consumers, it is usually a good idea to add an entry to the UPDATING (in the root of the Ports Collection).


20121112:
  AFFECTS: Users of devel/git
  AUTHOR: wxs@FreeBSD.org

  The git-daemon(1) process now runs as the git_daemon user. Please make
  sure that this user has appropriate permissions to the repositories.

Note: portshaker supports this.

Packages


For poudriere, this is called a bulk mode, where tests can be omitted for speeding up the process.

# poudriere bulk -j 91amd64 -p local devel/git

Example (may not work properly with Chrome)


The resulting packages (the port with all of its dependencies) can be then put online and used with pkg:

PACKAGESITE: http://example.org/pkgng/freebsd:9:x86:64/
repos:
  default : http://example.org/pkgng/
  repo1 : http://somewhere.org/pkgng/repo1/
  repo2 : http://somewhere.org/pkgng/repo2/

GNATS PRs


Submitting new ports or updates via the web site is easy.

Demonstration

Summary

Things That Have Been Missed...

Questions? Comments?

For Closing...


  • Consult the Porter's Handbook.

  • Do not forget keep eating your own dogfood! Pass maintainership when interest is lost.

  • Subscribe to freebsd-ports@FreeBSD.org.

  • Join EFNet / #bsdports on IRC.

  • Feel free to poke me at the conference... or later in email.


Enjoy BSDCan 2013!