2 The Package Split

For FreeBSD 4.4 over 4.1 gigabytes of packages were created. This causes a problem for CDROM distributions because we would like to ship as many packages as possible without making the user insert another disc to satisfy dependencies. The solution is to create ``clusters'' of like packages with similar dependencies and group these onto specific discs. This section describes the software and methodology used to create those package sets for the official FreeBSD release discs.

The scripts and other files needed to produce a package split can be found in the CVS tree in ports/Tools/scripts/release. Copy this directory to a machine that has enough free disk space to hold 2 to 3 times the size of the package set that you wish to split.

The following scripts are present in this directory:

Here is a checklist of things you will need to check or configure before going any further.

1. Edit config to denote the number of discs you have, their sizes, and whether you want them want to contain packages, distfiles, both, or neither.

2. Make sure you remove the gen directory if there is an old one laying around. This directory contains working files that will only be valid for the current split.

3. On your first pass through a split it is best to fake the copying of packages and distfiles. This will save both time and diskspace while you do a couple of trial runs to make sure things fit, etc. In the oneshot.pl set the fake variable to 1 and instead of actually copying the files it will touch(1) them. Be sure you turn this off or set fake to 0 before you give the resultant discs to the person that will be mastering the discs otherwise they will get a directory full of zero-sized files.

4. Make sure you have a recent copy of the print-cdrom-packages.sh and that it is from the correct release.

5. Check to make sure the XFree86 dependency in scrubindex.pl has the correct version number. You will also need to make sure this value is correct in doit.sh as well.

Next you will need to get a copy of the ports tree, packages, and distfiles from a recent build on the package cluster. See the setup.sh for a working example but essentially here is what needs to be done.

1. Grab a copy of ports.tar.gz and extract it into the ports directory alongside doit.sh and the scripts directory.

2. Remove the packages/distfiles directories or symlinks. Bento has these as symlinks and you will have mixed results if you do not get rid of them before proceeding.

3. Create a new ports/packages directory and copy the package set from the package building cluster.

4. Create a new ports/distfiles directory and copy the distfiles from the package building cluster. NOTE: if you do not want any distfiles simply create the directory and leave it empty. This directory must be present even if it does not contain anything.

Now we are finally ready for the fun task of actually splitting the packages. You start the processing by running ./doit.sh. Here is what it does the first time you run it.

1. Create a list of the restricted (can not be on the master FTP site) ports.

2. Asks you if you would like to remove the restricted ports. Most of the time you will want to answer (y)es here.

3. Create a list of the packages/distfiles that can not be put on the discs.

4. Asks you if you would like to remove the non-cdromable packages/distfiles. Most of the time you will want to answer (y)es here.

5. Copies the INDEX from the ports directory to the gen directory. In doing so it removes the lines for ports where the packages do not exist. It also checks to make sure that all of the required dependency packages are present.

6. Create a list of packages that are required on each disc.

7. Asks you if you would like to populate the discs. After populating each disc it will check for missing dependencies, scrub the INDEX file, and create the CHECKSUM.MD5 file.

8. Check to make sure the required packages made it on each disc and gives you a summary of the sizes of each disc.

After going through this the first time if you are lucky enough that all of the required packages built and fit on each disc. All you need to do is set fake to 0 in oneshot.pl and re-run ./doit.sh. The second and subsequent times around it will skip steps 1-5 above. If you want to re-run any of those steps refer to doit.sh for which files need to be removed to not short-circuit those steps. If you want to repeat all of these steps then the easiest way is to rm -rf gen.

Upon successful completion the packages/distfiles will be in the disc* directories and the leftover will be in the scratch directory.

What to do if things go wrong? Here is some common gotchas and workarounds.

If you get completely stuck and can not figure out why things are borked or how to fix them then email Steve Price <steve@FreeBSD.org> for assistance.