A plan for packaging

Dev
1

Packaging BitKeeper

Packaging BitKeeper has turned out to be a little challenging because of two main reasons:

  1. It bundles a bunch of other programs, some of them modified.

  2. The build system is not standardized (i.e. make && make install doesn’t install things where it’s expected.

This is my attempt to provide a roadmap for making it easier to package BitKeeper for the different distributions.

Unbundling BitKeeper

BitKeeper currently consists of the following modules:

  • diffutils
  • patch
  • Tcl/Tk – Really little-lang
  • The following Tk Packages:
    • Bwidget (Tcl/Tk module)
    • TkCon
    • TkTable
    • TkTreectrl
  • PCRE
  • Tomcrypt
  • Tommath

Diffutils & Patch

There is currently an attempt underway to drop the requirement for diffutils and patch.

PCRE

It seems it’d be relatively easy to drop the bundled PCRE as there don’t seem to be any modifications to them. We could just require libpcre-dev from the package system in order to build BitKeeper (the library is statically linked).

TomMath & TomCrypt

libtommath is only used as a requirement for libtomcrypt so I’ll only discuss libtomcrypt.

libtomcrypt is used in src/crypto.c in order to implement the bk crypto command. This command as far as I can tell, is only used for creating hashes in src/gui/tcltk/version.sh, src/build_upgrade_index.pl, src/update_buildenv, and src/win32/msysver.sh.

All of this infrastructure could be replaced with a requirement on the md5 of sha1 binary that distributions usually provide. On Windows there appears to be certUtil that can handle hashing files.[1]

The only other use for cryptographic utilities is MD5KEYS which are fundamental for BitKeeper.

Tcl/Tk and friends

The most modified package bundled with BitKeeper is Tcl/Tk. In reality, this is not Tcl/Tk but http://little-lang.org(The Little Language).

The Build System

The current build system creates a temporary holding area with all the files that will be installed in $(bk root)/src/tmp/bitkeeper. The directory can be created using the installdir target, like so:

make installdir

That installdir is designed to be a bundle, i.e. the bk executable expects to find binaries, and scripts it needs inside that path.

We should make installdir comply with the standard layout of distributions. That is, binaries in bin, libraries in lib, etc.

A Plan for Packaging

This is my first pass at a plan for packaging:

  1. Package little-lang so that BitKeeper can depend on it.
  2. Remove diffutils and patch dependency and just use the system’s.
  3. Remove PCRE as a bundle and just require libpcre-dev to be installed. (For Windows, require it as part of MSYS, for OSX get it via brew).
  4. Leave libtomcrypt and libtommath as they are.
  5. Maybe unbundle all the Tcl/Tk stuff?

[1] http://superuser.com/questions/245775/is-there-a-built-in-checksum-utility-on-windows-7/898377#898377

2

Good summary.

Remember the Fedora guy:

this is the .spec file he came up with, it includes package versions
http://copr-dist-git.fedorainfracloud.org/cgit/ngompa/BitKeeper/bitkeeper.git/tree/bitkeeper.spec

We also include:

  • lz4
  • zlib

And some people will take exception to the NetBSD stdio library.

On tomcrypt, we have 2 primary uses.

  • md5hash gets used a lot internally. Most notably for md5keys.
  • rand_getBytes() is used for the randbits on 1.0 keys.

Both could be replaced with something simpler, but we don’t need any local modifications so just providing the ability to use the system’s tomcrypt if it exists would be enough.

We should also look into cmake, mainly because we hate autotools and that is the alternative.

3

Also, if Little were updated to Tcl 8.6, does that mean we could use the system’s Tk and the standard packages for the various extensions?

4

I packaged BitKeeper for Void Linux (https://github.com/voidlinux/void-packages/blob/master/srcpkgs/bk/template)

It would be nice if the system versions of diff, patch, PCRE and libtomcrypt coud be used.

Passing CFLAGS currently fails as they are not propagated to gnu/* folders.

We focus on cross-compiling, but I can understand this is hard to add to an adhoc build system.

We also have a variant using Musl libc, this required two small hacks (see the template).

5

Thanks.

  • We have a project internally to remove ‘diff’ and ‘patch’ from the system. So that will remove the gnu/* folders.
  • We don’t have any special changes to PCRE or Tomcrypt/Tommath so will be adding hooks to use the system’s version soon.
  • I will look at incorporating those two changes you needed for Musl
  • TclTK, yeah we those are custom versions and we need them. We are hoping to package Little separately and change bk’s build so it can use the installed version, but it needs to be fixed to not conflict with the system’s TclTk.
1 Like
6

I would like to help. I use the commercial version and Debian at the job. So it will be good to package it into Debian and use the package instead of the bundle. I am going into DebConf 2016 were I will start working on this.

7

Cool. You plan on working on it at the conference? Maybe I need to talk Larry into sending me. :wink:

I think we need to package Little first. It is our Tcl/tk component, but the Tcl part is highly customized and so incompatible with the official tcl/tk packages. And bk builds a C extension for Tk to use for our GUIs so that Little component will need libraries and headers (liblittle-dev?). And these are going to need to be renamed so they don’t conflict with the official tcl packages.

Anyway, we can do a lot of the setup if we get an idea what you will need. How should the build be configured and what features are needed in the makefiles?

Perhaps we should get on the phone/skype/hangouts/etc sometime and talk about it.

8

I have never looked into the code of bitkeeper, so for now I don’t know what I need yet. One thing I may use is debian alioth.debian.org machine to store my work on the package side. Alioth machine is were it is stored most of repositories of code from Debian projects and packaging work. There are many of advantages to store my work on debian packaging in alitoh in a code repository.

Question:
There are tools to convert from a git/svn repository into bitkeeper?

So in the future the work can be self hosting. By converting the existing git/svn repo into bitkeeper.

https://lists.debian.org/debian-devel/2016/05/msg00177.html

But from this thread I have a lot of work to do before it is a proper Debian package, special the security issues with /tmp.

We can share the contacts on Hangouts by private email.

9

I have published a git->bk importer in another thread. It isn’t perfect yet and I can still break it with some crafty repositories, but I hope to have that fixed so.

We also include a fast-export to move stuff from bk->git.

If I understand the Alioth comment that is the standard place to host Debian packages, but those would be in a git repository. Yeah, that seems fine.

10

I hope when the bitkeeper is ready it will be an option on alioth, along side git, svn, mercurial and others. So then I will migrate from git to bitkeeper.

11

@calhariz: Alioth is a FusionForge system, AFAIK. So FusionForge would need support for BitKeeper first…

On another note, it’s good to see that progress is being made on improving the packagability of BitKeeper!

12

FYI
I have posted the following message on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=824057

I am a user of Debian and bitkeeper on my day job. So I intend to work on this ITP and with upstream to make it good enough for Debian. I have contacted already the upstream and he seams to be interested in helping me. My Plan is:

1 - Try to find all the bits that are needed before working on this.

2 - Code review?

3 - Create a git repository on alioth for the packaging work.

4 - Make a first package that may have bundled software, but is ready to be build by cowbuilder. Not yet ready for Debian.

5 - Make a second package without bundled software. May have security issues. I hope this is ready for Debian experimental and unstable only.

6 - Work with upstream to fix all the security issues, making it security ready to be installed on alioth. Upload to unstable/testing.

I think this plan can take a very long time. But step 5 should be reachable in a reasonable time frame. I will work on this during next DebConf and welcome help from other people, specially a DD, because I am not yet a DM.

Kind regards
Jose M Calhariz

13

Sounds like a good plan. One question I have though is how hard would it be to create one of those PPA repositories that I’ve seen for git[1]. There seems to be a tool for creating a debian package from one of those. I’m not suggesting this as an alternative from debian packaging, more as a testing ground and fast way of letting people play with the package. If there was a chance the instructions for installing bitkeeper in Ubuntu or Debian were:

sudo add-apt-repository ppa:bitkeeper/ppa
sudo apt-get update
sudo apt-get install bitkeeper

It would be a good step in the right direction. Does Debian have anything like the PPA stuff from Ubuntu?

[1] https://launchpad.net/~git-core/+archive/ubuntu/ppa

14

There is no tool. But is not difficult to create one by hand. My plan is when I reach step 4 to talk with bitkeeper.org to set up a private repository of deb packages for bitkeeper, to install it on Debian and possibly Ubuntu.

1 Like
15

Hi,

I asked my sponsor about good practices and he pointed me this page:

https://wiki.debian.org/UpstreamGuide

I did a cursory read and is full of good advice and links to more info.

16

I have a trivial question. What name to give to bitkeeper package and a name compatible with the paying version of it. I think that the ce version should be called bitkeeper-ce and the other version bitkeeper. What people think, specially people from other distributions.

17

I agree with your choices: community is -ce but I’m not a distro manager.

18

I intend to call it “bitkeeper” for Fedora. If there’s going to be some kind of “extended” version from BitMover, then it’ll be up to them to name the package appropriately. It’s less confusing for users and sensible in general as a distribution isn’t going to have two editions of BitKeeper in its repositories.

There’s a similar precedent in Debian. For example, Debian has GitLab CE packaged by the name “gitlab”. Again, it’s less confusing for users when the package is sensibly named.

19

There are more examples in Debian of that packaging name standard and I want if possible that bitkeeper ce package be named the same across several distributions. I will go with you Conan_Kudo and name my package simply bitkeeper.

20

Hi,

I’m a developer and also a NixOS maintainer, looking to incorporate BitKeeper into NixOS upstream. I’m also one of the maintainers of Glasgow Haskell Compiler, and have a bit of familiarity working with distro upstreams and organizing larger projects for many systems (including Windows). So I just wanted to chime in:

  • I also agree with everyone else that any upstream packages are best left as ‘bitkeeper’, and if BitMover wants to offer their own native system packages later on for Enterprisey Features, it’s probably best to call it’ bitkeeper-ee’ (Enterprise Edition) or something like that. In NixOS, I was merely going to call the package ‘bitkeeper’, and it’d be nice if we could all go with that.

  • The packaging plan looks fine to me. A minor suggestion: if BitKeeper itself has a limited subset of needed cryptographic functionality - why bother with libtomcrypt at all? From what you’ve said, you really only need it for some MD5 utilities - but MD5 is only a few hundred lines of code, and a custom utility that can work on both Linux and Windows is not much more. It’s also extremely unlikely that MD5 will be any performance bottleneck. I’d suggest just writing a simple, native MD5 utility in C that works on Windows and Unix systems and use that. This introduces a minor amount of complexity, but has other gains:

  • It completely avoids a dependency on the libtom libraries. In practice all distros deviate between packages at some level; when you have dependencies and want to work on many Linux systems, the effective complexity increases a lot, because you have to worry about the interactions between all the components, among all the different versions your users might have. libtom is very stable, so this is less of a problem, but also…

  • Because you avoid this dependency, it makes users and developers happier, IME. Users don’t have to build it (the difference between compiling an MD5 utility in two .c files and compiling all of libtom is quite big), or have it installed on their systems, the overall amount of trusted code goes down. For example, I just compiled BitKeeper on an ARM machine. On a machine like this, which takes probably 30-40 minutes to compile, making the code smaller with fewer dependencies would have been very desirable - why compile and depend on a whole library when you only need a few hundred lines of code? I estimate eliminating libtom from the build (ignoring all the other suggested improvements) would have decreased build time (and, IME, doing this helps make track dependencies more accurately; it’s much better to expose as much parallelism as possible at in the dependency graph; if you do something like depend on an .a file output by your build rules, you effectively serialize the critical path for anything that depends on that .a file. But also, having to work an entire foreign library into your own build, with accurate dependencies, sometimes sucks. Just getting rid of it avoids all this.)

  • If you want to make life easier on Windows, I strongly suggest you work towards a model of replacing the shell scripts you have with individual C programs that can replace it, where viable. Or with Little (which seems like you’ve already invested much in), I suppose. These are often larger, and more complex, but this significantly reduces build complexity on ‘foreign’ systems, means you don’t have to do things like ship complex environments to end users to support shell, and improves performance. git itself was originally a mash of Perl, shell, and C, but over time, the vast majority of git and all the user facing utilities were rewritten in C and designed to have few dependencies in mind. Not only are these usually faster, they can also avoid particular inefficiencies in the underlying system a script and Unix compat layer can’t; where Linux has very fast exec, CreateProcess on Windows is horribly slow and inefficient. This all, in total, has a big effect on systems like Windows and developing on it, and generally makes porting easier as well.

  • My understanding is that Windows wasn’t always a first class citizen of BitKeeper. As previously noted, this kind of shows in some ways (parts of bk use shell script, invoke custom build GNU tools that work on Windows, etc). If you’d like to help reduce the pain and make life easier for users and people trying to develop with bitkeeper on Windows, I strongly and heartily recommend an environment like MSYS2. MSYS2 is a very good environment for building Unix applications on Windows in a palatable way; we use it for GHC, so that developers can get a recent copy of make, autoconf, etc all on Windows and use it. These are stable ports that work well, and are updated frequently - MSYS2 even uses the pacman Package Manager for managing it all. MSYS2 allows you to build and install packages in an “MSYS environment” (which you can think of kind of like Cygwin - executables end up linking to a msys2.dll which provides some features), or using MinGW, which will produce ‘native’ Windows applications (only using things like ntdll.dll, kernel32.dll, etc and nothing else), so you can ship executables to users. It’s worth investigating and trying out; the net effect is you can immediately build native Windows programs, with Unix like workflows/tools, and a clean package manager, very quickly.

  • Regarding external packages: I have been meaning to set up a BitKeeper PPA for Ubuntu, but I haven’t done so yet. Another thing worth exploring is the OpenSuSE Build Service (OBS). OBS is free to open source developers, and offers a build system service with many supported package formats (rpm, deb) and systems (Ubuntu, Debian, RHEL, CentOS; ARMv7, PowerPC, x86/amd64, etc). It might be worth exploring for building lots of packages for end users; it’s quite a great service for free.

  • Finally, on the note of a build system: if you remove all of these 3rd party components, etc, I would imagine the complexity of the build system would drop dramatically, so from a pure complexity standpoint, it might be more tolerable. That said, something like CMake sounds like a good choice to look at for BitKeeper, long term. CMake has a bit shit syntax unfortunately (IT IS VERY VERBOSE LIKE PASCAL AND HAS LOTS OF CAPS), but it is quite good for portable applications that end-users will use, and offers some nice benefits. For example, using cmake to generate .ninja files (instead of Makefiles) for use with ninja is great - ninja is unbelievably fast, especially at rebuilds, and will make your life better. You can build millions of lines of code (like LLVM for example) and if nothing has changed, ninja has a rebuild time of almost 0 seconds, and if it has, it will rebuild and calculate that efficiently.

This is all a bit long - but having played with BitKeeper, it’s clearly a very interesting product I’d like to see continue to develop and evolve. Hopefully some of these suggestions can help bring it to a wider audience more easily.