Push after cp: Failed to locate BAM data for the following deltas

Bugs
1

We’ve never deliberately used BAM but it’s on by default and we have a few binary files big enough to have been automatically flagged as BAM. Someone just tried to bk cp a tree including some of them. Everything seemed fine until they came to bk push. We had no bam server configured, so I figured that was just an operational problem for us to solve. In the following transcript, I give my best shot at setting it up and I still can’t get it to work:

martind@sirius:~/tmp/D149938$ mkdir parent
martind@sirius:~/tmp/D149938$ cd parent
martind@sirius:~/tmp/D149938/parent$ bk init
martind@sirius:~/tmp/D149938/parent$ bk bam server
This repository has no BAM server.
martind@sirius:~/tmp/D149938/parent$ bk bam server .
Set BAM server to .
martind@sirius:~/tmp/D149938/parent$ cd ..
martind@sirius:~/tmp/D149938$ bk clone parent child
Clone file:///home/martind/tmp/D149938/parent
   -> file:///home/martind/tmp/D149938/child
clone                              100% |==============================| OK
martind@sirius:~/tmp/D149938$ cd child
martind@sirius:~/tmp/D149938/child$ dd seek=64K bs=1 if=/dev/null of=64KiB.bin
0+0 records in
0+0 records out
0 bytes copied, 0.000192978 s, 0.0 kB/s
martind@sirius:~/tmp/D149938/child$ bk add 64KiB.bin 
64KiB.bin revision 1.1: +65536 -0 = 65536
martind@sirius:~/tmp/D149938/child$ bk commit -y"bam test 7"
ChangeSet revision 1.2: +1
martind@sirius:~/tmp/D149938/child$ bk push
Push file:///home/martind/tmp/D149938/child
  -> file:///home/martind/tmp/D149938/parent
push                               100% |==============================| OK
martind@sirius:~/tmp/D149938/child$ bk bam push
martind@sirius:~/tmp/D149938/child$ bk cp 64KiB.bin bk-cp-bam.bin
 [BAM xfer]                        100% |==============================| OK
bk-cp-bam.bin revision 1.2: +0 -0 = 65536
martind@sirius:~/tmp/D149938/child$ bk commit -y"bam test 8"
ChangeSet revision 1.3: +1
martind@sirius:~/tmp/D149938/child$ bk push
Push file:///home/martind/tmp/D149938/child
  -> file:///home/martind/tmp/D149938/parent
bk: failed to get repository read lock. |==============================|
sfio.c:800 (7.3.3): fread: Invalid argument
get: failed to fetch BAM data
Failed to locate BAM data for the following deltas:
	00000000.-Na8tWwWifzvKLV8IkdbrQ martind@sirius.us.dev.bluearc.com|bk-cp-bam.bin|20211130234954|00000 61a6b8a1BhFrOXfgwHjKUfhz-mROcA
Check failed.  Resolve not completed.
Your repository should be restored to its previous state.
We are running a full consistency check to verify this, please wait.
check passed.
resolve: RESYNC directory left intact.
====================================================
martind@sirius:~/tmp/D149938/child$ bk version
BitKeeper version is bk-7.3.3 for x86_64-glibc23-linux
Built by: wscott@debian40-64.bitkeeper.com in /build/dev-oss-wscott/src
Built on: Sat Dec 29 2018 04:23:45 PST (36 months ago)
Running on: x86_64-glibc23-linux,4.9.0-15-amd64
martind@sirius:~/tmp/D149938/child$

In the original setup, the parent was an ssh away and the failure presented with an indefinite hang, with the same bk abort being needed in the parent, though without the error message that would warn of that:

martind@sirius:~/tmp/D149938/child$ bk parent sirius:/home/martind/tmp/D149938/parent
Add parent sirius:/home/martind/tmp/D149938/parent
martind@sirius:~/tmp/D149938/child$ bk pull
Pull sirius:/home/martind/tmp/D149938/parent
  -> file:///home/martind/tmp/D149938/child
Nothing to pull.
martind@sirius:~/tmp/D149938/child$ bk push
Push file:///home/martind/tmp/D149938/child
  -> sirius:/home/martind/tmp/D149938/parent
push: remote locked, trying again...
push: remote locked, trying again...
push: remote locked, trying again...
push: remote locked, trying again...
^C
martind@sirius:~/tmp/D149938/child$ cd ../parent
martind@sirius:~/tmp/D149938/parent$ ls RESYNC
BitKeeper
martind@sirius:~/tmp/D149938/parent$

The only way I’ve managed to get this to work is to set the parent’s bam server to the child temporarily and pull the change. Every time you bk cp a big file? I must be missing something.

2

It has been forever since I worked on that code. My guess is that ‘bk cp’ doesn’t handle BAM at all. Since the data didn’t change the BAM data already exists on both sides, but you have to update the metadata to make the new delta key point at the BAM data as well. Maybe try ‘bk bam repair’ before you push.

3

No there is some support. See the BAM stuff in src/cp.c and src/t/t.cp has testcases demonstrating running ‘bk cp’ on a BAM file. But the testcase didn’t try pushing a repo about doing a ‘bk cp’ of a BAM file.

4

Maybe try ‘bk bam repair ’ before you push.

Maybe that was obsoleted by the second reply but I had a go anyway:

martind@sirius:~/tmp/D149938/child$ bk bam repair
bam repair: no BAM sources found, use -@<url>?
martind@sirius:~/tmp/D149938/child$ bk bam check
Loading list of BAM deltas, 2 found (1 loose) using 128KB.
 [bam]                             100% |==============================| OK
All BAM data was found, check passed.
martind@sirius:~/tmp/D149938/child$ bk bam repair -@bk-cp-bam.bin
Loading list of BAM deltas, 2 found (1 loose) using 128KB.
 [bam]                             100% |==============================| OK
All BAM data was found, repair passed.
martind@sirius:~/tmp/D149938/child$ bk push
Push file:///home/martind/tmp/D149938/child
  -> file:///home/martind/tmp/D149938/parent
get: no server for BAM data.       100% |==============================|
get: failed to fetch BAM data
Failed to locate BAM data for the following deltas:
	00000000.-Na8tWwWifzvKLV8IkdbrQ martind@sirius.us.dev.bluearc.com|bk-cp-bam.bin|20211201000103|00000 61a6bb3eGRTmMYEKKLyQBlSxHeCv2A
Check failed.  Resolve not completed.
Your repository should be restored to its previous state.
We are running a full consistency check to verify this, please wait.
check passed.
resolve: RESYNC directory left intact.
====================================================
martind@sirius:~/tmp/D149938/child$

I did mean to mention that we found another way of doing what we were trying to achieve, without making copies of files we hadn’t realized were large, so it’s not even mildly inconveniencing us, just something I thought I should report in case others etc.