Welcome to the Kiln Knowledge Exchange

Question

2

Thanks in advance for your help,

I have contacted FogCreek / Kiln, who directed me to this --chunked option {http://kiln.stackexchange.com/questions/396/what-if-im-having-trouble-importing-a-very-large-repository}... but it still does not seem to work... they suggested I try posting here to see if the community has any suggestions.

I have a tried the hg push --chunked http://{my kiln repositry} command only to get the following results: searching for changes searching for changes searching for changes abort: out of memory

We do have some large files checked into our repository - in particular .net framework 3.5 install, msde ect... which subversion handled no problem.. I saw a post that said that Hg will run out of memory on 10 meg file if the machine doing the push has 2 gig of ram or less... could that be affecting this? Is there any work around for large repositories with large files?

Any ideas?

-Deva

edited May 17 2010 at 15:46

Benjamin Pollack♦♦
16.1k●1●20●27

asked May 12 2010 at 18:11

Deva
41●3

Rob Sobers · Answer 1 · 2010-11-29 15:52:46Z UTC

3

Mercurial can handle small binary files just fine; in fact, we have a lot of them (some not really that small, in fact) in Kiln's own repository. The problem comes because Mercurial requires roughly 2 * sum(file sizes) RAM to make a new changeset. Since Tortoise uses a 32-bit version of Mercurial, that means that commits need to be less than about 500 MB each for Mercurial to deal with them. Do you know whether you're over that limit?

ANSWER: I spoke to Deva on the phone. One of his changesets was well over 500 MB all by itself, which was indeed causing the problem. There were two parts to the solution:

Push from a 64-bit machine.
Bypass Kiln entirely for the initial push, since Kiln was running on a 32-bit machine.

All of us on the Kiln team are keenly aware that Mercurial's handling of large binaries needs to be improved, and we're actively working on it right now. Stay tuned for announcements on that front over the next several months.

edited Nov 29 2010 at 15:52

Rob Sobers♦♦

answered May 12 2010 at 20:48

Benjamin Pollack♦♦
16.1k●1●20●27

	mercurial.selenic.com/wiki/BigfilesExtension that is where I read the 10 mb file comment... I am trying to migrate our main subversion repo over to kiln, first do a 1) hg convert svn://source/svn target 2) cd \target 3) hg push --chunked http://{my kiln repositry} How can I check the size of the push? I did a dir of the \target (which is just a .hg folder) that is 2,799,137,106 bytes aka ~2.x gig... we went with this approach as the kiln importer was giving me other errors. I have successfully imported 4 other repositories, but this one is the largest by far. – Deva May 14 2010 at 13:00
	I am doing this from the command line, on the svn / hg server (they are both on the same box) The OS on the server is 32 bit windows 2003, but my box is a 64 bit windows 7. Can I copy the target directory to my 64 bit box, and do the hg push --chunked from there? - hm will try now, and post back the results. – Deva May 14 2010 at 13:05
	It is still runing, but it has gotten further than before: searching for changes searching for changes searching for changes remote: transaction abort! remote: rollback completed remote: abort: received changelog group is empty searching for changes remote: transaction abort! remote: rollback completed remote: abort: received changelog group is empty searching for changes remote: transaction abort! remote: rollback completed remote: abort: received changelog group is empty searching for changes – Deva May 14 2010 at 13:55
	Okay, so it looks as if you're no longer running out of memory; is that right? I also have some Kiln On Demand crashes from last night that appear to match this, showing timeouts. If that is the situation you're in, we'll figure out why things are timing out. – Benjamin Pollack♦ May 14 2010 at 15:36
	Yes after several hours or running - 6-9 hrs I got this message: abort: error: An established connection was aborted by the software in your host machine The exact log is : searching for changes searching for changes searching for changes remote: transaction abort! remote: rollback completed ---- repeats several more times... (50+) then error ---- ---------------- Not really sure what the next step should be, I am going to retrie the command... – Deva May 14 2010 at 23:17

show 2 more comments

When hg push --chunked still throws abort: out of memory what should you do?

1 Answer