Feature #338

using the toolkit installer on "slow" (<=10 MBit/s) connections - tar snapshots instead of git?

Added by Simon Schulz almost 10 years ago. Updated over 8 years ago.

Status:ClosedStart date:2014-11-22
Priority:NormalDue date:
Assignee:Jan Moringen% Done:

20%

Category:build-generatorSpent time:-
Target version:-

Description

I am trying to set up a distribution at home over a DSL Connection. This takes forever...

What about a policy that we use tar snapshots for all (or at least external) projects files where possible instead of git?
For example opencv/morse etc take ages to download or fail due to timeouts.
(it seems like the opencv git is also extremly slow right now, its syncing with <100kb/s)

The situation how it is right now (checking out during job config, then twice during build) is not really usable to me.
My solution right now is that i switched back to install the (large) projects manually instead of using the toolkit installer which i really do not want to do.


Related issues

related to Cognitive Interaction Toolkit - Feature #412: multiple cloning of gits - an idea for load reduction for... Resolved 2015-06-10

History

#1 Updated by Simon Schulz almost 10 years ago

  • Description updated (diff)

#2 Updated by Simon Schulz almost 10 years ago

the build gen could add a timeout for git pulls. i now do this manually:

job configuration, and under git plugin section:

Click "Add" 
Click "Advanced clone behaviours"
then i set the checkout timeout to 120

#3 Updated by Simon Schulz almost 10 years ago

a nice workaround:
open the project settings and replace the git url with /home/sschulz/src/opencv
after you checked out opencv to ~/src/opencv

#4 Updated by Jan Moringen about 9 years ago

  • Category set to build-generator
  • Status changed from New to In Progress
  • Assignee set to Jan Moringen
  • % Done changed from 0 to 20
Current progress:
  • Clone operations during build have reduced from two to one (kind of generated jobs has been changed matrix project -> "normal" project)
  • Incremental download of updates (i.e. git pull) has been prototypically implemented for the build-generator side and is currently undergoing testing (see #412)
  • Incremental download for the Jenkins side may be possible in a similar fashion but this needs exploration

I vote against tar-archives as an additional form of distribution/retrieval because of the added complexity.

#5 Updated by Jan Moringen about 9 years ago

  • related to Feature #412: multiple cloning of gits - an idea for load reduction for servers + slow connections added

#6 Updated by Martin Wiechmann almost 9 years ago

Please consider this as an alternative:

git clone -b <tagName/branchName> --depth 1

From the git-clone manpage:

--branch <name>, -b <name>

Instead of pointing the newly created HEAD to the branch pointed to by the cloned repository’s HEAD, point to <name> branch instead. In a non-bare repository, this is the branch that will be checked out. --branch can also take tags and detaches the HEAD at that commit in the resulting repository.

The option --depth 1 creates a shallowed copy and efficently reduces both remote system load and transfer time.

#7 Updated by Jan Moringen over 8 years ago

  • Status changed from In Progress to Closed

Martin Wiechmann wrote:

Please consider this as an alternative:

[...]

From the git-clone manpage:

--branch <name>, -b <name>

Instead of pointing the newly created HEAD to the branch pointed to by the cloned repository’s HEAD, point to <name> branch instead. In a non-bare repository, this is the branch that will be checked out. --branch can also take tags and detaches the HEAD at that commit in the resulting repository.

The build-generator already uses this.

The option --depth 1 creates a shallowed copy and efficently reduces both remote system load and transfer time.

This is a configuration option as well.

Also available in: Atom PDF