Bug 559735 - EGit nightly doesn't install in STS 4.5.1
Summary: EGit nightly doesn't install in STS 4.5.1
Status: RESOLVED FIXED
Alias: None
Product: EGit
Classification: Technology
Component: Releng (show other bugs)
Version: 5.7   Edit
Hardware: PC All
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-31 04:01 EST by Thomas Wolf CLA
Modified: 2020-02-05 02:48 EST (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Wolf CLA 2020-01-31 04:01:42 EST
I just installed STS 4.5.1 on a new computer and tried to install EGit nightly into it. STS 4.5.1 is an Eclipse 2019-12 (4.14 platform).

EGit 5.7.0 could not be installed from the EGit nightly update site. The problem is that the org.apache.httpcomponents.httpclient 4.5.10 included has a dependency on org.apache.commons.codec.binary [1.13, 1.14], which doesn't exist in any of the 2019-12 update sites included by default in STS 4.5.1.

I had to manually add https://download.eclipse.org/tools/orbit/downloads/drops/S20200128200239/repository as additional update site to be able to install EGit nightly.

Can we improve that? I'd suggest to stop including third-party bundles in the EGit p2 repository and instead make it a composite repository referencing the correct Orbit repo. That should avoid such troubles, and still let the EGit p2 repo be the only repo that a user has to add when he wants to install EGit.

Alternatively, include org.apache.commons.codec 1.13 in the EGit update site.

Thoughts?
Comment 1 Michael Keppler CLA 2020-01-31 07:29:28 EST
At work I let Tycho add all transitive dependencies to the update site: https://wiki.eclipse.org/Tycho/eclipse-repository#Creating_a_self-contained_p2_repository. That is guaranteed to work and doesn't require any updating of metadata over time.

I'm not sure if there is a reliable way to test the correctness of the composite approach with each build, on the other hand. So by referencing Orbit in the repo, we could still run into the same issue by just forgetting to update the URL in the repo. That is not hypothetical, since right now the repo refers to neon staging for whatever reason: https://git.eclipse.org/r/#/c/156910/.
Comment 2 Thomas Wolf CLA 2020-01-31 10:27:11 EST
<includeAllDependencies> might be quite a bit of overkill. Wouldn't that include the whole platform, too?
Comment 3 Michael Keppler CLA 2020-02-01 04:18:41 EST
It's 92 versus 26 MB in my quick local test for the egit repo.

I'm wondering if doing _both_ the linking and the transitive inclusion is yet another option. It would lead to having a fail-safe update site, and the included pointers might help finding updates immediately (in cases where we rely on the old versions by default).

I'm also not sure if the non-transitive approach works at all, if the checkbox "check all update sites..." in the update manager is unchecked. Are the links provided by us still used in that case or not?
Comment 4 Thomas Wolf CLA 2020-02-01 09:58:12 EST
(In reply to Michael Keppler from comment #3)
> I'm also not sure if the non-transitive approach works at all, if the
> checkbox "check all update sites..." in the update manager is unchecked. Are
> the links provided by us still used in that case or not?

IIRC they are.
Comment 5 Michael Keppler CLA 2020-02-02 01:39:08 EST
Is this issue a duplicate of bug 559631? Both are dealing with httpclient problems when installing egit nightly. Or are there different aspects?
Comment 6 Thomas Wolf CLA 2020-02-02 06:26:59 EST
(In reply to Michael Keppler from comment #5)
> Is this issue a duplicate of bug 559631? Both are dealing with httpclient
> problems when installing egit nightly. Or are there different aspects?

I don't think it's a duplicate. But something's not right.

In that other bug, EGit nightly was installed into an I-build (2020-03) containing httpclient 4.5.6 (and commons.codec 1.10), and EGit included 4.5.10 and apparently that install also pulled in commons.codec 1.13. So everything was fine, except that the httpclient 4.5.6 remained unused. It didn't cause any user-visible problems, and I would expect platform to update to httpclient 4.5.10 before 2020-03 is released anyway. (Additionally something included and used httpclient 4.5.2 and commons.codec 1.9.0.) 

In my case I wanted to install into a slightly older Eclipse (2019-12), and the install was not even possible without adding a 2020-03 Orbit as update site. I think it should be possible to install EGit by just adding the EGit update site.

Also, there is https://ci.eclipse.org/egit/job/egit.gerrit/1150/console which failed with

[INFO] Resolving class path of MavenProject: org.eclipse.egit:org.eclipse.egit.mylyn.ui:5.7.0-SNAPSHOT @ /home/jenkins/agent/workspace/egit.gerrit/repo/org.eclipse.egit.mylyn.ui/pom.xml
[INFO] Computing target platform for MavenProject: org.eclipse.egit.feature:org.eclipse.egit.mylyn:5.7.0-SNAPSHOT @ /home/jenkins/agent/workspace/egit.gerrit/repo/org.eclipse.egit.mylyn-feature/pom.xml
[INFO] Resolving dependencies of MavenProject: org.eclipse.egit.feature:org.eclipse.egit.mylyn:5.7.0-SNAPSHOT @ /home/jenkins/agent/workspace/egit.gerrit/repo/org.eclipse.egit.mylyn-feature/pom.xml
[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.egit.mylyn.feature.group 5.7.0.qualifier
[ERROR]   Missing requirement: org.apache.httpcomponents.httpclient 4.5.6.v20190503-0009 requires 'java.package; org.apache.commons.codec.binary [1.10.0,1.11.0)' but it could not be found
[ERROR]   Cannot satisfy dependency: org.eclipse.egit.feature.group 5.7.0.qualifier depends on: org.eclipse.equinox.p2.iu; org.eclipse.jgit.http.apache.feature.group [5.7.0,5.8.0)
[ERROR]   Cannot satisfy dependency: org.eclipse.egit.mylyn.feature.group 5.7.0.qualifier depends on: org.eclipse.equinox.p2.iu; org.eclipse.egit.feature.group [5.7.0,5.8.0)
[ERROR]   Cannot satisfy dependency: org.eclipse.jgit.http.apache.feature.group 5.7.0.202002010318 depends on: org.eclipse.equinox.p2.iu; org.apache.httpcomponents.httpclient [4.5.6.v20190503-0009,4.5.6.v20190503-0009]

This used the JGit 5.7.0-SNAPSHOT update site, which includes httpclient 4.5.10, so why was this trying to use 4.5.6?
Comment 7 Michael Keppler CLA 2020-02-02 07:36:13 EST
I was just going to link the failed Jenkins build, but you already did. I'm really  confused by the builds. My other change at https://git.eclipse.org/r/#/c/156910/ also had the updated Orbit change commit as parent, and it was green during review, but is red on master due to the httpclient issue: https://ci.eclipse.org/egit/job/egit/320/

I'm not sure what is going on there. But it seems like we need a short term solution, if even CI builds are failing due to it.

Regarding your suggestion to link to Orbit: Do you really mean to create a composite update site, or do you mean to just add repository references, e.g. adding this to category.xml?

<repository-reference location="https://orbit" enabled="true" />
Comment 8 Eclipse Genie CLA 2020-02-02 08:10:53 EST
New Gerrit change created: https://git.eclipse.org/r/157024
Comment 9 Thomas Wolf CLA 2020-02-02 08:13:58 EST
(In reply to Eclipse Genie from comment #8)
> New Gerrit change created: https://git.eclipse.org/r/157024

That's what I was thinking about. It'll do the trick. (Just verified with an install from a locally built repo into a fresh STS 4.5.1.)

But as you mentioned before, we'll have to remember to update this URL every time we change the Orbit used in the target platform. So I'm not 100% sure this is the best way to go. Or this there some way to automate this?
Comment 10 Eclipse Genie CLA 2020-02-02 13:35:59 EST
Gerrit change https://git.eclipse.org/r/157024 was merged to [master].
Commit: http://git.eclipse.org/c/egit/egit.git/commit/?id=7d3e66aa073bcffce787a4b09fd3443bc7aea25a
Comment 11 Thomas Wolf CLA 2020-02-02 16:02:18 EST
(In reply to Eclipse Genie from comment #10)
> Gerrit change https://git.eclipse.org/r/157024 was merged to [master].
> Commit:
> http://git.eclipse.org/c/egit/egit.git/commit/
> ?id=7d3e66aa073bcffce787a4b09fd3443bc7aea25a

Don't know what's going on.

When building the repo locally, I can install into STS 4.5.1 by just adding that local repo via a file:// URI with that link added.

If I use the EGit nightly update site https://download.eclipse.org/egit/updates-nightly/ , which now also includes this link, I still cannot install.

:-(

So I guess we have to add commons.codec 1.13 all the same (in the JGit repo already).
Comment 12 Thomas Wolf CLA 2020-02-02 16:19:17 EST
Besides, the maven httpcomponents.httpclient 4.5.10 depends on commons.codec 1.11. Why does the Orbit bundle specify 1.13?

Anyway, commons.codec has no further dependencies, so just adding that should be sufficient.
Comment 13 Eclipse Genie CLA 2020-02-02 16:46:05 EST
New Gerrit change created: https://git.eclipse.org/r/157030
Comment 14 Thomas Wolf CLA 2020-02-04 17:08:21 EST
(In reply to Thomas Wolf from comment #11)
> Don't know what's going on.
> 
> When building the repo locally, I can install into STS 4.5.1 by just adding
> that local repo via a file:// URI with that link added.
> 
> If I use the EGit nightly update site
> https://download.eclipse.org/egit/updates-nightly/ , which now also includes
> this link, I still cannot install.
> 
> :-(

I suspect that might have been due to a problem in STS 4.5.1 which probably picked up a p2 repo mirror that didn't have the Orbit link yet. See below.

Now it worked (with EGit 5.7.0.202002031020 and JGit 5.7.0.202002022348) from https://download.eclipse.org/egit/updates-nightly/.

But there's some very ugly caching and/or mirroring going on, or STS 4.5.1 has a funky reload mechanism. I tried several times, and while the first time it worked, visiting https://download.eclipse.org/egit/updates-nightly/ at that time showed EGit features versioned 5.7.0.202002042113 and JGit features 5.7.0.202002041709. Even reloading the repo info from the server didn't get these version numbers.

Was there a bug in Eclipse 2019-12 relating to p2 caching information or going to mirrors without telling?

About 15 minutes later I tried again (again in a fresh STS 4.5.1). Again got the versions from 2020-02-03/02 shown, not the 2020-02-04 ones, but then the install  failed with

  An error occurred while collecting items to be installed
  No repository found containing: osgi.bundle,org.eclipse.egit,5.7.0.202002031020

And STS 4.5.1 still kept telling me that was the latest version. Even though I knew better, and even though other Eclipse instances (2018-12 and 2019-06 on the same machine, but also a 2020-03 on another machine) all picked up the correct 5.7.0.202002042113.

In any case it appears that adding the Orbit link in the EGit repo has indeed solved the problem mentioned in comment 0.

The second change https://git.eclipse.org/r/157030 adds commons.codec, but that is just the "belt and suspenders" solution. (Might be nice for people without Internet connection working with a local copy of the update site, though.)
Comment 15 Thomas Wolf CLA 2020-02-04 17:39:48 EST
(In reply to Thomas Wolf from comment #14)
> (In reply to Thomas Wolf from comment #11)
> > Don't know what's going on.
> Was there a bug in Eclipse 2019-12 relating to p2 caching information or
> going to mirrors without telling?

Bug 553783 is *exactly* what I am seeing in STS 4.5.1, including the bit about changing https to http and back.
Comment 16 Eclipse Genie CLA 2020-02-04 19:34:51 EST
Gerrit change https://git.eclipse.org/r/157030 was merged to [master].
Commit: http://git.eclipse.org/c/jgit/jgit.git/commit/?id=708525c67e008acc33dd8259cdcf11d2f3e9107b
Comment 17 Thomas Wolf CLA 2020-02-05 02:48:24 EST
EGit nightly repo now (as of feature versions EGit 5.7.0.202002050730 and JGit 5.7.0.202002050102) contains both the Orbit link and commons.codec 1.13.