zzz.i2p

Development discussions
mtn/git meeting notes « Meetings « I2P Development
 
Thu, 02 Jan 2020, 03:53pm #1
zzz
Administrator
Zzz

- 12/29/19 git meeting notes:
- We may be the only project left using mtn.
- zzz asked mtn maintainers to port to libbotan 2; no response. zzz tried porting the deb pkg to use libbotan 2 but got stuck. Not a long-term solution anyway.
- git resume still not a thing. Workaround is download bundles built weekly
- Most of the team is not using in-net mtn; anonymity and dogfooding not a priority
- str4d's git.repo.i2p is up and current; it's using some basic serving software, we can do better
- idk has a git site, b32 only, using basic serving software, we can do better. idk to reg. a hostname and add bote and muwire
- best sw is probably gitlab? meeh to set one up, and add i2p.i2p, i2p.www, bote, muwire. this will be in-net and clearnet
- there's dozens of branches in mtn. Most are either on GH now or are one-offs, need to review what's missing in GH
- we aren't GH or Microsoft haters; GH is fine
- main user of mtn is zzz; ~900 checkins in 2019; impact to productivity and schedules unknown
- build scripts, build docs, default tunnel, user docs all need to be changed for git
- website auto-updater must be changed for git
- sadie and zzz at roughly same skill level and workflow with git now
- sadie not pushing her revs solely due to mtn; will switch to git push when we switch
- 3 month target for sadie/zzz proficient using git push in test project and/or muwire (start with idk repo, then switch to mikal repo)
- 6 month target for switch-flip to git. No gradual change possible.
- After switch-flip, we will push/pull to/from our gitlab. Triggers will synch with GH
- PRs from GH can be synced with gitlab and pulled in there
- Bug tracking comes with many git servers, but trac replacement is really a different topic
- Git plugin for trac presumably works and should be installed after the switch

Fri, 20 Mar 2020, 04:59pm #2
idk
Contributor

The Git Situation as of 3/20/2020
=================================

Git Bundles:
------------

Given that the bundles have to contain much of the history and tags, I think weekly may be more often than necessary for git bundle generation. I would suggest that bundle generation take place after generating the release tag, and then have the users fetch the difference between the bundle and the current head using a hosted git service(Presumably git.idk.i2p at this point).

In-Net Use:
-----------

The most reliable way to get the whole repo is going to be the git bundle + bittorrent for the forseeable future. Instructions for obtaining the bundle and setting up a repository from it can be viewed here: http://i2p-projekt.i2p/en/blog/post/2020/03/18/..., there are a number of people helping seed that bundle so it can be obtained reliably. When we officially move to git, this will become part of the new developers guide.

HTTP/S Cloning if the i2p.i2p repository remains problematic for many people because of hangups in transmission. Cloning to --depth 1 with SSH works most of the time.

Once you have a repository, git fetch can be used in a more-or-less resumable manner. TL:DR once you have a source you can clone from, i.e. a bundle, using git inside I2P seems pretty easy. My dogfooding of in-i2p git with the i2p.i2p repository has thusfar been limited(But not nonexistent), but I have been using it for my out-of-tree projects successfully for weeks. A users guide, with a suggested workflow, has been made available on the blog, https://geti2p.net/en/blog/post/2020/03/06/git-... and it will be incorporated into the new developers guide when we move to Git.

Git Services:
-------------

git.idk.i2p - I will continue to administer this gitlab even if it doesn't become the official project gitlab. It is physically located in my home, but has offsite backups which will be made available to the team before git launch. These backups do not include the tunnel keys or the SSH host keys, but do include all repositories and non-sensitive config files.

### [Gitlab Web Interface](http://git.idk.i2p) - http://git.idk.i2p

- Base32: 7qeve4v2chmjdqlwpa3vl7aojf3nodbku7vepnjwrsxljzqipz6a.b32.i2p
- Onion: 47ggr2fa3vnwfyhvgskzdmr3i32eijwymxohtxsls45dulmriwxszjad.onion

git.repo.i2p - This is the longest-running git service on I2P. It is using cgit.

git.psi.i2p - This service is currently 502 Unavailable. It was using Gitea.

Gitlab - My impression of gitlab is that it's incredibly easy to administer. I have had almost no downtime this month, last month it was less than an hour. I have made a guide to hosting a Gitlab instance on I2P, and the various implications of this, available on the project blog, https://geti2p.net/en/blog/post/2020/03/16/gitl... and when we officially move to git this will take the place of the mtn hosting instructions.

Branches
--------

There are still some missing branches on my gitlab. I'll add the ones on github today, but for the ones that are not on Github it may take longer depending on how the non-included repositories work with mtn's export scripts. MuWire is under Zab's github account, I'd prefer that if MuWire is going to be hosted on this repository, that he export the repository to a gitlab account of his own and fork it to the i2p_hackers group instead, so that his working branch remains in his control.

Build Scripts, configs:
-----------------------

`ant bundle` builds a git bundle and a torrent for the bundle from a *complete* git repository(not a shallow clone).

An i2ptunnel.config.d file for the git service is already ready, it is trivial add it to the i2ptunnel.config file where it will be migrated automatically. An *example* without the working base32 and too-short tunnels can be found here for now https://github.com/eyedeekay/git-over-i2p/blob/..., but the tunnel wizard is probably still easier for most people as of 3/20/20.

What's left?
============

Best I can figure is:

Missing branches need to be converted to gitlab repositories [Done]

Github export tool is giving a 500 error when using a token. There are a number of possible culprits, a resolution should be available soon. [Tentatively Done]

Website auto-update has not been examined yet.

Git hooks need to be written for syncing with github.

Somebody, probably me, needs to finish figuring out what's wrong with Tracboat so that we can turn trac tickets into gitlab issues.

Last edited: Sun, 22 Mar 2020, 07:19pm by idk

Sun, 22 Mar 2020, 04:28pm #3
zab
I2P Legend

MuWire is now imported to GitLab. You can fork at http://git.idk.i2p/zlatinb/muwire

I haven't figured out how to propagate changes to and from GitHub, may need to be done manually.


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Sun, 22 Mar 2020, 06:34pm #4
zab
I2P Legend

Automatic push works. So going forward all MW development will happen on GitLab, with the GitHub being a read-only mirror of sorts.


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Mon, 23 Mar 2020, 12:00pm #5
zzz
Administrator
Zzz

Thanks idk for the long status writeup. I haven't started playing with your gitlab yet but hope to soon.

In the meantime I have several questions:

- I assume that instead of the torrent bundle, or in-net cloning, people could just do a one-time clone from github if they haven't already? But this doesn't seem to be documented anywhere?

- What is the "i2p_hackers group"? What's its function and why is it called this? It sounds like some test account and its presence some of the gitlab urls that have been shared confuses me.

- In the December meeting we agreed that git migration did not encompass a trac-to-git migration for issues tracking, and that it's a separate topic TBD. I don't believe we've had any meetings on the topic since. Some people on IRC are assuming that this effort does include a migration for issues and even that it can happen first, or now. What's your thinking on this topic or is there a plan already? To avoid confusion we should make it clear what we're doing w.r.t. tickets

- Your post is a good summary of where we are but vague on the plan going forward. Is there a schedule you haven't published other than what we came up with in December, which was 3 months to trials and 6 months overall? Is that still true? I'm concerned that I'm holding things up. Echelon says I'm not, but you said in IRC you're almost done, so that sounds like I'm about to be.

-Please correct any thing that's changed from the OP. For example, echelon said that we will not be migrating to a mikal server. Anything else?

- zab got gitlab-to-github syncing working but says github will be readonly. I thought in December we said the syncing could be bidirectional? PRs would go both ways?

- Is there a way to browse git.idk.i2p without logging in? Seems like there should be... should be a requirement... we shouldn't require people to register to get our code.

Mon, 23 Mar 2020, 12:18pm #6
zab
I2P Legend

- zab got gitlab-to-github syncing working but says github will be readonly. I thought in December we said the syncing could be bidirectional? PRs would go both ways?

That's what I thought too after the December meeting but couldn't find anything in the GitLab options that allows for such syncing. What does work is a one-time import of all GitHub PRs and issues, but after one has to choose a primary platform and stick with that.

According to the GL docs it's possible to do either "push" or "pull" syncing, but for some reason I could set up only "push" syncing to GH. Furthermore, the push syncing crushes any diverging changes made to the target repo as if they never happened.


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Mon, 23 Mar 2020, 12:22pm #7
zzz
Administrator
Zzz

Also: In the March monthly meeting we discussed email address selection issues and how to make things work right with the github bridge. It would be good to put that info on the registration page, especially that they should use the same address as on github if they wish things to be linked correctly.

And if they don't, do mail.i2p addresses work? Should users use i2pmail.org addresses instead?

Mon, 23 Mar 2020, 01:26pm #8
zab
I2P Legend

re: email - I never got a registration link or any other email from GL for that matter. Idk approved my registration request manually.

Maybe the easiest thing to do is to remove the email confirmation step and approve requests manually. The commits on GH will always appear from the user that has set up the push sync; for that they need to enter their GH authentication token into GL.


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Mon, 23 Mar 2020, 02:56pm #9
zzz
Administrator
Zzz

hmm. that's not the way it was explained in the meeting, as I understood it:

20:31:49 <zzz> I have a question about user names on git.idk - do we need to pick a username unused on GH, or do we need to defensively register it on GH, to make it all work right?
20:32:12 <nextloop> zzz: github identifies the committers based on e-mail addresses.
20:32:17 <zzz> there was a report on zzz.i2p a while back that there are several fake zzz-i2p accounts on GH. is that a problem?
20:32:42 <nextloop> so if you add your email you use for i2p git to github the commit will be linked to your account
20:33:16 <nextloop> eyedeekay: is the regular torrent archive already in place? if i remember correctly you were working on that
20:33:48 <eyedeekay> Well it's generatable, but there's nothing scheduling it yet
20:34:32 <zzz> so I need to register on git.idk with a valid clearnet email address if I want to (before or after) register on GH? or that's a local setup thing?
20:34:55 <zzz> anyway, we're in the weeds here, sorry, I'll work with idk to figure it out
20:35:17 <eyedeekay> You don't need to pick an unused GH username AFAIK, you could work entirely from the gitlab instance and we wouldn't need github at all
20:35:17 <eche|on> clearnet email should be in this case the i2pmail.org address IMHO
20:35:46 <nextloop> zzz: yes for github you need to verify the email. use i2p-mail.org perhaps?
20:35:54 <eche|on> currently the plan is to use the gitlab (idk in i2p net git instance) for our work and sync to github
20:36:23 <eche|on> the trac tickets would be on in-net gitlab server
20:36:25 <nextloop> eyedeekay: i would be motivated to setup such an automatic archiving
20:36:27 <zzz> I just want to make sure it's not linked to some fake zzz account when it gets bridged to GH
20:36:47 <eche|on> (sorry for the hassle, gitlab and github are both servers with lots of features around git, both do nearly the same tasks)
20:37:18 <eche|on> valid point, zzz

Mon, 23 Mar 2020, 03:04pm #10
zab
I2P Legend

Hmm, https://docs.gitlab.com/ee/user/project/reposit... also http://git.idk.i2p/zlatinb/muwire/-/settings/re... (expand "Mirroring Repositories")

This user will be the author of all events in the activity feed that are the result of an update

Maybe I misunderstand; I haven't had commits by anyone else since MuWire moved to GitLab so I can't test.


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Mon, 23 Mar 2020, 05:22pm #11
zzz
Administrator
Zzz

the last GH rev is 3 days ago but the last gitlab rev is 6 days ago, so it doesn't appear that the syncing from GH to gitlab is working yet.

Mon, 23 Mar 2020, 06:38pm #12
zab
I2P Legend

zzz wrote:

the last GH rev is 3 days ago but the last gitlab rev is 6 days ago, so it doesn't appear that the syncing from GH to gitlab is working yet.

It needs to be enabled per repository. It's working for MuWire


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Mon, 23 Mar 2020, 09:01pm #13
idk
Contributor

Took a look at the posts today, I have answers for some of these questions and I'm tracking down answers for the rest. I'll be back this evening with the rest, but for now

- Yes, it is possible to check out from github, then add gitlab as an alternate remote or replace the github remote. Effectively this is not different from cloning via a bundle and switching to the gitlab remote. I shall document this in a blog soon.

- I created the i2p_hackers group for us to co-own the "Project" repositories, analogous to how github has an I2P group of which team participants are members. This is so that we can use our own accounts to keep our working copies of the code, but also so that we can co-own the project group in case something happens to me. Right now Echelon and I co-own this group, me since I'm git admin, and Ech as a backup.

- AFAIK we won't be migrating to a server administered by Mikal. I've been doing this reasonably successfully so far, I have reasonable availability to maintain it. No sense making more work for him if I'm doing it fine right now. I don't think a consensus has been reached that migrating away from trac is the right thing to do, but it would be a backup bugtracker when trac goes down.

- Repositories can be in 3 states, public, internal, or private. Only Admins and the repo owner(group members if the repo is owned by a group) can see private repositories, they always require login. Internal repositories can be seen by anyone logged in and can be forked/cloned by anyone logged in. They are only private in this trivial sense. Finally public repositories can be seen from http://git.idk.i2p/explore/projects. Currently new repositories are configured to be public.

- Re: github-to-gitlab syncing, I went back over the options, previously I had used the repository URL and a regular git clone, which is why they aren't syncing yet. That can be fixed one way or the other, the preferred way would be without having to re-create them all from scratch by running the import tool again. I am working on an answer for this, I don't think re-running the import tool is required but I'll try to know all the implications of that by tonight.

Mon, 23 Mar 2020, 10:05pm #14
zzz
Administrator
Zzz

thanks for the answers.

Next question:

mtn was somewhat of a pioneer in enforcing signed checkins, having a trust list, and ensuring that only committers on the trust list were included in the visible head.

git was many years behind, but eventually added some features with GPG-signed checkins, although as I understand it these features are not commonly used?

What's the security design of our new setup? Are we going to require GPG signatures? Or is there going to be an access control in the i2p_hackers group?

The i2p_hackers group membership is not the same as the github admins, I don't think, because nextloop has push privs to github for the syncing. There's 34 approved members in my _MTN/montonerc trust list, although most have vanished. What's the process going to be for push privs to your gitlab (I guess on a per-project basis) and who's going to adminsiter that?

Mon, 23 Mar 2020, 10:22pm #15
zab
I2P Legend

Access control in the mtn sense is not really necessary and only somewhat applicable to the git model.

You can fork i2p_hackers/i2p.i2p to zzz/i2p.i2p and cut releases from there. Then when someone else wants to make a release (bus factor) they can fork off of your fork, and so on. There is no "golden standard" enforced with a distributed acl.


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Tue, 24 Mar 2020, 03:48pm #16
idk
Contributor

If nextl00p creates an account I'll give push access to i2p_hackers immediately.

With a little more elaboration, the way the i2p_hackers repository ought to work, at least via the WebUI***, in my view is this:

Developer Process
=================

0. Developer wants to work on a feature or fix a bug
1. Developer forks i2p.i2p to his own user namespace and clones from there if developer has not done so already
1b. Developer updates updates the code of local master branch if necessary
2. Developer creates a "branch"* locally to develop the feature on
3. Developer implements feature or fixes bug
4. Developer commits changes and pushes to remote
5. When the Developer is ready, they submit a merge request. Possibly, the final commit before a merge request needs to be signed.

Code-Review Process
===================

0. Once a merge request is submitted, it is not yet in the upstream repository.
1. The Gitlab Web UI will present the merge request in a tab, on the repository where the merge request was submitted.
2. A developer with access to the i2p_hackers group must "close" the merge request. This can be configured to require the developer to "review" the changeset from the branch, in which case each diff displayed in the WebUI will be accompanied by the ability to comment on the changes and request further revision or clarification.**
3. After the developers are satisfied with the changes, they close the merge request. If you wish, it might be worth the trouble of creating a tag and signing it at this time.

* One of the differences between Monotone and Git is the use of the term branch. This refers to the Git usage of the term, where a branch is essentially a working copy of the code from the repository where it was branched from. Repository "Forks" are technically branches too, just with a different UI to create the user namespace.
** At this time, anyone in the i2p_hackers group can close their own merge requests and review is voluntary, not enforced.
*** Only covering the Gitlab WebUI at the moment. There are alternatives, including via e-mail for instance, which can be used to send in merge requests that become available in Gitlab.

Tue, 24 Mar 2020, 04:56pm #17
zzz
Administrator
Zzz

re: nextloop, perhaps we want the reverse - to remove him from GH access once we 'flip da switch' to ensure his script doesn't corrupt GH

re: "i2p_hackers", if this is the official access-controlled development group, we should rename it to something official-sounding, like "i2p developers"

re: GPG (#14 above), one reason jrandom chose mtn 15 years ago was its strong security model, with both access control and signed commits. We should affirmatively decide whether we want to continue this in git, with GPG signatures of every commit, or not.

re: process (#15 above), you're proposing a branch and merge request for every change? Or the above is only for contributors not in the i2p_hackers group? I'm very confused. In any case this sounds very different than the way we do things now. Are you saying we shouldn't just push changes to the repo? Or is all the above the way it would have to work if we wanted to GPG sign everything?

If all of this is getting too hard to do via forum posts or IRC, maybe it's time for a meeting to figure things out.

Sat, 28 Mar 2020, 12:40pm #18
zab
I2P Legend

zab wrote:

Hmm, https://docs.gitlab.com/ee/user/project/reposit... also http://git.idk.i2p/zlatinb/muwire/-/settings/re... (expand "Mirroring Repositories")

This user will be the author of all events in the activity feed that are the result of an update

Maybe I misunderstand; I haven't had commits by anyone else since MuWire moved to GitLab so I can't test.

Now I have an example commit. On GL:

http://git.idk.i2p/zlatinb/muwire/commit/bbf973...

and on GH it appears as:

https://github.com/zlatinb/muwire/commit/bbf973...


email: zab@mail.i2p Irc2P/keybase: zlatinb
blog: http://zab.i2p
MuWire: http://muwire.i2p
MuCats: http://mucats.i2p
MuWire nickname: zlatinb@3k2gijdfdcuczkfypfddj4qsnnf744mj

Wed, 01 Apr 2020, 07:12pm #19
idk
Contributor

GPG Post
========

Easy part first, this is Gitlab documentation that overviews how Gitlab and GPG-verified commits work. https://docs.gitlab.com/ee/user/project/reposit.... You upload public keys per this section: https://docs.gitlab.com/ee/user/project/reposit..., and then set GPG up with your git client like so: https://docs.gitlab.com/ee/user/project/reposit.... All pretty straightforward. The last part, however, should be different for us, instead of configuring git to sign all commits across all repositories with the same key, it would behoove those of us who are anonymous to configure signing on a local basis.

## So, instead of this

git config --global commit.gpgsign true

Which would affect all git use

## do this:

git config --local commit.gpgsign true

which will only affect the local repository.

Security - Preliminary Overview
===============================

As for signing and merging, it is my opinion that we should not squash commits on a merge so we can preserve signatures, and also sign tags. As for the security of the release process, what we need to do is decide upon a "Release Branch" under a shared namespace which supports maintainer, developer, and owner roles, i.e. the master branch on the "production" i2p-developers group.

This branch is the "canonical" i2p, we don't release anything unless it's merged here, and most crucially, this is the common base that we tell new developers to fork and the version we generate the git bundle from. As for security on this repository, I think the problem becomes how we decide on what goes in. Gitlab has options for that, we can, for instance, require peer review on any merge request into the i2p-developers/i2p.i2p namespace. This does *not* apply to namespaces forked by users, and shouldn't in my opinion.

More to come soon.

Sun, 12 Apr 2020, 07:05pm #20
idk
Contributor

Let me know your thoughts on these T&C for Git before I make them official.

Terms and Conditions for http://git.idk.i2p
===========================================

This is a work-in-progress document. It is not yet finalized. It will be soon.

Privacy Policy
--------------

git.idk.i2p and it's administrators will never share any information with any
third parties, except as required by law in the jurisdictions where it is
hosted(USA and Canada). Logs are kept for 24 hours only, and are never shared
with a third party. Account information given by the user may be visible on a
public profile, it is the user's responsibility to audit the information that
they share publicly.

Acceptance
----------

Accounts are open to the public. They approved by an admin on a case-by-case
basis. While real names and real e-mail addresses are not required, if we
suspect your account of being malicious in any way, it may never be approved,
a right which we reserve in perpetuity. Once you account is approved, the
following conditions apply.

If you cannot abide the terms and conditions below, or simply don't trust me to
host your repositories responsibly, please [consider hosting your own Gitlab
service](https://geti2p.net/en/blog/post/2020/03/16/gitl...).

Abuse
-----

Attempting to use git.idk.i2p to spam, communicate with or distribute malware,
or publicize and distribute materials that are illegal in the US or Canada are
strictly prohibited. Accounts found to be abusing the service to do this will
be deactivated permanently.

Files that may infringe the copyright of a specific organization will be
reviewed on a case-by-case basis and files or repositories will be removed if
the violation actually occurred.

Attempting to attack git.idk.i2p, including but not limited to by resource
wasting/exhaustion, user impersonation, defamatory tactics against the project
or other developers, attempts to exploit underlying git or SSH services, or
attempts to de-anonymize the service are prohibited. Accounts found to be
mounting attacks will be deactivated.

git.idk.i2p services are provided free of charge by idk for use by the I2P
project and by the I2P community.

Mon, 13 Apr 2020, 04:30pm #21
zzz
Administrator
Zzz

it's -> its

If it were me I'd go even further, but it depends how much resources you're throwing at this. Some ideas:

- primarily for i2p-related projects of reasonable size, e.g. don't fork the linux kernel here, don't use it as an image host, etc.

- language about resource limitations (project size, number of projects, file size, etc.) even if it isn't an attack, things could be deleted, keep your own backups