[repoman] transport mechanism

[Jason Chu]

On Mon, Jul 09, 2007 at 02:04:04PM +0100, Essien Ita Essien wrote:
> Paul Mattal wrote:
> > One aspect of this project that has bugged me more than any other: the 
> > transport mechanism. People building binary packages need to 
> > authenticate themselves to the server and upload their built packages.
> > 
> > What has been used until now?
> > 
> > Developers currently use scp as the transport; we all have accounts on 
> > the development machine and devtools automatically runs scp to copy 
> > files up to the server.
> > 
> > Upsides: existing, effective, secure, easy to automate using keys so you 
> > don't have to log in each time
> > 
> > Downsides: you need to have a system account on the machine, something 
> > you don't always want, and keeps you from deploying the system multiple 
> > times on the same machine
> > 
> > TUs currently use a custom python client-server for upload.
> > 
> > Upsides: don't need a system account, easy to automate by storing 
> > password locally on machine
> > 
> > Downsides: not mature, nonstandard, insecure (not SSL encrypted), 
> > requires running another/a less-scrutinized server on a box
> > 
> > Other options: We could write another custom client/server, with SSL. If 
> > we want to hook into something existing, we could try to use apache 
> > WebDAV or http upload with SSL. We could consider sfs and just do a cp 
> > to the remote filesystem.
> > 
> > On the whole, I think the custom client/server is still winning for me, 
> > at the moment, though I wish support for SSL servers in Python was 
> > better. But I really feel like there should be something standard we 
> > could use for this authentication and transport.
> > 
> > Any ideas?

The new one I learned was using ssh from a chroot.  That's how repo.or.cz
works and that's how phrakture planned on doing projects.archlinux.org.
That way "developers" can have accounts but they don't need shells on the
hosting system.  You can even limit the commands they can execute using
something like rsh.

> The first question that occurs to me is: "Why Transport At All?"
> 
> Basically, once we have a properly specified PKGBUILD and related files, 
> we can supposedly rebuild the package just like the developer did on 
> their box.
> 
> (umm... keep remembering my initial bias based on integration :P), but I 
> think this is a superior solution to actually uploading *built* 
> files...the ONLY problem is that the server will spend time BUILDING 
> stuff... so we could do this:
> 
> Have one server to BUILD, and another server to HOST.
> 
> Upside: No need to have account on server to upload stuff... just 
> check-in to the SCM and boom.
> 
> Downsides: This is actually both Integration and Hosting, not just 
> Hosting now... so is integration something that the repoman design would 
> want to cater for?

I think even if we do have automated integration, we will still need to be
able to build/upload some packages manually.  Not all packages can be built
automatically.

Not only that but automation is really hard, we're going to have to go
manually for a while till we get things right ;)

> > One thought that just occurred to me.. we could take a page out of the 
> > monotone trick book and separate trust from transport. We could add a 
> > file to the PKGBUILD directory which gets stored in the SCM and contains 
> > the sha1sum of the BINARY package that corresponds to this PKGBUILD 
> > built for a particular architecture. Then we don't need to care where or 
> > who a binary package comes from.. anyone can upload the binary package 
> > to the server (without authentication) as long as the package has the 
> > proper sha1sum. This also has the benefit of keeping developers from 
> > accidentally uploading the wrong binary package.

As we talked about in the dev meeting, the problem with this is that
two compiles of the same package will have different sha1sums.  This is
because of timestamps within the tar, the order of files within the tar,
the contents of the binaries (I'm pretty sure linked libraries aren't
ordered, for example.

Saying that one PKGBUILD creates a package with a sha1sum of X means we
can't autobuild at all.

> > I guess one problem with this is that someone can still DoS your server 
> > by uploading lots of bogus binary packages, that while discarded as 
> > fakes, still take up server bandwidth. But is there something like this 
> > we can do here?

Jason
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://archlinux.org/pipermail/repoman/attachments/20070709/523a1b96/attachment.bin