[repoman] first db and tool syntax work

[Jason Chu]

I didn't get the original mail, so I'm replying to Dan's email.

On Tue, Jul 10, 2007 at 11:57:28PM -0400, Dan McGee wrote:
> On 7/7/07, Paul Mattal <paul at mattal.com> wrote:
> > So here's a first shot at the DB model:
> >
> >    https://subzero.elys.com/svn/repoman/trunk/repoman/repos/models.py
> 
> I don't do a whole lot of work with databases, so looks good to me. :)
> 
> If we are going to use sha1sums (which seems fine to me for now until
> we get into any GPG stuff or something), then you can cut that
> signature field down to size 40 (or 41 if you need the null
> character).

If you plan on enforcing dependencies (like you say below), they should
probably be tracked in the db too...

> > And here's my first thoughts on what the syntax of the tool would look like.
> >
> >    https://subzero.elys.com/svn/repoman/trunk/doc/REPOMAN.txt
> >
> > Of these, the second seems more tentative to me right now. There could
> > be a neater way to do this. I was thinking about if we should make the
> > commandline options as pacman-like as possible or not. I have spec'd
> > this out doing more word operation options, like you encounter in SCMs
> > commandline tools.
> 
> I like SCM-like operations. They are rarely ambiguous, unlike the -c
> flag in pacman which has two drastically different meanings.

I also like the SCM-like operations.

As a side note, what about things like moving packages from one repo to
another?  Is that just an add and a remove?

> > Happy to hear thoughts on these!
> >
> > Some things I thought about in process:
> >
> > 1) The repoman CLI can act as upload client and as management client. It
> > would be nice to just have the client authenticate and talk to the DB
> > directly for making updates. I think most DBs like postgres and mysql
> > offer a way to do encrypted connections directly to them, and hopefully
> > these mechanisms of connection are supported through the Django and
> > Python DB abstraction layers.
> 
> If we can do something like this, or at the very least have a minimal
> server program that doesn't always need to be running and can be
> communicated with through some secure channel, that would be good.

I dislike direct connections to the db.  I much prefer
daemons/scripts/abstraction layers because dbs change and aren't renouned
for their network security.

Something has to do updates to the filesystem on the repoman server anyway.

> > 2) The commit step gets separated from the upload step so you can do a
> > bunch of work and then upload all the new packages at once via the most
> > efficient transport. Thanks to package signatures, when you choose to
> > put the same binary package into another repo, the server can tell it
> > already has that package and doesn't require you to re-upload it!

How exactly will repoman commit if multiple SCMs are supported?  A commit
in SVN is very much different from a commit in git (especially because the
git commit isn't visible on any remote servers right away).

I'm not sure if repoman can handle this or if other scripts that talk to
repoman would (one for SVN layout 1, one for git layout 1, one for git
layout 2, etc).  Some way to get the commit to some visible place, figure
out how to globally refer to that place, and tell repoman.

When it was first explained to me, I really liked the idea of not depending
on an SCM layout or type.  If I wanted to track my packages in a flat
directory structure, I think repoman should be able to handle that.

> What about some kind of field in the BinaryPackage table that
> indicates how the package was/is built? This could perhaps signal the
> server to accept a commit without a sha1sum at commit time. The
> autobuilder would then fill this data in when it finished the build.
> This field could also be used to enforce testing of automatically
> built packages, perhaps.
> 
> > 3) There needs to be a custom server component of some sort anyway in
> > order to handle updating the binary repositories themselves on disk and
> > rebuilding the db files. So more than ever, I'm resolved to write that
> > server component properly with SSL connection to the client and have it
> > handle the file transfer step, too. I'll probably start with the
> > codebase from the AUR tupkgs server.
> 
> +1, and probably not bad to at least glance at the old codebase. WRT
> reuse- unless it is exactly what you want, don't try to meld it into
> shape- just rewrite it. (/me kicks himself for not doing this with
> libalpm)

I'm not sure if this was covered by the ssh uploading from before, but I
really like the idea of repoman just using the same ssh connection/config
and calling scripts on the repoman server.  Then there's no SSL to worry
about, no custom network protocol, etc.

> > 4) Repos can depend on other repos. This means that it can only have
> > dependencies in the same repo or in repos in its dependency tree. So
> > extra depends on current, and this keeps you from putting packages into
> > current that depend on packages in extra.
> 
> How do you plan on enforcing this? And what if makedepends are allowed
> to break the rule, but not depends?

Like I said above, the only way to enforce this in the long run is to track
depends and makedepends in the db.  It sucks, but there's no other way to
know for sure if some operation (say, delete packageA) will cause a
dependency problem.

Jason
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://archlinux.org/pipermail/repoman/attachments/20070710/f83e71d0/attachment.bin