Re: package management

On Mon, Dec 17 2007, Maciek Pasternacki wrote:

On Fri, 14 Dec 2007 12:47:41 +0200
Kamen T <ka...@xxxxxxxxxxx> wrote:

2. CVS (and similar) packages;

Fetching packages from version control systems (CVS, SVN, and
darcs are supported, plugins for other systems are possible. I
am considering at least Bazaar-NG, Mercurial, and git. Arch/baz
will not be supported, unless someone else writes and tests the
code -- I can't handle those, I need a tutorial to simply check
out the code).

IMO the version control systems should be the most important source
of software for cl-librarian. I'd also give higher priority to git.

They are indeed. Tarballs need to be supported anyway, since many
important libraries (e.g. Edi Weitz's software) don't publish source
in version control systems, but main reason was to automate updating
libraries from VC.

loliveira supports darcs repos of Edi's software. They can be found
here: http://common-lisp.net/~loliveira/ediware/

Git is definitely on TODO and I will note it as higher priority.

3. Packages installed using the OS package management;

No support.

That's fine. There'd be no modules (SMH suggested that we do not
use the word packages and he's right) that couldn't be found on a
version control system. The only issue here is dependencies. OS
maintainers who deal with dependancies of CL modules usually make
sure that a release is a bunch of working software and the question
is how cl-librarian could benefit from all that work. Perhaps when
you get to a point when cl-librarian is well documented and
likable, the OS maintainers would support only the dependancies of
cl-librarian only ;) as it would take care of everything else.

I don't think so. Librarian is meant to create complete
self-contained deployment description rather than compose with OS.

It seems you misunderstood me. I do not find it a good idea OS
maintainers to support CL libs directly. I'd rather have an
architecture where that work is delegated to a module management
system (potentially Librarian).

I have plenty of issues. I update a package to the latest
version to find out that I had to update another package that
had higher priority.

Central concept in Librarian is a `library shelf', which is a set
of libraries. Such shelf should be updated at once.

I checked the web page and it didn't get very clear to me, so could
you please elaborate on that? Does it mean one specifies a set of
modules and cl-librarian takes care to find, obtain, install,
update (when commanded) them? So what's the idea of multiple
shelfs?

Idea of multiple shelfs comes from some of my usage patterns: single
instance of Librarian manages multiple shelves used by multiple
projects. And, shelf can include other shelves -- that means that
in Web project using Hunchentoot and Postmodern, I don't have to
copy-paste all Hunchentoot and Postmodern dependencies, I just
include shelves HUNCHENTOOT and POSTMODERN (both are included as
example shelves in Librarian's repo):

(defshelf project (postmodern hunchentoot)
((cl-json darcs-repo :source "http://common-lisp.net/project/cl-json/darcs/cl-json";)
(parenscript darcs-repo :source "http://common-lisp.net/project/ucw/repos/parenscript";)))

Hmm, I still don't understand. I have simple questions. Let me summarize:

project: name of the shelf
postmodern, hunchentoot: dependencies
cl-json, parenscript: what are they in your terminology?

What's more cl-json depends on Parenscript and I don't see how is that expressed?

Nevertheless, the `project' shelf directory contains *all* libraries
and modules, so that if I update nearby shelf to test
latest-and-greatest experimental Postmodern patches, I don't risk
destabilizing the Project. Only when I operate on Project itself,
its contents are touched.

I just thought of other use of multiple shelves: when updating a
shelf, I can ask Librarian to update only subset of it: single
specific repo (if I actually want to ask for trouble), or only
contents of included shelf (so that I can, e.g., update Hunchentoot
without updating Postmodern).

I will probably work on Librarian's documentation during Christmas.
In meantime, feel free to ask (here on the group, or to private
mail, my name at my surname dot net will get fastest reaction).

Thanks, I can wait for the documentation.

Generally, when installing software, a management system should
download the latest and greatest software by default. However, in
certain cases (when the automatic unit-tests fail or when commanded
by the user) the system should communicate the problem with the
user so that s/he can take action like asking it to install an
explicitly chosen older version.

I didn't think of tests yet. Testing downloaded modules can be
good, as far as they support a single protocol (ASDF defines
TEST-OP, this should be fine). Definitely for the TODO.

Hmm... if I could think of easy and cheap way of reversible
upgrades, that would be great for checking out new releases. As for
now, I don't include final application code in the shelf (I just use
the shelf before loading application's .asd file), but if I put
application inside shelf and only started it up from outside, this
would be great way to deploy upgrades.

Gentoo's Portage supports "multiple slots" for that purpose. So you
can have two packages of the same software - for example Emacs-21 and
Emacs-22. By default Portage installs the latest version of a
package. However, if the user wishes to, he can install an older (or
newer unstable) version of the package in a different "slot".

I'd suggest that you investigate the Gentoo package management system
that is called Portage. It has some very powerful ideas. That's its
handbook:

http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1#doc_chap2

I'll take look at it when I have a moment. Probably not before
Christmas, though.

If you want to freeze some package, use tarball (TODO: specify
branches/tags of repository).

Using branches/tags seems much more important than using tarballs
to me.

For time being: for two out of three supported version control
systems (SVN and darcs), branches are included in repository path,
so they are supported. As for tags, you can just make a tarball of
any given tag and point Librarian to a tarball in local filesystem.
Yes, it is a kludge, and VC tags and CVS branches will be supported
-- I just said how you can work around it until then.

There is no actual support for updating yet, but it should not be
much work.

Installing new machine is a lot of work.

Not anymore ;) I think I will put up a skeleton project out of my
first serious Librarian deployment as a usage example.

Splendid!

Deployment skeleton is now in skel/ directory of Librarian
repository.

Sometimes the software works on one of the machines and not on
the other.

I didn't try to solve any portability issues (that's one more
reason to just ignore distribution-provided Lisp libraries), and
I have no idea how a Librarian could be useful for this. I just
use compatibility layers, like usocket, to handle differences.

I meant something different, e.g. when a package from the OS
management tool takes precedence over a package from asdf-install
or vice-versa.

Paths of shelved ASDF systems are PUSHed at beginning of
ASDF:*CENTRAL-REGISTRY*, so they should have priority over
system-provided ones. However, packages supplied by OS and not
downloaded by Librarian are visible. We try not to interact with
OS-provided libraries, but if user uses apt-get, ASDF-Install, and
Librarian at once with the same libraries, he is asking for trouble
;)

What about the use case multiple applications to run in the same Lisp
process? Although it is a remote possibility I still find it
possible. How will an application find the way to its libraries in the
*central-registry*?

Do you think Librarian should detect if library it is about to
download (or enable already downloaded one) is already available in
the system? I personally try to avoid using any OS-supplied
libraries, simply because they too often become a pain when
something goes wrong, so I don't have much experience in resolving
such conflicts.

I'd rather not allow anything (OS or else) mess with my
*central-registry* except my library management system. I'd reset the
registry and completely ignore the OS-supplied libraries.

I also have problems with old FASL files.

I haven't thought of this yet, one more thing to the TODO section.
It will certainly be useful.

One more thing: Librarian works only with ASDF systems. (LOAD
"load.lisp") style packages are not and will not be supported. You
can work around this by creating fake .asd files that loads
load.lisp. MK-Defsystem is also not supported (unless someone helps
me with supporting it -- I didn't ever need to use it and I don't
suppose I will).

Stelian Ionescu, the guy who currently maintains the Gentoo
distribution told me that they've decided to use asdf-binary-locations
instead of common-lisp-controller as the latter is not actively
supported. I plan to investigate the issues with the FASL files today.

Oh, ASDF-binary-locations can be great addition to Librarian. I don't like
to stuff it with dependencies, but this one may be worth it: don't put
compiled FASL files into repositories, only in bin/ dir that can be just
simply removed, and which can have subdirectories for different Lisp
implementations... yes, I think I like it :)

A side question: why cl-librarian is targeted to power users? Does it
mean it would be hard to use by general users?

At this moment, certainly, it is work in progress. Finally there may be easy
way to use it, but I didn't mean `single-click ready Lisp installation for
newbies', but `simplify deployment for Lisp developers that already know
their way around ASDF', and they are my secondary target (my primary target
is specifically myself). If it becomes easy to use by general users, or even
newbies, I won't oppose, but only if it won't require making it less useful
for power users.

--
Kamen
.

Relevant Pages

  • Re: package management
    ... Fetching packages from version control systems (CVS, SVN, and darcs ... No support. ... Librarian is meant to create complete self-contained ... Such shelf should be updated at once. ...
    (comp.lang.lisp)
  • Re: package management
    ... CL Librarian can fetch packages from ASDF-Install (no GPG signature checking ... I don't consider support in Librarian, ... Using .asd files from packages' subdirectories is ...
    (comp.lang.lisp)
  • Re: package management
    ... CL Librarian can fetch packages from ASDF-Install (no GPG signature ... No support. ... I don't consider support in Librarian, ... distribution has its own system for package management and one ...
    (comp.lang.lisp)
  • Re: package management
    ... The only issue here is dependencies. ... Librarian is meant to create complete ... Such shelf should be updated at once. ... I have on TODO to make Librarian do test loading of shelf's packages after ...
    (comp.lang.lisp)
  • Re: Anybody Tried Novell Desktop Yet?
    ... who will pay out for the support ... > upgrades for NLD as a default. ... Which is a place where users are not normally allowed to install their own ... software or upgrade packages themselves; it is all done in a controlled ...
    (alt.os.linux.suse)