Re: Source retrieval

[Enrico Weigelt <weigelt at metux dot de>]

* Yann E. MORIN <yann.morin.1998@anciens.enib.fr> wrote:

Hi,

> The idea sounds interesting...

:)

> > Example: 
> > http://sourcefarm.metux.de/robot/query-download-url/?package=binutils&version=2.18.0.0&encoding=tar.gz
> 
> Well, here are a few suggestions:
>  - default is to redirect to a randomly-picked URL from the list of mirrors

It already does :)
If you only want the URL instead of downloading, just do a raw HTTP request 
(w/o following the redirect) and parse the reply header.

I'm also planning another bot which prints out a list of URLs.
(eg. if you want to walk through the whole list, in case of URLs fail)

>  - allow retrieval of the canonical URL

The http redirector provides that canonical URL. 
(maybe it could look a bit more convenient ;-o)

Clients that are interested in an canonical URL should only use this one,
and the actual locations behind are out of scope.

>  - allow retrieval of the MD5sum for the package

Yep, already prepared that. 
But: the scanner needs some way to get that MD5. Most projects provide them themselves,
so we'd first need to download all the tarballs (-> *HUGE* traffic :().

>  - allow retrieval of the snapshots or pre-releases

Ugh, that's one thing I explicitly left out when designing CSDB - it should only
contain releases. But snaphots aren't releases and don't really fit into the 
normalized versionspace. An possible solution could be giving them separate 
package names (eg. w/ special prefix/suffix).

> Which would give the following API (only a suggestion):
> 
> http://..../query-download-url/ : retrieve a package tarball. Parameters:
>  - package, mandatory: the package name
>  - version, mandatory: the package version
>  - encoding, mandatory, the type of extension to retrieve, from the following
>    list: {tar,tar.gz,tar.bz2} (others?)

Actually, it's not the filename suffix, but the encoding. By convention there's
exactly one name per encoding, eg. for *.tbz2 archives "tar.bz2" is used.

>  - canon, optional: if not set, or set to 'no', will choose a URL amongst the
>    mirrors; if set to 'yes', will redirect to the canonical URL only; if set
>    to anything other than 'yes' or 'no', will invalidate the query.

I'd prefer to treat all mirrors equal (perhaps with an optionally weight).
Maybe we could add some geographical/topological information, so more intelligent
clients can choose the nearest mirror.

> New URL:
> http://..../query-md5-url/ : retrieve a package MD5 sum. Parameters:
>  - the sames as for the above, having the same meaning.

ACK. That would be fine (as long as we *have* these information).

> > I could mirror it on some of my boxes and add a record to CSDB :)
> 
> I guess you and I have the same problem. If our mirrors get used, then the
> bandwidth usage will rise to the point where we'd have almost no bandwidth
> left for our own use...

Right, so we should get more mirrors.
I'll try to aquire some spare resources from some of my customers.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

--
For unsubscribe information see http://sourceware.org/lists.html#faq