Re: multidimensional optimization

[Gerard Jungman <jungman at lanl dot gov>]

Fabrice Rossi wrote:
>
> So, we need at least something like:
> struct gsl_vec_function_struct
> {
>   double (* function) (const gsl_vector * x, void * params);
>   void * params;
> };
> Maybe we can avoid using gsl_vector, but then, what is the point in having
> vectors?

It's tricky of course. Some people will want to be completely decoupled
from gsl_vector, if all they are interested in is the optimization
library. As usual, we probably need both; but that is a trivial
addition.
 

> But for neural networks, we should focus on optimization techniques that use
> the gradient, so we need another function in the struct, for instance:
> void (* gradient) (const gsl_vector * x,gsl_vector *gradient, void * params);

It should be arranged parallel to the one-dimensional functions:

  Existing 1D Objects       Multi-Dim Objects
   gsl_function       <-->   gsl_function_vec
   gsl_function_fdf   <-->   gsl_function_grad_vec

or some such names...


> Having gsl_vector* and not gsl_vector** is deliberate. For many functions,
> asking the vector to be preallocated is a good idea, because reallocating the
> vector at each iteration is a waste of time, and is not regligeable compared
> to the time needed to compute the gradient itself.

Absolutely. Ideally we like to push all memory management functionality
to the highest reasonable level in the API; otherwise we would have to
use
tricks which would be too hard to maintain in plain C (although
straightforward in the C++ world... sigh).


> For gradient based optimization, we will need to make onedimensional
> optimization, which is very easy thanks to the params member of
> gsl_function_struct. I think therefore that implementing toy version of a
> simple conjugate gradient can be quite easy.

Yes, I think so too. It's just a question of writing and testing the
implementation. It should be high quality... Numerical Recipes attitudes
must be checked at the door.


> Is there something wrong in this design?

Probably not. 

-- 
G. Jungman