Re: Embedding a REPL in a Java application

[Jamison Hope <jrh at theptrgroup dot com>]

On Nov 18, 2009, at 12:48 PM, I wrote:

> On Nov 18, 2009, at 2:15 AM, Per Bothner wrote:
>
> > Not sure what's going on.  What happens with a shared repl is that
> > it creates a (thread-specific) environment, and then sets that to
> > share (indirect to) the parent environment.  I suspect the problem is
> > that the "parent environment" in our case ends up being something
> > unsuitable (for a reason I'm not clear on).
>
> That's (more or less) what I figured. I'm still trying to grok what  
> exactly goes on behind the scenes with repls and Environments and  
> Threads (oh my!).
>
> > Perhaps creating a fresh environment before you start, and then
> > passing that for each shared REPL would work.  Right now I'm a
> > bit swamped with my day job, but I'll try to look at it when
> > I get a chance.
>
> OK, thanks, I know how that is. Luckily(?) for me, this *is* my day  
> job right now (well, part of it, anyway). If I make any discoveries  
> I'll be sure to post.

Success! I've discovered a fix.

First, I found (via jdb) that I was (sometimes) getting a  
NullPointerException. When this happened, the GUI remained responsive,  
but the ReplPane behaved like a vanilla JTextPane: no Kawa prompt, no  
evaluation in response to an 'enter' key press, just a blinking cursor.
> Exception occurred: java.lang.NullPointerException (to be caught at:  
> gnu.mapping.RunnableClosure.run(), line=107  
> bci=91)"thread=Thread-6", kawa.Shell.getOutputConsumer(), line=126  
> bci=20
>
> Thread-6[1] where
>   [1] kawa.Shell.getOutputConsumer (Shell.java:126)
>   [2] kawa.Shell.run (Shell.java:181)
>   [3] kawa.Shell.run (Shell.java:165)
>   [4] mil.navy.nrl.apps.util.scheme.JRHReplDocument$2.apply0  
> (JRHReplDocument.java:109)
>   [5] gnu.mapping.RunnableClosure.run (RunnableClosure.java:105)
>   [6] gnu.mapping.Future.run (Future.java:40)


Shell.java:126 says:
>       return Language.getDefaultLanguage().getOutputConsumer(out);

so a NPE must mean that Language.getDefaultLanguage() returned null.  
This is despite the fact that I'm calling Scheme.registerEnvironment()  
well before constructing my ReplDocument. So the first step of my  
solution was to make sure that the language is set on the Future  
thread, by changing the ReplDocument constructor's anonymous repl to  
look like this:
>     thread = new Future( new kawa.repl(language) {
>         public Object apply0 () {
>           Language.setDefaults( language ); // THIS IS TO AVOID A NPE
>           Shell.run( language, Environment.getCurrent() );
>           SwingUtilities.invokeLater( new Runnable() {
>               public void run() {
>                 JRHReplDocument.this.fireDocumentClosed();
>               }});
>           return Values.empty;
>         }
>       }, penvironment, in_p, out_stream, err_stream );

That reliably got rid of the NPE. Now the ReplPane looked like a  
ReplPane, albeit one with lots of problems: instead of the Kawa  
prompt, the whole text area was full of red stack traces, and the GUI  
remained unresponsive until I got an OutOfMemoryError (that's a lot of  
exceptions!). The stack traces mentioned an unbound location, so next  
I caught a gnu.mapping.UnboundLocationException:
> Exception occurred: gnu.mapping.UnboundLocationException (to be  
> caught at: kawa.Shell.run(), line=280 bci=305)"thread=Thread-6",  
> gnu.mapping.Location.get(), line=67 bci=23
>
> Thread-6[1] where
>   [1] gnu.mapping.Location.get (Location.java:67)
>   [2] gnu.expr.Compilation.getCurrent (Compilation.java:2,855)
>   [3] gnu.kawa.lispexpr.LispLanguage.parse (LispLanguage.java:51)
>   [4] gnu.expr.Language.parse (Language.java:503)
>   [5] kawa.Shell.run (Shell.java:244)
>   [6] kawa.Shell.run (Shell.java:184)
>   [7] kawa.Shell.run (Shell.java:165)
>   [8] mil.navy.nrl.apps.util.scheme.JRHReplDocument$2.apply0  
> (JRHReplDocument.java:109)
>   [9] gnu.mapping.RunnableClosure.run (RunnableClosure.java:105)
>   [10] gnu.mapping.Future.run (Future.java:40)


That shows LispLanguage choking on this line:
>     Compilation save_comp = Compilation.getCurrent();
Apparently, Compilation.setCurrent() had not yet been called, and so  
that Location was unbound. Since save_comp is only used to restore the  
state after parse() finishes its work, the value didn't strike me as  
being very important. So, I made sure to call Compilation.setCurrent()  
before Shell.run():
>     thread = new Future( new kawa.repl(language) {
>         public Object apply0 () {
>           Language.setDefaults( language ); // <- otherwise,  
> occasional NPE
>           Compilation.setCurrent( null );   // <- otherwise,  
> consistent ULE
>           Shell.run( language, Environment.getCurrent() );
>           SwingUtilities.invokeLater( new Runnable() {
>               public void run() {
>                 JRHReplDocument.this.fireDocumentClosed();
>               }});
>           return Values.empty;
>         }
>       }, penvironment, in_p, out_stream, err_stream );


Now, I get a nice happy green Kawa prompt in each REPL window, and if  
I evaluate "(define x 3)" in one window, I see x bound to 3 in the  
other. Yay.

So altogether, I made three edits to ReplDocument:
(1) I replaced the use of SwingContent and CharBuffer by GapContent  
and String (in a way which I'm sure is terribly inefficient).
(2) I call Language.setDefaults() on the repl thread before calling  
Shell.run().
(3) I call Compilation.setCurrent() on the repl thread before calling  
Shell.run().

Now that it works, I went back and reverted #1, and everything still  
works. So it appears that the real fix was #2 and #3. There's  
definitely a race condition somewhere, because if I take out #2, it  
sometimes works and sometimes fails. It always fails without #3.  
Hopefully, Per, the fact that this fix works will help you track down  
the underlying bug. I wouldn't know where to begin to look.


Whew!


-Jamie

--
Jamison Hope
The PTR Group
www.theptrgroup.com