Skip to content

CHill's Ramblings

Thoughts, ideas, and other such things I feel like putting here. About basically any topic.

Serializing compiled Jython code objects (works on Google AppsEngine for Java)

Posted on August 11, 2010August 2, 2021 By darkhelm 1 Comment on Serializing compiled Jython code objects (works on Google AppsEngine for Java)
0 0
Read Time:4 Minute, 11 Second

In the process of working on the ShowSort project, and updating it to some of the more recent API changes for Google Web Toolkit (GWT) and Google AppEngine for Java (GAE/J), I ran into a dilemma — I wanted the sorting algorithm processing to be handled through a series of Task Queue API tasks, and I wanted to break apart the tasks in such a way to provide the largest amount of time for a sorting algorithm to run — which meant splitting up the actual execution of the sorting algorithm from the compiling of the sorting algorithm into bytecodes.

Well, as I was first using the JSR-223 implementation of Java scripting support (just for fun), I would make a compiled script using the Compilable interface in one step, save it to the memcache, and then stick a new task on the queue to execute the compiled script. Much to my surprise, this didn’t work — because the compiled script created by Jython isn’t Serializable (a requirement for the memcache to work).

So, my next attempt was to use the Jython classes directly, specifically the PythonInterpreter class, and compile my script in the first step (task), save to memcache, execute on the next step (task). Once again, it failed to work, despite the PyCode object claiming to be Serializable, it wasn’t able to be serialized, because some of the components of the object did not implement Serializable, specifically since the actual compiled object is of type PyTableCode, which inherits from PyBaseCode, which has a few elements that are not Serializable (like CompilerFlags for example).

Stumped again, I began googling for help (like anyone would do), and got to a lot of dead ends. Some people tried XML Serialization (which the prime Java one to use, XStream, doesn’t work out of the box for GAE/J), others offered making secondary version of objects, inherited from the original, non-serializable objects, and constructing some elaborate conversion from a PyTableCode object into something home-brewed that was serializable… but that was a bit crazy.

My final idea, which actually worked, was to take a look at the Jython source code. Wandering a while in the source, I stumbled upon a class called: org.python.modules._py_compile — the actual class that is used within Jython to do compiler operations. From that class, I derived a process that when it compiles, it doesn’t create the PyTableCode object yet, instead it gives me a byte[] array — which is very serializable. I can store the byte[] array (which is the compiled bytecodes of my script from the Jython compiler) into my object, am able to serialize the object, and then when I want to execute my code, I use another Jython object’s method, called  org.python.core.BytecodeLoader.makeCode() to construct the PyTableCode object needed to be executed, using the byte[] array I had saved. The bulk of the time for compilation is still in the compile step, the only thing I can see the makeCode() method doing is wrapping it into something Jython’s PythonInterpreter class can use. The final results of my attempt are below:

package org.darkhelm.showsort.server;

import java.io.ByteArrayOutputStream;
import java.io.Serializable;
import java.io.StringReader;
import java.util.Arrays;

import java.util.logging.Level;
import java.util.logging.Logger;

import org.python.compiler.Module;

import org.python.core.BytecodeLoader;
import org.python.core.CompileMode;
import org.python.core.CompilerFlags;
import org.python.core.ParserFacade;
import org.python.core.PyCode;

import org.python.util.PythonInterpreter;

/**
*
* @author cliffhill
*/
public class Code implements Serializable {

private static Logger log = Logger.getLogger("ShowSort.Code");
private static final String VAR_NAME = "array";
private static final String LIB_PYTHON_PATH = "WEB-INF/lib-python/Lib.zip";
private static final String FILENAME = "<script>";
private static final String NAME = "sort$py";

private String code;
private byte[] compiledCode;

public Code() {
this(null);
}

public Code(String code) {
setCode(code);
}

public String getCode() {
return code;
}

public void setCode(String code) {
this.code = code;
this.compiledCode = null;
}

public void compile() {
log.finest("Code to be compiled:n" + code);
String realCode = "import sysnnsys.path.insert(0, '" + LIB_PYTHON_PATH + "')nn" + code + "nsort(" + VAR_NAME + ")";
log.finest("Real Code to be compiled:n" + realCode);

StringReader codeReader = new StringReader(realCode);
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
try {
org.python.antlr.base.mod node;
try {
node = ParserFacade.parse(codeReader, CompileMode.exec, FILENAME, new CompilerFlags());
} finally {
codeReader.close();
}
Module.compile(node, byteStream, NAME, FILENAME, true, false, null, org.python.core.imp.NO_MTIME);
compiledCode = byteStream.toByteArray();

log.finest("Compiled code:n" + Arrays.toString(compiledCode));
} catch(Throwable t) {
log.log(Level.SEVERE, null, t);
}
}

public void exec(Array array) {
PythonInterpreter pi = new PythonInterpreter();

log.finest("Interpreter to use: " + pi);

log.finest("setting local variable '" + VAR_NAME + "' to: " + array);

pi.set(VAR_NAME, array);

PyCode execCode = BytecodeLoader.makeCode(NAME, compiledCode, FILENAME);

log.finest("Executable compiled code: " + execCode);

pi.exec(execCode);
}
}

I hope this helps someone else who has dealt with the limitations of the non-serializable Jython compiled script objects.

Share

Facebook
Twitter
Pinterest
LinkedIn

About Post Author

darkhelm

chill@darkhelm.org
http://darkhelm.org
Happy
Happy
0 0 %
Sad
Sad
0 0 %
Excited
Excited
0 0 %
Sleepy
Sleepy
0 0 %
Angry
Angry
0 0 %
Surprise
Surprise
0 0 %
Uncategorized

Post navigation

Previous Post: The Return of ShowSort.
Next Post: SyntaxHighlighter Widget for GWT 2.0

Average Rating

5 Star
0%
4 Star
0%
3 Star
0%
2 Star
0%
1 Star
0%
(Add your review)

One thought on “Serializing compiled Jython code objects (works on Google AppsEngine for Java)”

  1. Quasi says:
    November 23, 2012 at 9:13 am

    I found this post very useful, since i want to load compiled Jython code from a Database. For that i adjusted this code slightly to have conversion functions from PyCode to byte[] and back. Only the meaning of the FILENAME and NAME is not 100% clear to me.
    Thanks,
    Enrico

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Archives

  • October 2021
  • August 2021
  • October 2019
  • November 2018
  • October 2016
  • September 2016
  • November 2015
  • September 2013
  • May 2013
  • October 2012
  • April 2012
  • March 2012
  • December 2010
  • November 2010
  • September 2010
  • August 2010
  • July 2010
  • January 2010

Categories

  • america
  • bitsy
  • blueprints
  • ejb
  • glassfish
  • gwt-syntaxhighlighter
  • jpa
  • jython
  • lies
  • network
  • politics
  • Uncategorized

Recent Posts

  • To Dave Hines, my friend, may you now have the joy you had spread to everyone around you.
  • Router Fun
  • False Peace
  • Moving away from the google universe.
  • The problem with people abusing scripture to attack entertainment

Recent Comments

  1. darkhelm on To Dave Hines, my friend, may you now have the joy you had spread to everyone around you.
  2. Matt Sutton on To Dave Hines, my friend, may you now have the joy you had spread to everyone around you.
  3. Unknown on Jonah, Jonah, did not obey God immediately…
  4. 1seanv on A Christian’s Response To: A Christian’s View of World of Warcraft (published in 2008)
  5. Unknown on Jonah, Jonah, did not obey God immediately…

Copyright © 2023 CHill's Ramblings.

Powered by PressBook WordPress theme