Showing posts with label msgpack. Show all posts
Showing posts with label msgpack. Show all posts

Thursday, February 6, 2014

multithreading MessagePack usage without java PermGen memory error


If you use new MessagePack() too many time, perm gem memory error occurs soon.

MessagePack msgpack = new MessagePack();
byte[] bytes = msgpack.write(object);

You can test this with
-XX:+TraceClassLoading -XX:+TraceClassUnloading
VM parameters or JConsole.

[Loaded learningtest.msgpack.MessagePackTest$MyMessage_$$_Template_1527741453_0 from __JVM_DefineClass__]
MyMessage [name=msgpack, version=0.6]
[Loaded learningtest.msgpack.MessagePackTest$MyMessage_$$_Template_831556519_0 from __JVM_DefineClass__]
MyMessage [name=msgpack, version=0.6]
[Loaded learningtest.msgpack.MessagePackTest$MyMessage_$$_Template_1872305359_0 from __JVM_DefineClass__]
MyMessage [name=msgpack, version=0.6]

MessagePack.write()
TemplateRegistry.lookup()
TemplateRegistry.lookupAfterBuilding()
TemplateRegistry.buildAndRegister()
AbstractTemplateBuilder.buildTemplate()
DefaultBuildContext.buildTemplate()
BuildContext.build()
BuildContext.createClass()
CtClass.toClass()


But, writing with the MessagePack instance is not thread-safe.
msgpack.setOutputStream(outputStream);
msgpack.write(object);

Insteads, you can should the packer and the unpacker from a singleton MessagePack instance

MessagePack msgpack = new MessagePack(); // singleton
Packer packer = msgpack.createPacker(outputStream); // createPacker every time
packer.write(object);

for (int i = 0; i < 100; i++) {
final int iTask = i;
tasks.add(new Thread() {

@Override
public void run() {
try {
System.out.println(iTask + " start ");
File file = new File("eraseme" + iTask);
FileOutputStream outputStream = null;

outputStream = new FileOutputStream(file);
SomeClass info = new SomeClass();
try {
packer.write(info);
catch (IOException e) {
System.out.println(iTask + " write failed " + e.getMessage());
assertTrue(false);
}
outputStream.close();
FileInputStream inputStream = new FileInputStream(file);
Unpacker unpacker = msgpack.createUnpacker(inputStream);
// Deserialize directly using a template
SomeClass reply = null;
try {
reply = (ReplyMessage.ImageInfoMessage) unpacker.read(SomeClass.class);
catch (IOException e) {
System.out.println(iTask + " read failed " + e.getMessage());
assertTrue(false);
}
System.out.println(iTask + " finished " + reply + " " + file.getAbsolutePath());
catch (Exception exc) {
System.out.println("exc " + exc.getMessage());
assertTrue(false);
}
};
});
}
for (Thread task : tasks) {
task.start();
}
for (Thread task : tasks) {
task.join();
}

assertTrue(true);


Or, we should use pool of MessagePack instances.
I suggest the apache commons pool.
http://commons.apache.org/proper/commons-pool/