Why so slow, Antville.org? #229

Open
opened 2020-12-11 16:52:15 +00:00 by p3k · 0 comments
p3k commented 2020-12-11 16:52:15 +00:00 (Migrated from github.com)

Logfiles

→ du -sh /var/log/{apache2,helma}/*.log | sort -h
4.0K  /var/log/helma/helma.test.event.log
80K   /var/log/helma/helma.manage.access.log
252K  /var/log/helma/helma.antville.event.log
964K  /var/log/apache2/access.log
2.3M  /var/log/helma/helma.log
11M   /var/log/helma/helma.antville.access.log
25M   /var/log/helma/helma.antville.sql.log
152M  /var/log/apache2/error.log
180M  /var/log/apache2/other_vhosts_access.log

Heap, Memory and CPU

→ ./scripts/jpsstat.sh 
=====  ==============================  =======  =======  ======  ======  =====
 PID                Name               CurHeap  MaxHeap  CurRAM  MaxRAM  %_CPU
=====  ==============================  =======  =======  ======  ======  =====
23836  launcher.jar                       3157     3157    1408    1408    196

Download jpsstat.sh

Thread Dump

jstack $(pidof java) > /tmp/thread.dump

Download thread.dump

Threads

→ top -H
  PID S %CPU %MEM    TIME+ COMMAND
 4750 R 21.9 46.9  2018:26 antville-63
18455 R 21.9 46.9  1730:26 antville-28
22373 R 21.5 46.9  1719:52 antville-95
  682 R 21.2 46.9 18870:14 antville-2
 6780 R 21.2 46.9  2021:13 antville-101
 8200 R 21.2 46.9  2018:09 antville-102
23344 R 21.2 46.9  1728:18 antville-29
24019 R 20.9 46.9 18871:59 antville-1
 3782 R 20.9 46.9  2022:36 antville-99

PID needs to be converted to hex to search for it in the thread dump (source).

antville-1: 24019 → awk '/0x5dd3/,/^$/' /tmp/thread.dump
"antville-1" #49 prio=5 os_prio=0 tid=0x00007fa630001800 nid=0x5dd3 runnable [0x00007fa639bfd000]
   java.lang.Thread.State: RUNNABLE
        at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:1972)
        at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
        at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
        at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
        at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
        at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
        at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
        at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
        at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
        at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
        at org.mozilla.javascript.Context$1.run(Context.java:533)
        at org.mozilla.javascript.Context.call(Context.java:544)
        at org.mozilla.javascript.Context.call(Context.java:531)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
        at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
        at java.lang.Thread.run(Thread.java:748)
antville-2: 682 → awk '/0x2aa/,/^$/' /tmp/thread.dump
"antville-2" #233 prio=5 os_prio=0 tid=0x000055ad4c79a000 nid=0x2aa runnable [0x00007fa649e1b000]
   java.lang.Thread.State: RUNNABLE
        at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2059)
        at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
        at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
        at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
        at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
        at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
        at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
        at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
        at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
        at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
        at org.mozilla.javascript.Context$1.run(Context.java:533)
        at org.mozilla.javascript.Context.call(Context.java:544)
        at org.mozilla.javascript.Context.call(Context.java:531)
        at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
        at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
        at java.lang.Thread.run(Thread.java:748)
antville-28: 18455 → awk '/0x4817/,/^$/' /tmp/thread.dump
"antville-28" #86963 prio=5 os_prio=0 tid=0x00007fa6040d3800 nid=0x4817 runnable [0x00007fa639afc000]
   java.lang.Thread.State: RUNNABLE
	at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2017)
	at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
	at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
	at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
	at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
	at java.lang.Thread.run(Thread.java:748)
antville-29: 23344 → awk '/0x5b30/,/^$/' /tmp/thread.dump
"antville-29" #87365 prio=5 os_prio=0 tid=0x00007fa60c0af800 nid=0x5b30 runnable [0x00007fa60a28a000]
   java.lang.Thread.State: RUNNABLE
	at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2321)
	at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
	at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
	at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
	at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
	at java.lang.Thread.run(Thread.java:748)
antville-63: 4750 → awk '/0x128e/,/^$/' /tmp/thread.dump
"antville-63" #82806 prio=5 os_prio=0 tid=0x00007fa64c06d800 nid=0x128e runnable [0x00007fa60b9f5000]
   java.lang.Thread.State: RUNNABLE
	at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2039)
	at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
	at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
	at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
	at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.framework.core.Skin.renderAsString(Skin.java:251)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkinAsString(HopObject.java:287)
	at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
	at java.lang.Thread.run(Thread.java:748)
antville-95: 22373 → awk '/0x5765/,/^$/' /tmp/thread.dump
"antville-95" #87250 prio=5 os_prio=0 tid=0x00007fa630072000 nid=0x5765 runnable [0x00007fa60b2ed000]
   java.lang.Thread.State: RUNNABLE
	at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2017)
	at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
	at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
	at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
	at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
	at java.lang.Thread.run(Thread.java:748)
antville-99: 3782 → awk '/0xec6/,/^$/' /tmp/thread.dump
"antville-99" #82705 prio=5 os_prio=0 tid=0x00007fa604290800 nid=0xec6 runnable [0x00007fa6098d6000]
   java.lang.Thread.State: RUNNABLE
	at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2059)
	at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
	at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
	at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
	at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
	at java.lang.Thread.run(Thread.java:748)
antville-101: 6780 → awk '/0x1a7c/,/^$/' /tmp/thread.dump
"antville-101" #82996 prio=5 os_prio=0 tid=0x00007fa63006d000 nid=0x1a7c runnable [0x00007fa648f3b000]
   java.lang.Thread.State: RUNNABLE
	at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2017)
	at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
	at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
	at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
	at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
	at java.lang.Thread.run(Thread.java:748)
antville-102: 8200 → awk '/0x2008/,/^$/' /tmp/thread.dump
→ awk '/0x2008/,/^$/' thread.dump 
"antville-102" #83117 prio=5 os_prio=0 tid=0x000055ad4c7c1000 nid=0x2008 runnable [0x00007fa609c82000]
   java.lang.Thread.State: RUNNABLE
	at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2321)
	at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406)
	at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437)
	at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272)
	at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894)
	at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695)
	at helma.framework.core.Skin$Macro.render(Skin.java:777)
	at helma.framework.core.Skin.render(Skin.java:296)
	at helma.framework.core.Skin.renderAsString(Skin.java:251)
	at helma.scripting.rhino.HopObject.jsFunction_renderSkinAsString(HopObject.java:287)
	at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450)
	at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739)
	at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287)
	at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101)
	at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479)
	at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508)
	at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107)
	at org.mozilla.javascript.Context$1.run(Context.java:533)
	at org.mozilla.javascript.Context.call(Context.java:544)
	at org.mozilla.javascript.Context.call(Context.java:531)
	at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316)
	at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414)
	at java.lang.Thread.run(Thread.java:748)
# Logfiles ```plain → du -sh /var/log/{apache2,helma}/*.log | sort -h 4.0K /var/log/helma/helma.test.event.log 80K /var/log/helma/helma.manage.access.log 252K /var/log/helma/helma.antville.event.log 964K /var/log/apache2/access.log 2.3M /var/log/helma/helma.log 11M /var/log/helma/helma.antville.access.log 25M /var/log/helma/helma.antville.sql.log 152M /var/log/apache2/error.log 180M /var/log/apache2/other_vhosts_access.log ``` # Heap, Memory and CPU ```plain → ./scripts/jpsstat.sh ===== ============================== ======= ======= ====== ====== ===== PID Name CurHeap MaxHeap CurRAM MaxRAM %_CPU ===== ============================== ======= ======= ====== ====== ===== 23836 launcher.jar 3157 3157 1408 1408 196 ``` [Download jpsstat.sh](https://github.com/amarjeetanandsingh/jps_stat/blob/master/jpsstat.sh) # Thread Dump ```plain jstack $(pidof java) > /tmp/thread.dump ``` [Download thread.dump](https://github.com/antville/antville/files/5680474/thread.dump.log) # Threads ```plain → top -H PID S %CPU %MEM TIME+ COMMAND 4750 R 21.9 46.9 2018:26 antville-63 18455 R 21.9 46.9 1730:26 antville-28 22373 R 21.5 46.9 1719:52 antville-95 682 R 21.2 46.9 18870:14 antville-2 6780 R 21.2 46.9 2021:13 antville-101 8200 R 21.2 46.9 2018:09 antville-102 23344 R 21.2 46.9 1728:18 antville-29 24019 R 20.9 46.9 18871:59 antville-1 3782 R 20.9 46.9 2022:36 antville-99 ``` > PID needs to be converted to hex to search for it in the thread dump ([source](https://tech.asimio.net/2016/02/11/Troubleshoot-high-CPU-usage-in-Java-applications.html)). <details> <summary>antville-1: 24019 → <code>awk '/0x5dd3/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-1" #49 prio=5 os_prio=0 tid=0x00007fa630001800 nid=0x5dd3 runnable [0x00007fa639bfd000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:1972) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-2: 682 → <code>awk '/0x2aa/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-2" #233 prio=5 os_prio=0 tid=0x000055ad4c79a000 nid=0x2aa runnable [0x00007fa649e1b000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2059) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-28: 18455 → <code>awk '/0x4817/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-28" #86963 prio=5 os_prio=0 tid=0x00007fa6040d3800 nid=0x4817 runnable [0x00007fa639afc000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2017) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-29: 23344 → <code>awk '/0x5b30/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-29" #87365 prio=5 os_prio=0 tid=0x00007fa60c0af800 nid=0x5b30 runnable [0x00007fa60a28a000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2321) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-63: 4750 → <code>awk '/0x128e/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-63" #82806 prio=5 os_prio=0 tid=0x00007fa64c06d800 nid=0x128e runnable [0x00007fa60b9f5000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2039) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.framework.core.Skin.renderAsString(Skin.java:251) at helma.scripting.rhino.HopObject.jsFunction_renderSkinAsString(HopObject.java:287) at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-95: 22373 → <code>awk '/0x5765/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-95" #87250 prio=5 os_prio=0 tid=0x00007fa630072000 nid=0x5765 runnable [0x00007fa60b2ed000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2017) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-99: 3782 → <code>awk '/0xec6/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-99" #82705 prio=5 os_prio=0 tid=0x00007fa604290800 nid=0xec6 runnable [0x00007fa6098d6000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2059) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-101: 6780 → <code>awk '/0x1a7c/,/^$/' /tmp/thread.dump</code></summary> <pre>"antville-101" #82996 prio=5 os_prio=0 tid=0x00007fa63006d000 nid=0x1a7c runnable [0x00007fa648f3b000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2017) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details> <details> <summary>antville-102: 8200 → <code>awk '/0x2008/,/^$/' /tmp/thread.dump</code></summary> <pre>→ awk '/0x2008/,/^$/' thread.dump "antville-102" #83117 prio=5 os_prio=0 tid=0x000055ad4c7c1000 nid=0x2008 runnable [0x00007fa609c82000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:2321) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2406) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2437) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:272) at org.mozilla.javascript.regexp.NativeRegExp.execIdCall(NativeRegExp.java:2718) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.scripting.rhino.HopObject.jsFunction_renderSkin(HopObject.java:218) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.invokeDirectFunction(RequestEvaluator.java:894) at helma.framework.core.Skin$Macro.invokeAsMacro(Skin.java:695) at helma.framework.core.Skin$Macro.render(Skin.java:777) at helma.framework.core.Skin.render(Skin.java:296) at helma.framework.core.Skin.renderAsString(Skin.java:251) at helma.scripting.rhino.HopObject.jsFunction_renderSkinAsString(HopObject.java:287) at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:450) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2739) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:287) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:101) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1479) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:405) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3508) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:107) at org.mozilla.javascript.Context$1.run(Context.java:533) at org.mozilla.javascript.Context.call(Context.java:544) at org.mozilla.javascript.Context.call(Context.java:531) at helma.scripting.rhino.RhinoEngine.invoke(RhinoEngine.java:316) at helma.framework.core.RequestEvaluator.run(RequestEvaluator.java:414) at java.lang.Thread.run(Thread.java:748)</pre> </details>
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: antville/antville#229
No description provided.