Trace number 1781083

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth-5-0 2009-03-19 (4 threads)? (MO) 2057.77 830.233

General information on the benchmark

NameAPPLICATIONS/satComp09_BioInstances/
ndhf_xits_09_UNSAT.cnf
MD5SUM3cd74d1813a4408b4123afc23df07d7d
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1910
Number of clauses167931
Sum of the clauses size500698
Maximum clause length10
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 23312
Number of clauses of size 3164588
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 531

Solver Data 

0.20/0.26	c Command: HOME/ttsth-5-0 -num-threads 4 HOME/instance-1781083-1243685221.cnf 
0.20/0.26	c Using SATTIMEOUT of 10000
0.20/0.26	c Using SATRAM of 2047
0.20/0.26	c Size of subset node = 16
0.20/0.26	c Variables = 1910
0.20/0.26	c Clauses = 167931
0.20/0.26	c Literals = 500698
0.20/0.26	c Reduced size:
0.20/0.26	c Variables = 1910
0.20/0.26	c Clauses = 167931
0.20/0.26	c Literals = 500698
0.20/0.26	c a2
0.70/0.71	c a1
1.50/1.56	c Independent sub-problems = 1
1.50/1.56	c Sub-problem with 1910 variables and 167931 clauses
1.50/1.56	c using permutation method 4
1.50/1.56	c count = 1
1.50/1.56	c Ordering finished 1.490 seconds
1.50/1.56	c mapped 1.560 seconds
1.50/1.56	c permuted 1.560 seconds
1.50/1.56	c about to sort
1.60/1.62	c sorted
2.90/2.92	c clauses now 167931
2.90/2.92	c init finished 1.650 seconds
2.90/2.92	c tree built 2.920 seconds
2.90/2.92	c 
2.90/2.92	c PA progress                      | CPU progress Partial assignments | improvements      CPU time        PA/sec        nodes
2.90/2.92	c ---------------------------------|----------------------------------|------------------------------------------------------
2.90/2.92	c ................................ | ..........                       |                        2.9           0.0             
2.90/2.92	c ................................ | ..........                     1 |                        2.9           0.3             
2.90/2.92	c ................................ | ..........                     2 |                        2.9           0.7             
2.90/2.92	c ................................ | ..........                     3 |                        2.9           1.0             
2.90/2.92	c ................................ | ..........                     4 |                        2.9           1.4             
2.90/2.92	c ................................ | ..........                     5 |                        2.9           1.7             
2.90/2.93	c ................................ | ..........                  1000 | 31                     2.9         341.3          288
199.91/86.77	c ................................ | ..........               1164000 | 108598               200.0        5819.1     20158148
400.84/171.94	c ................................ | ..........               1617000 | 142424               400.9        4033.6     31964397
599.99/251.72	c ................................ | ..........               2176000 | 195695               600.0        3626.6     46445448
800.48/332.31	c ................................ | ..........               2789000 | 231533               800.5        3484.1     60970953
1001.87/410.43	c ................................ | *.........               3136000 | 258665              1001.9        3130.0     74583848
1201.08/490.41	c ................................ | *.........               3241000 | 265767              1201.3        2697.8     84078953
1402.65/571.28	c ................................ | *.........               3359000 | 272704              1402.9        2394.4     94647106
1600.18/650.44	c ................................ | *.........               3444000 | 277174              1600.2        2152.2    103653951
1800.07/726.63	c ................................ | *.........               3573000 | 283824              1800.1        1984.8    112070405
2000.67/807.28	c ................................ | **........               3712000 | 292321              2000.8        1855.2    120504802

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver version 3.2.9 (svn:503) (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w /tmp/evaluation-result-1781083-1243685221/watcher-1781083-1243685221 -o /tmp/evaluation-result-1781083-1243685221/solver-1781083-1243685221 -C 10000 -W 1000 -M 2047 HOME/ttsth-5-0 -num-threads 4 HOME/instance-1781083-1243685221.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 10000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 10030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 2096128 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 2147328 KiB
Current StackSize limit: 8192 KiB


[startup+0 s]
/proc/loadavg: 6.42 6.54 5.99 5/113 31540
/proc/meminfo: memFree=3056768/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=1272 CPUtime=0.01
/proc/31540/stat : 31540 (ttsth-5-0) R 31538 31540 32205 0 -1 4202496 227 0 0 0 1 0 0 0 20 0 1 0 1133186110 1302528 182 4294967295 134512640 135237372 3217414048 3217412644 134886158 0 0 0 0 0 0 0 17 2 0 0 0
/proc/31540/statm: 318 182 75 177 0 140 0

[startup+0.098135 s]
/proc/loadavg: 6.42 6.54 5.99 5/113 31540
/proc/meminfo: memFree=3056768/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=4936 CPUtime=0.09
/proc/31540/stat : 31540 (ttsth-5-0) R 31538 31540 32205 0 -1 4202496 1112 0 0 0 9 0 0 0 20 0 1 0 1133186110 5054464 1067 4294967295 134512640 135237372 3217414048 3217413496 134732614 0 0 0 0 0 0 0 17 2 0 0 0
/proc/31540/statm: 1234 1067 75 177 0 1056 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 4936

[startup+0.10813 s]
/proc/loadavg: 6.42 6.54 5.99 5/113 31540
/proc/meminfo: memFree=3056768/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=5332 CPUtime=0.1
/proc/31540/stat : 31540 (ttsth-5-0) R 31538 31540 32205 0 -1 4202496 1218 0 0 0 10 0 0 0 20 0 1 0 1133186110 5459968 1173 4294967295 134512640 135237372 3217414048 3217412572 134639534 0 0 0 0 0 0 0 17 2 0 0 0
/proc/31540/statm: 1333 1173 75 177 0 1155 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 5332

[startup+0.308111 s]
/proc/loadavg: 6.42 6.54 5.99 5/113 31540
/proc/meminfo: memFree=3056768/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=20524 CPUtime=0.3
/proc/31540/stat : 31540 (ttsth-5-0) R 31538 31540 32205 0 -1 4202496 4904 0 0 0 30 0 0 0 21 0 1 0 1133186110 21016576 4859 4294967295 134512640 135237372 3217414048 3217413440 134749343 0 0 0 0 0 0 0 17 2 0 0 0
/proc/31540/statm: 5131 4864 76 177 0 4953 0
Current children cumulated CPU time (s) 0.3
Current children cumulated vsize (KiB) 20524

[startup+0.708077 s]
/proc/loadavg: 6.42 6.54 5.99 5/113 31540
/proc/meminfo: memFree=3056768/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=22708 CPUtime=0.7
/proc/31540/stat : 31540 (ttsth-5-0) R 31538 31540 32205 0 -1 4202496 5336 0 0 0 70 0 0 0 22 0 1 0 1133186110 23252992 5290 4294967295 134512640 135237372 3217414048 3217413056 134680362 0 0 0 0 0 0 0 17 2 0 0 0
/proc/31540/statm: 5677 5290 76 177 0 5499 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 22708

[startup+1.50802 s]
/proc/loadavg: 6.42 6.54 5.99 4/116 31543
/proc/meminfo: memFree=3037564/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=24176 CPUtime=1.5
/proc/31540/stat : 31540 (ttsth-5-0) R 31538 31540 32205 0 -1 4202496 7586 0 0 0 150 0 0 0 25 0 1 0 1133186110 24756224 5916 4294967295 134512640 135237372 3217414048 3217413280 134680467 0 0 0 0 0 0 0 17 2 0 0 0
/proc/31540/statm: 6044 5916 83 177 0 5866 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 24176

[startup+3.10788 s]
/proc/loadavg: 6.15 6.48 5.97 5/114 31545
/proc/meminfo: memFree=3026908/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=77892 CPUtime=3.27
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 13018 0 0 0 326 1 0 0 25 0 5 0 1133186110 79761408 10922 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 19473 10922 95 177 0 19295 0
Current children cumulated CPU time (s) 3.27
Current children cumulated vsize (KiB) 77892

[startup+6.3076 s]
/proc/loadavg: 6.15 6.48 5.97 7/118 31553
/proc/meminfo: memFree=2998616/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=99312 CPUtime=9.65
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 18240 0 0 0 962 3 0 0 25 0 5 0 1133186110 101695488 16144 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 24828 16144 95 177 0 24650 0
[pid=31540/tid=31546] ppid=31538 vsize=99312 CPUtime=1.74
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 1440 0 0 0 173 1 0 0 25 0 5 0 1133186402 101695488 16144 4294967295 134512640 135237372 3217414048 3081777168 134546034 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=99312 CPUtime=1.62
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 1507 0 0 0 161 1 0 0 25 0 5 0 1133186402 101695488 16144 4294967295 134512640 135237372 3217414048 3073380656 134680229 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=99312 CPUtime=1.61
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 1217 0 0 0 161 0 0 0 25 0 5 0 1133186402 101695488 16144 4294967295 134512640 135237372 3217414048 3064942912 134749343 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=99312 CPUtime=1.76
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 1473 0 0 0 176 0 0 0 25 0 5 0 1133186402 101695488 16144 4294967295 134512640 135237372 3217414048 3056548528 134551002 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 9.65
Current children cumulated vsize (KiB) 99312

[startup+12.7071 s]
/proc/loadavg: 6.12 6.46 5.97 8/120 31565
/proc/meminfo: memFree=2964672/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=129856 CPUtime=22.43
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 25833 0 0 0 2233 10 0 0 25 0 5 0 1133186110 132972544 23737 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 32464 23737 95 177 0 32286 0
[pid=31540/tid=31546] ppid=31538 vsize=129856 CPUtime=5.09
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 2854 0 0 0 507 2 0 0 25 0 5 0 1133186402 132972544 23737 4294967295 134512640 135237372 3217414048 3081792720 134589006 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=129856 CPUtime=4.82
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 3953 0 0 0 480 2 0 0 25 0 5 0 1133186402 132972544 23737 4294967295 134512640 135237372 3217414048 3073380084 134680504 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=129856 CPUtime=4.9
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 2804 0 0 0 488 2 0 0 25 0 5 0 1133186402 132972544 23737 4294967295 134512640 135237372 3217414048 3064962960 134562380 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=129856 CPUtime=4.71
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 3619 0 0 0 468 3 0 0 25 0 5 0 1133186402 132972544 23737 4294967295 134512640 135237372 3217414048 3056555200 134562309 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 22.43
Current children cumulated vsize (KiB) 129856

[startup+25.506 s]
/proc/loadavg: 6.10 6.45 5.97 8/118 31581
/proc/meminfo: memFree=2902252/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=196000 CPUtime=48.69
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 41347 0 0 0 4850 19 0 0 25 0 5 0 1133186110 200704000 39251 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 49000 39251 95 177 0 48822 0
[pid=31540/tid=31546] ppid=31538 vsize=196000 CPUtime=11.41
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 5663 0 0 0 1137 4 0 0 25 0 5 0 1133186402 200704000 39251 4294967295 134512640 135237372 3217414048 3081778208 134562388 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=196000 CPUtime=12.3
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 9730 0 0 0 1225 5 0 0 25 0 5 0 1133186402 200704000 39251 4294967295 134512640 135237372 3217414048 3073364492 134550034 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=196000 CPUtime=10.96
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 5592 0 0 0 1094 2 0 0 25 0 5 0 1133186402 200704000 39251 4294967295 134512640 135237372 3217414048 3064977072 134749353 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=196000 CPUtime=11.11
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 7759 0 0 0 1104 7 0 0 25 0 5 0 1133186402 200704000 39251 4294967295 134512640 135237372 3217414048 3056587632 134749353 0 0 0 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 48.69
Current children cumulated vsize (KiB) 196000

[startup+51.1037 s]
/proc/loadavg: 6.35 6.48 6.00 7/118 31808
/proc/meminfo: memFree=2808468/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=294476 CPUtime=112.14
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 64876 0 0 0 11184 30 0 0 25 0 5 0 1133186110 301543424 62780 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 73619 62780 95 177 0 73441 0
[pid=31540/tid=31546] ppid=31538 vsize=294476 CPUtime=24.16
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 10214 0 0 0 2411 5 0 0 25 0 5 0 1133186402 301543424 62780 4294967295 134512640 135237372 3217414048 3081764752 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=294476 CPUtime=35.64
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 17146 0 0 0 3555 9 0 0 25 0 5 0 1133186402 301543424 62780 4294967295 134512640 135237372 3217414048 3073364236 134680506 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=294476 CPUtime=25.61
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 10735 0 0 0 2556 5 0 0 25 0 5 0 1133186402 301543424 62780 4294967295 134512640 135237372 3217414048 3064991600 134591065 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=294476 CPUtime=23.81
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 14178 0 0 0 2371 10 0 0 25 0 5 0 1133186402 301543424 62780 4294967295 134512640 135237372 3217414048 3056600208 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 112.14
Current children cumulated vsize (KiB) 294476

[startup+102.319 s]
/proc/loadavg: 6.40 6.46 6.02 8/119 32052
/proc/meminfo: memFree=2662228/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=445640 CPUtime=231.73
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 100913 0 0 0 23130 43 0 0 25 0 5 0 1133186110 456335360 98817 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 111410 98817 95 177 0 111232 0
[pid=31540/tid=31546] ppid=31538 vsize=445640 CPUtime=57.07
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 17538 0 0 0 5700 7 0 0 25 0 5 0 1133186402 456335360 98817 4294967295 134512640 135237372 3217414048 3081764432 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=445640 CPUtime=61.62
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 23177 0 0 0 6150 12 0 0 25 0 5 0 1133186402 456335360 98817 4294967295 134512640 135237372 3217414048 3073377408 134550989 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=445640 CPUtime=60.71
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 23485 0 0 0 6061 10 0 0 25 0 5 0 1133186402 456335360 98817 4294967295 134512640 135237372 3217414048 3064966544 134590160 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=445640 CPUtime=49.41
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 24110 0 0 0 4928 13 0 0 25 0 5 0 1133186402 456335360 98817 4294967295 134512640 135237372 3217414048 3056595360 134587914 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 231.73
Current children cumulated vsize (KiB) 445640

[startup+162.314 s]
/proc/loadavg: 6.43 6.44 6.03 8/119 32288
/proc/meminfo: memFree=2534496/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=583020 CPUtime=378.84
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 133254 0 0 0 37822 62 0 0 25 0 5 0 1133186110 597012480 131158 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 145755 131158 95 177 0 145577 0
[pid=31540/tid=31546] ppid=31538 vsize=583020 CPUtime=98.25
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 23389 0 0 0 9813 12 0 0 25 0 5 0 1133186402 597012480 131158 4294967295 134512640 135237372 3217414048 3081762816 134562290 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=583020 CPUtime=105.92
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 32459 0 0 0 10579 13 0 0 25 0 5 0 1133186402 597012480 131158 4294967295 134512640 135237372 3217414048 3073367824 134587827 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=583020 CPUtime=90.44
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 30603 0 0 0 9025 19 0 0 25 0 5 0 1133186402 597012480 131158 4294967295 134512640 135237372 3217414048 3064940176 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=583020 CPUtime=81.31
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 34200 0 0 0 8114 17 0 0 25 0 5 0 1133186402 597012480 131158 4294967295 134512640 135237372 3217414048 3056573680 134562388 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 378.84
Current children cumulated vsize (KiB) 583020

[startup+222.309 s]
/proc/loadavg: 6.54 6.46 6.06 9/119 32627
/proc/meminfo: memFree=2375824/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=746180 CPUtime=526.84
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 172473 0 0 0 52603 81 0 0 25 0 5 0 1133186110 764088320 170377 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 186545 170377 95 177 0 186367 0
[pid=31540/tid=31546] ppid=31538 vsize=746180 CPUtime=128.13
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 28005 0 0 0 12799 14 0 0 25 0 5 0 1133186402 764088320 170377 4294967295 134512640 135237372 3217414048 3081765408 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=746180 CPUtime=142.42
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 47137 0 0 0 14225 17 0 0 25 0 5 0 1133186402 764088320 170377 4294967295 134512640 135237372 3217414048 3073362828 134562272 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=746180 CPUtime=140.91
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 41873 0 0 0 14066 25 0 0 25 0 5 0 1133186402 764088320 170377 4294967295 134512640 135237372 3217414048 3065015808 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=746180 CPUtime=112.46
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 42855 0 0 0 11222 24 0 0 25 0 5 0 1133186402 764088320 170377 4294967295 134512640 135237372 3217414048 3056619872 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 526.84
Current children cumulated vsize (KiB) 746180

[startup+282.304 s]
/proc/loadavg: 6.48 6.44 6.08 10/119 384
/proc/meminfo: memFree=2198140/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=936508 CPUtime=676.44
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 217422 0 0 0 67537 107 0 0 25 0 5 0 1133186110 958984192 215326 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 234127 215326 95 177 0 233949 0
[pid=31540/tid=31546] ppid=31538 vsize=936508 CPUtime=187.74
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 41216 0 0 0 18753 21 0 0 25 0 5 0 1133186402 958984192 215326 4294967295 134512640 135237372 3217414048 3081764480 134562338 0 0 0 0 0 0 0 -1 0 0 0 0

################
# More data... #
################


[startup+462.308 s]
/proc/loadavg: 6.58 6.50 6.16 9/119 1124
/proc/meminfo: memFree=1744900/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=1411716 CPUtime=1131.44
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 330251 0 0 0 112977 167 0 0 25 0 5 0 1133186110 1445597184 328155 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 352929 328155 95 177 0 352751 0
[pid=31540/tid=31546] ppid=31538 vsize=1411716 CPUtime=289.95
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 53265 0 0 0 28965 30 0 0 25 0 5 0 1133186402 1445597184 328155 4294967295 134512640 135237372 3217414048 3081761744 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=1411716 CPUtime=321.96
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 105377 0 0 0 32151 45 0 0 25 0 5 0 1133186402 1445597184 328155 4294967295 134512640 135237372 3217414048 3073396192 134562309 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=1411716 CPUtime=283.01
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 82619 0 0 0 28259 42 0 0 25 0 5 0 1133186402 1445597184 328155 4294967295 134512640 135237372 3217414048 3064962224 134562380 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=1411716 CPUtime=233.6
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 76387 0 0 0 23311 49 0 0 25 0 5 0 1133186402 1445597184 328155 4294967295 134512640 135237372 3217414048 3056585200 134562380 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1131.44
Current children cumulated vsize (KiB) 1411716

[startup+522.313 s]
/proc/loadavg: 6.59 6.52 6.19 7/118 1373
/proc/meminfo: memFree=1626984/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=1535280 CPUtime=1280.99
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 360108 0 0 0 127909 190 0 0 25 0 5 0 1133186110 1572126720 358012 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 383820 358012 95 177 0 383642 0
[pid=31540/tid=31546] ppid=31538 vsize=1535280 CPUtime=320.05
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 54524 0 0 0 31970 35 0 0 25 0 5 0 1133186402 1572126720 358012 4294967295 134512640 135237372 3217414048 3081763660 134562323 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=1535280 CPUtime=363.43
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 116737 0 0 0 36291 52 0 0 25 0 5 0 1133186402 1572126720 358012 4294967295 134512640 135237372 3217414048 3073396032 134549905 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=1535280 CPUtime=312.91
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 87826 0 0 0 31246 45 0 0 25 0 5 0 1133186402 1572126720 358012 4294967295 134512640 135237372 3217414048 3064958240 134562290 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=1535280 CPUtime=281.68
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 88418 0 0 0 28111 57 0 0 25 0 5 0 1133186402 1572126720 358012 4294967295 134512640 135237372 3217414048 3056588960 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1280.99
Current children cumulated vsize (KiB) 1535280

[startup+582.308 s]
/proc/loadavg: 6.53 6.53 6.21 7/118 1660
/proc/meminfo: memFree=1507364/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=1660912 CPUtime=1430.43
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 389608 0 0 0 142843 200 0 0 25 0 5 0 1133186110 1700773888 387512 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 415228 387512 95 177 0 415050 0
[pid=31540/tid=31546] ppid=31538 vsize=1660912 CPUtime=350.13
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 55744 0 0 0 34977 36 0 0 25 0 5 0 1133186402 1700773888 387512 4294967295 134512640 135237372 3217414048 3081763344 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=1660912 CPUtime=393.62
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 124763 0 0 0 39307 55 0 0 25 0 5 0 1133186402 1700773888 387512 4294967295 134512640 135237372 3217414048 3073378624 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=1660912 CPUtime=342.64
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 93268 0 0 0 34216 48 0 0 25 0 5 0 1133186402 1700773888 387512 4294967295 134512640 135237372 3217414048 3064962672 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=1660912 CPUtime=341.12
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 103230 0 0 0 34052 60 0 0 25 0 5 0 1133186402 1700773888 387512 4294967295 134512640 135237372 3217414048 3056590272 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1430.43
Current children cumulated vsize (KiB) 1660912

[startup+642.313 s]
/proc/loadavg: 6.34 6.49 6.21 9/118 1937
/proc/meminfo: memFree=1401212/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=1771716 CPUtime=1580
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 415935 0 0 0 157783 217 0 0 25 0 5 0 1133186110 1814237184 413839 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 442929 413839 95 177 0 442751 0
[pid=31540/tid=31546] ppid=31538 vsize=1771716 CPUtime=380.02
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 56823 0 0 0 37962 40 0 0 25 0 5 0 1133186402 1814237184 413839 4294967295 134512640 135237372 3217414048 3081763792 134562344 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=1771716 CPUtime=445.29
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 135539 0 0 0 44468 61 0 0 25 0 5 0 1133186402 1814237184 413839 4294967295 134512640 135237372 3217414048 3073377168 134562290 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=1771716 CPUtime=372.66
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 100123 0 0 0 37214 52 0 0 25 0 5 0 1133186402 1814237184 413839 4294967295 134512640 135237372 3217414048 3064962976 134562351 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=1771716 CPUtime=379.11
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 110847 0 0 0 37848 63 0 0 25 0 5 0 1133186402 1814237184 413839 4294967295 134512640 135237372 3217414048 3056580208 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1580
Current children cumulated vsize (KiB) 1771716

[startup+702.318 s]
/proc/loadavg: 6.35 6.47 6.22 7/118 2212
/proc/meminfo: memFree=1310612/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=1869508 CPUtime=1739.59
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 438843 0 0 0 173724 235 0 0 25 0 5 0 1133186110 1914376192 436747 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 467377 436747 95 177 0 467199 0
[pid=31540/tid=31546] ppid=31538 vsize=1869508 CPUtime=420.15
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 58522 0 0 0 41965 50 0 0 25 0 5 0 1133186402 1914376192 436747 4294967295 134512640 135237372 3217414048 3081763216 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=1869508 CPUtime=485.69
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 141747 0 0 0 48506 63 0 0 25 0 5 0 1133186402 1914376192 436747 4294967295 134512640 135237372 3217414048 3073399184 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=1869508 CPUtime=421.72
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 110110 0 0 0 42115 57 0 0 25 0 5 0 1133186402 1914376192 436747 4294967295 134512640 135237372 3217414048 3064976064 134562344 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=1869508 CPUtime=409.11
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 115861 0 0 0 40847 64 0 0 25 0 5 0 1133186402 1914376192 436747 4294967295 134512640 135237372 3217414048 3056582112 134562351 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1739.59
Current children cumulated vsize (KiB) 1869508

[startup+762.312 s]
/proc/loadavg: 6.49 6.50 6.25 7/118 2491
/proc/meminfo: memFree=1190956/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=1995252 CPUtime=1888.76
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 468438 0 0 0 188627 249 0 0 25 0 5 0 1133186110 2043138048 466342 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 498813 466342 95 177 0 498635 0
[pid=31540/tid=31546] ppid=31538 vsize=1995252 CPUtime=453.46
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 60086 0 0 0 45292 54 0 0 25 0 5 0 1133186402 2043138048 466342 4294967295 134512640 135237372 3217414048 3081759200 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=1995252 CPUtime=517.25
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 148859 0 0 0 51660 65 0 0 25 0 5 0 1133186402 2043138048 466342 4294967295 134512640 135237372 3217414048 3073379872 134562341 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=1995252 CPUtime=476.03
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 124633 0 0 0 47543 60 0 0 25 0 5 0 1133186402 2043138048 466342 4294967295 134512640 135237372 3217414048 3064963264 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=1995252 CPUtime=439.11
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 122257 0 0 0 43842 69 0 0 25 0 5 0 1133186402 2043138048 466342 4294967295 134512640 135237372 3217414048 3056588556 134519936 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1888.76
Current children cumulated vsize (KiB) 1995252

[startup+822.307 s]
/proc/loadavg: 6.56 6.54 6.28 7/118 2775
/proc/meminfo: memFree=1108552/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=2081440 CPUtime=2038.12
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 489169 0 0 0 203549 263 0 0 25 0 5 0 1133186110 2131394560 487073 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 520360 487073 95 177 0 520182 0
[pid=31540/tid=31546] ppid=31538 vsize=2081440 CPUtime=508.46
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 61983 0 0 0 50792 54 0 0 25 0 5 0 1133186402 2131394560 487073 4294967295 134512640 135237372 3217414048 3081763424 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=2081440 CPUtime=550.72
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 155891 0 0 0 55003 69 0 0 25 0 5 0 1133186402 2131394560 487073 4294967295 134512640 135237372 3217414048 3073379184 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=2081440 CPUtime=505.87
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 128613 0 0 0 50518 69 0 0 25 0 5 0 1133186402 2131394560 487073 4294967295 134512640 135237372 3217414048 3064962128 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=2081440 CPUtime=470.15
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 130079 0 0 0 46945 70 0 0 25 0 5 0 1133186402 2131394560 487073 4294967295 134512640 135237372 3217414048 3056584528 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2038.12
Current children cumulated vsize (KiB) 2081440



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+830.107 s]
/proc/loadavg: 6.52 6.53 6.27 7/118 2787
/proc/meminfo: memFree=1094396/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=2096200 CPUtime=2057.63
/proc/31540/stat : 31540 (ttsth-5-0) S 31538 31540 32205 0 -1 4202496 492615 0 0 0 205499 264 0 0 25 0 5 0 1133186110 2146508800 490519 4294967295 134512640 135237372 3217414048 3217413644 134629544 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 524050 490519 95 177 0 523872 0
[pid=31540/tid=31546] ppid=31538 vsize=2096200 CPUtime=512.37
/proc/31540/task/31546/stat : 31546 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 62218 0 0 0 51183 54 0 0 25 0 5 0 1133186402 2146508800 490519 4294967295 134512640 135237372 3217414048 3081766160 134562416 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=2096200 CPUtime=554.62
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 157043 0 0 0 55392 70 0 0 25 0 5 0 1133186402 2146508800 490519 4294967295 134512640 135237372 3217414048 3073396288 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31548] ppid=31538 vsize=2096200 CPUtime=509.77
/proc/31540/task/31548/stat : 31548 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 129045 0 0 0 50908 69 0 0 25 0 5 0 1133186402 2146508800 490519 4294967295 134512640 135237372 3217414048 3064975456 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=31540/tid=31549] ppid=31538 vsize=2096200 CPUtime=477.95
/proc/31540/task/31549/stat : 31549 (ttsth-5-0) R 31538 31540 32205 0 -1 4202560 131706 0 0 0 47725 70 0 0 25 0 5 0 1133186402 2146508800 490519 4294967295 134512640 135237372 3217414048 3056598624 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2057.63
Current children cumulated vsize (KiB) 2096200

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+830.227 s]
/proc/loadavg: 6.52 6.53 6.27 7/118 2787
/proc/meminfo: memFree=1094396/3368532 swapFree=1954888/1959888
[pid=31540] ppid=31538 vsize=0 CPUtime=2057.77
/proc/31540/stat : 31540 (ttsth-5-0) Z 31538 31540 32205 0 -1 4203524 492624 0 0 0 205504 273 0 0 18 0 2 0 1133186110 0 0 4294967295 0 0 0 0 0 0 0 0 0 4294967295 0 0 17 1 0 0 0
/proc/31540/statm: 0 0 0 0 0 0 0
[pid=31540/tid=31547] ppid=31538 vsize=0 CPUtime=554.71
/proc/31540/task/31547/stat : 31547 (ttsth-5-0) R 31538 31540 32205 0 -1 4203588 157043 0 0 0 55392 79 0 0 25 0 2 0 1133186402 0 0 4294967295 0 0 0 0 0 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2057.77
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31540 and gives
#  childrusage.ru_utime.tv_sec=2055
#  childrusage.ru_utime.tv_usec=40000
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=730000
# CPU time returned by wait4() is 2057.77
# while last known CPU time is 2057.77
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 830.233
CPU time (s): 2057.77
CPU user time (s): 2055.04
CPU system time (s): 2.73
CPU usage (%): 247.855
Max. virtual memory (cumulated for all children) (KiB): 2096200

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2055.04
system time used= 2.73
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 492624
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 811
involuntary context switches= 55612

runsolver used 0 second user time and 0 second system time

The end

Launcher Data 

Begin job on  at 2009-05-30 14:07:01
IDJOB=1781083
IDBENCH=71093
IDSOLVER=632
FILE ID=/1781083-1243685221
PBS_JOBID=
Free space on /tmp= 15876 MiB

SOLVER NAME= ttsth-5-0 2009-03-19 (4 threads)
BENCH NAME= SAT09/APPLICATIONS/satComp09_BioInstances/ndhf_xits_09_UNSAT.cnf
COMMAND LINE= HOME/ttsth-5-0 -num-threads MAXNBTHREAD BENCHNAME
RUNSOLVER COMMAND LINE= runsolver --timestamp  -w /tmp/evaluation-result-1781083-1243685221/watcher-1781083-1243685221 -o /tmp/evaluation-result-1781083-1243685221/solver-1781083-1243685221 -C 10000 -W 1000 -M 2047  HOME/ttsth-5-0 -num-threads 4 HOME/instance-1781083-1243685221.cnf

TIME LIMIT= 10000 seconds
MEMORY LIMIT= 2047 MiB
MAX NB THREAD= 4

MD5SUM BENCH= 3cd74d1813a4408b4123afc23df07d7d
RANDOM SEED= 8039214

Linux (none) 2.6.21.5 #6 SMP Wed Jul 4 17:16:38 CEST 2007 i686 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3992.43
clflush size	: 64

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.11
clflush size	: 64

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.16
clflush size	: 64

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.105
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.15
clflush size	: 64


/proc/meminfo:
MemTotal:      3368532 kB
MemFree:       3056720 kB
Buffers:         77080 kB
Cached:         123120 kB
SwapCached:        720 kB
Active:         142960 kB
Inactive:        72852 kB
HighTotal:     2489216 kB
HighFree:      2346180 kB
LowTotal:       879316 kB
LowFree:        710540 kB
SwapTotal:     1959888 kB
SwapFree:      1954888 kB
Dirty:            3724 kB
Writeback:           0 kB
AnonPages:       14896 kB
Mapped:           6224 kB
Slab:            84484 kB
SReclaimable:    74904 kB
SUnreclaim:       9580 kB
PageTables:        748 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3644152 kB
Committed_AS:   117276 kB
VmallocTotal:   114680 kB
VmallocUsed:      1444 kB
VmallocChunk:   113072 kB

Free space on /tmp at the end= 15876 MiB
End job on  at 2009-05-30 14:20:53