Trace number 1781113

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth-5-0 2009-03-19 (4 threads)? (TO) 1976.22 1000.21

General information on the benchmark

NameAPPLICATIONS/satComp09_BioInstances/
rpoc_xits_14_UNKNOWN.cnf
MD5SUM74b3bf25aa16f613ee15d6b4b9bc9486
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2220
Number of clauses160694
Sum of the clauses size478924
Maximum clause length15
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 23434
Number of clauses of size 3157237
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 523

Solver Data 

0.26/0.30	c Command: HOME/ttsth-5-0 -num-threads 4 HOME/instance-1781113-1243685663.cnf 
0.26/0.30	c Using SATTIMEOUT of 10000
0.26/0.30	c Using SATRAM of 2047
0.26/0.30	c Size of subset node = 16
0.26/0.30	c Variables = 2220
0.26/0.30	c Clauses = 160694
0.26/0.30	c Literals = 478924
0.26/0.30	c Reduced size:
0.26/0.30	c Variables = 2220
0.26/0.30	c Clauses = 160694
0.26/0.30	c Literals = 478924
0.26/0.30	c a2
0.66/0.73	c a1
1.51/2.47	c Independent sub-problems = 1
1.51/2.47	c Sub-problem with 2220 variables and 160694 clauses
1.51/2.47	c using permutation method 4
1.51/2.47	c count = 1
1.51/2.47	c Ordering finished 1.480 seconds
1.51/2.47	c mapped 1.550 seconds
1.51/2.47	c permuted 1.550 seconds
1.51/2.47	c about to sort
1.61/2.62	c sorted
2.51/4.49	c clauses now 160694
2.51/4.49	c init finished 1.630 seconds
2.51/4.49	c tree built 2.580 seconds
2.51/4.49	c 
2.51/4.49	c PA progress                      | CPU progress Partial assignments | improvements      CPU time        PA/sec        nodes
2.51/4.49	c ---------------------------------|----------------------------------|------------------------------------------------------
2.51/4.49	c ................................ | ..........                       |                        2.6           0.0             
2.51/4.49	c ................................ | ..........                     1 |                        2.6           0.4             
2.51/4.49	c ................................ | ..........                     2 |                        2.6           0.8             
2.51/4.49	c ................................ | ..........                     3 |                        2.6           1.2             
2.51/4.49	c ................................ | ..........                     4 |                        2.6           1.6             
2.51/4.49	c ................................ | ..........                     5 |                        2.6           1.9             
2.61/4.61	c ................................ | ..........                  1000 |                        2.6         384.6         1034
200.09/107.62	c ................................ | ..........                946000 | 113117               200.1        4727.6     17258133
400.05/208.80	c ................................ | ..........               1400000 | 197659               400.3        3497.6     29887143
600.35/308.34	c ................................ | ..........               1746000 | 262145               600.4        2908.0     38887703
800.29/409.67	c ................................ | ..........               2028000 | 317853               800.3        2533.9     49284050
1000.27/511.08	c ................................ | *.........               2336000 | 375203              1000.5        2334.9     59317696
1200.69/611.71	c ................................ | *.........               2609000 | 431921              1200.7        2172.9     66967237
1399.99/711.30	c ................................ | *.........               2855000 | 482792              1400.2        2039.0     74516967
1600.90/811.52	c ................................ | *.........               3111000 | 533666              1601.0        1943.2     81468381
1800.61/912.34	c ................................ | *.........               3332000 | 579475              1800.7        1850.4     87635870

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver version 3.2.9 (svn:503) (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w /tmp/evaluation-result-1781113-1243685663/watcher-1781113-1243685663 -o /tmp/evaluation-result-1781113-1243685663/solver-1781113-1243685663 -C 10000 -W 1000 -M 2047 HOME/ttsth-5-0 -num-threads 4 HOME/instance-1781113-1243685663.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 10000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 10030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 2096128 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 2147328 KiB
Current StackSize limit: 8192 KiB


[startup+0 s]
/proc/loadavg: 8.53 8.49 8.05 7/114 25870
/proc/meminfo: memFree=2588348/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1144 CPUtime=0.01
/proc/25869/stat : 25869 (ttsth-5-0) R 25867 25869 14986 0 -1 4202496 182 0 0 0 0 1 0 0 18 0 1 0 1122703600 1171456 138 4294967295 134512640 135237372 3219589568 3219589016 134732614 0 0 0 0 0 0 0 17 3 0 0 0
/proc/25869/statm: 286 138 75 177 0 108 0

[startup+0.075738 s]
/proc/loadavg: 8.53 8.49 8.05 7/114 25870
/proc/meminfo: memFree=2588348/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=2200 CPUtime=0.03
/proc/25869/stat : 25869 (ttsth-5-0) R 25867 25869 14986 0 -1 4202496 450 0 0 0 2 1 0 0 18 0 1 0 1122703600 2252800 406 4294967295 134512640 135237372 3219589568 3219589016 134732614 0 0 0 0 0 0 0 17 3 0 0 0
/proc/25869/statm: 550 406 75 177 0 372 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 2200

[startup+0.105732 s]
/proc/loadavg: 8.53 8.49 8.05 7/114 25870
/proc/meminfo: memFree=2588348/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=3500 CPUtime=0.06
/proc/25869/stat : 25869 (ttsth-5-0) R 25867 25869 14986 0 -1 4202496 757 0 0 0 5 1 0 0 18 0 1 0 1122703600 3584000 713 4294967295 134512640 135237372 3219589568 3219588092 134639534 0 0 0 0 0 0 0 17 3 0 0 0
/proc/25869/statm: 875 714 75 177 0 697 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 3500

[startup+0.305716 s]
/proc/loadavg: 8.53 8.49 8.05 7/114 25870
/proc/meminfo: memFree=2588348/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=10668 CPUtime=0.26
/proc/25869/stat : 25869 (ttsth-5-0) R 25867 25869 14986 0 -1 4202496 2505 0 0 0 24 2 0 0 18 0 1 0 1122703600 10924032 2461 4294967295 134512640 135237372 3219589568 3219589260 134534043 0 0 0 0 0 0 0 17 3 0 0 0
/proc/25869/statm: 2667 2461 75 177 0 2489 0
Current children cumulated CPU time (s) 0.26
Current children cumulated vsize (KiB) 10668

[startup+0.705688 s]
/proc/loadavg: 8.53 8.49 8.05 7/114 25870
/proc/meminfo: memFree=2588348/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=21268 CPUtime=0.66
/proc/25869/stat : 25869 (ttsth-5-0) R 25867 25869 14986 0 -1 4202496 5098 0 0 0 64 2 0 0 18 0 1 0 1122703600 21778432 5053 4294967295 134512640 135237372 3219589568 3219587980 134680502 0 0 0 0 0 0 0 17 3 0 0 0
/proc/25869/statm: 5317 5053 76 177 0 5139 0
Current children cumulated CPU time (s) 0.66
Current children cumulated vsize (KiB) 21268

[startup+1.50562 s]
/proc/loadavg: 8.53 8.49 8.05 7/115 25879
/proc/meminfo: memFree=2568088/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=22940 CPUtime=1.09
/proc/25869/stat : 25869 (ttsth-5-0) R 25867 25869 14986 0 -1 4202496 5929 0 0 0 107 2 0 0 19 0 1 0 1122703600 23490560 5570 4294967295 134512640 135237372 3219589568 3219588696 134732614 0 0 0 0 0 0 0 17 3 0 0 0
/proc/25869/statm: 5735 5570 77 177 0 5557 0
Current children cumulated CPU time (s) 1.09
Current children cumulated vsize (KiB) 22940

[startup+3.1055 s]
/proc/loadavg: 8.53 8.49 8.05 7/115 25888
/proc/meminfo: memFree=2566164/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=27440 CPUtime=1.89
/proc/25869/stat : 25869 (ttsth-5-0) R 25867 25869 14986 0 -1 4202496 7212 0 0 0 186 3 0 0 20 0 1 0 1122703600 28098560 6696 4294967295 134512640 135237372 3219589568 3219557316 134732535 0 0 0 0 0 0 0 17 3 0 0 0
/proc/25869/statm: 6860 6696 88 177 0 6682 0
Current children cumulated CPU time (s) 1.89
Current children cumulated vsize (KiB) 27440
heavy processes:
  %CPU=83 pid=12341 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=84 pid=12739 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=84 pid=13124 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R
  %CPU=85 pid=13512 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R

[startup+6.30525 s]
/proc/loadavg: 8.49 8.49 8.05 9/118 25898
/proc/meminfo: memFree=2549272/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=77704 CPUtime=5.04
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 11566 0 0 0 499 5 0 0 23 0 5 0 1122703600 79568896 10952 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 19426 10952 95 177 0 19248 0
[pid=25869/tid=25895] ppid=25867 vsize=77704 CPUtime=0.69
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 321 0 0 0 69 0 0 0 25 0 5 0 1122704049 79568896 10952 4294967295 134512640 135237372 3219589568 3083064084 134634454 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=77704 CPUtime=0.62
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 477 0 0 0 61 1 0 0 24 0 5 0 1122704049 79568896 10952 4294967295 134512640 135237372 3219589568 3074662304 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=77704 CPUtime=0.55
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 435 0 0 0 54 1 0 0 25 0 5 0 1122704058 79568896 10952 4294967295 134512640 135237372 3219589568 3066281504 134562278 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=77704 CPUtime=0.59
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 540 0 0 0 59 0 0 0 25 0 5 0 1122704058 79568896 10952 4294967295 134512640 135237372 3219589568 3057878976 134562309 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.04
Current children cumulated vsize (KiB) 77704
heavy processes:
  %CPU=83 pid=12341 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=84 pid=12739 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/Isolet/isoletTrain.arff /users/tao/kegl/Research/Sam
  %CPU=84 pid=13124 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R
  %CPU=85 pid=13512 uid=2256 cmd=./multiboost --fileformat arff --traintest /users/tao/kegl/Research/Samples/LetterRecognition/letterTrain.arff /users/tao/kegl/R

[startup+12.7048 s]
/proc/loadavg: 8.45 8.48 8.05 10/118 25899
/proc/meminfo: memFree=2524808/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=101680 CPUtime=17.06
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 17388 0 0 0 1699 7 0 0 23 0 5 0 1122703600 104120320 16774 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 25420 16774 95 177 0 25242 0
[pid=25869/tid=25895] ppid=25867 vsize=101680 CPUtime=3.71
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 1451 0 0 0 371 0 0 0 25 0 5 0 1122704049 104120320 16774 4294967295 134512640 135237372 3219589568 3083024416 134591106 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=101680 CPUtime=3.52
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 1902 0 0 0 351 1 0 0 25 0 5 0 1122704049 104120320 16774 4294967295 134512640 135237372 3219589568 3074669648 134546026 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=101680 CPUtime=3.55
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 1881 0 0 0 353 2 0 0 25 0 5 0 1122704058 104120320 16774 4294967295 134512640 135237372 3219589568 3066237360 134680367 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=101680 CPUtime=3.69
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 2361 0 0 0 368 1 0 0 25 0 5 0 1122704058 104120320 16774 4294967295 134512640 135237372 3219589568 3057919296 134680311 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 17.06
Current children cumulated vsize (KiB) 101680

[startup+25.5038 s]
/proc/loadavg: 8.35 8.45 8.05 9/118 25899
/proc/meminfo: memFree=2480820/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=145200 CPUtime=41
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 27914 0 0 0 4084 16 0 0 23 0 5 0 1122703600 148684800 27300 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 36300 27300 95 177 0 36122 0
[pid=25869/tid=25895] ppid=25867 vsize=145200 CPUtime=9.91
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 3409 0 0 0 989 2 0 0 25 0 5 0 1122704049 148684800 27300 4294967295 134512640 135237372 3219589568 3083062304 134562290 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=145200 CPUtime=9.53
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 4588 0 0 0 950 3 0 0 25 0 5 0 1122704049 148684800 27300 4294967295 134512640 135237372 3219589568 3074663808 134749353 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=145200 CPUtime=9.35
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 5113 0 0 0 931 4 0 0 25 0 5 0 1122704058 148684800 27300 4294967295 134512640 135237372 3219589568 3066262456 134549686 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=145200 CPUtime=9.63
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 5011 0 0 0 959 4 0 0 25 0 5 0 1122704058 148684800 27300 4294967295 134512640 135237372 3219589568 3057858416 134680311 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 41
Current children cumulated vsize (KiB) 145200

[startup+51.1118 s]
/proc/loadavg: 8.30 8.43 8.05 10/119 25916
/proc/meminfo: memFree=2412216/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=217852 CPUtime=91.19
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 45111 0 0 0 9096 23 0 0 23 0 5 0 1122703600 223080448 44497 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 54463 44497 95 177 0 54285 0
[pid=25869/tid=25895] ppid=25867 vsize=217852 CPUtime=22.61
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 5627 0 0 0 2257 4 0 0 25 0 5 0 1122704049 223080448 44497 4294967295 134512640 135237372 3219589568 3083057652 134561074 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=217852 CPUtime=22.11
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 8267 0 0 0 2208 3 0 0 25 0 5 0 1122704049 223080448 44497 4294967295 134512640 135237372 3219589568 3074661532 134562323 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=217852 CPUtime=21.65
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 11641 0 0 0 2159 6 0 0 25 0 5 0 1122704058 223080448 44497 4294967295 134512640 135237372 3219589568 3066273664 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=217852 CPUtime=22.23
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 9783 0 0 0 2216 7 0 0 25 0 5 0 1122704058 223080448 44497 4294967295 134512640 135237372 3219589568 3057826096 134562344 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 91.19
Current children cumulated vsize (KiB) 217852

[startup+102.308 s]
/proc/loadavg: 8.50 8.46 8.08 9/118 26089
/proc/meminfo: memFree=2288396/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=345348 CPUtime=189.56
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 75612 0 0 0 18912 44 0 0 23 0 5 0 1122703600 353636352 74998 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 86337 74998 95 177 0 86159 0
[pid=25869/tid=25895] ppid=25867 vsize=345348 CPUtime=47.19
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 10766 0 0 0 4714 5 0 0 25 0 5 0 1122704049 353636352 74998 4294967295 134512640 135237372 3219589568 3083057488 134562320 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=345348 CPUtime=46.78
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 15480 0 0 0 4672 6 0 0 25 0 5 0 1122704049 353636352 74998 4294967295 134512640 135237372 3219589568 3074680592 134590150 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=345348 CPUtime=46.08
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 21383 0 0 0 4595 13 0 0 25 0 5 0 1122704058 353636352 74998 4294967295 134512640 135237372 3219589568 3066262368 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=345348 CPUtime=46.93
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 18190 0 0 0 4676 17 0 0 25 0 5 0 1122704058 353636352 74998 4294967295 134512640 135237372 3219589568 3057876736 134590232 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 189.56
Current children cumulated vsize (KiB) 345348

[startup+162.313 s]
/proc/loadavg: 8.77 8.55 8.14 9/118 26280
/proc/meminfo: memFree=2172584/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=467768 CPUtime=307.71
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 104700 0 0 0 30709 62 0 0 23 0 5 0 1122703600 478994432 104086 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 116942 104086 95 177 0 116764 0
[pid=25869/tid=25895] ppid=25867 vsize=467768 CPUtime=76.88
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 14793 0 0 0 7682 6 0 0 25 0 5 0 1122704049 478994432 104086 4294967295 134512640 135237372 3219589568 3083059888 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=467768 CPUtime=76.91
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 23451 0 0 0 7682 9 0 0 25 0 5 0 1122704049 478994432 104086 4294967295 134512640 135237372 3219589568 3074662432 134562309 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=467768 CPUtime=74.95
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 30332 0 0 0 7475 20 0 0 25 0 5 0 1122704058 478994432 104086 4294967295 134512640 135237372 3219589568 3066265360 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=467768 CPUtime=76.38
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 26331 0 0 0 7614 24 0 0 25 0 5 0 1122704058 478994432 104086 4294967295 134512640 135237372 3219589568 3057869904 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 307.71
Current children cumulated vsize (KiB) 467768

[startup+222.308 s]
/proc/loadavg: 8.37 8.48 8.13 9/118 26472
/proc/meminfo: memFree=2056816/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=587508 CPUtime=426.75
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 132874 0 0 0 42597 78 0 0 23 0 5 0 1122703600 601608192 132260 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 146877 132260 95 177 0 146699 0
[pid=25869/tid=25895] ppid=25867 vsize=587508 CPUtime=106.61
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 18143 0 0 0 10652 9 0 0 25 0 5 0 1122704049 601608192 132260 4294967295 134512640 135237372 3219589568 3083045440 134590534 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=587508 CPUtime=106.14
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 30916 0 0 0 10599 15 0 0 25 0 5 0 1122704049 601608192 132260 4294967295 134512640 135237372 3219589568 3074651216 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=587508 CPUtime=105.59
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 40230 0 0 0 10537 22 0 0 25 0 5 0 1122704058 601608192 132260 4294967295 134512640 135237372 3219589568 3066269600 134749314 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=587508 CPUtime=105.83
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 33792 0 0 0 10554 29 0 0 25 0 5 0 1122704058 601608192 132260 4294967295 134512640 135237372 3219589568 3057881424 134562290 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 426.75

################
# More data... #
################

/proc/meminfo: memFree=1599992/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1064328 CPUtime=1022.72
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 246213 0 0 0 102123 149 0 0 23 0 5 0 1122703600 1089871872 245599 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 266082 245599 95 177 0 265904 0
[pid=25869/tid=25895] ppid=25867 vsize=1064328 CPUtime=254.41
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 32720 0 0 0 25425 16 0 0 25 0 5 0 1122704049 1089871872 245599 4294967295 134512640 135237372 3219589568 3083058208 134562309 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1064328 CPUtime=253.16
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 62864 0 0 0 25283 33 0 0 25 0 5 0 1122704049 1089871872 245599 4294967295 134512640 135237372 3219589568 3074659776 134562420 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1064328 CPUtime=255.29
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 83769 0 0 0 25475 54 0 0 25 0 5 0 1122704058 1089871872 245599 4294967295 134512640 135237372 3219589568 3066206496 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1064328 CPUtime=257.27
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 57067 0 0 0 25684 43 0 0 25 0 5 0 1122704058 1089871872 245599 4294967295 134512640 135237372 3219589568 3057872092 134636641 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1022.72
Current children cumulated vsize (KiB) 1064328

[startup+582.31 s]
/proc/loadavg: 8.36 8.39 8.20 9/118 27606
/proc/meminfo: memFree=1527996/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1138028 CPUtime=1143.02
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 263749 0 0 0 114136 166 0 0 23 0 5 0 1122703600 1165340672 263135 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 284507 263135 95 177 0 284329 0
[pid=25869/tid=25895] ppid=25867 vsize=1138028 CPUtime=283.82
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 35460 0 0 0 28364 18 0 0 25 0 5 0 1122704049 1165340672 263135 4294967295 134512640 135237372 3219589568 3083061376 134562407 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1138028 CPUtime=283.01
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 67942 0 0 0 28263 38 0 0 25 0 5 0 1122704049 1165340672 263135 4294967295 134512640 135237372 3219589568 3074675664 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1138028 CPUtime=285.25
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 90157 0 0 0 28465 60 0 0 25 0 5 0 1122704058 1165340672 263135 4294967295 134512640 135237372 3219589568 3066257824 134562278 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1138028 CPUtime=288.35
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 60397 0 0 0 28788 47 0 0 25 0 5 0 1122704058 1165340672 263135 4294967295 134512640 135237372 3219589568 3057885136 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1143.02
Current children cumulated vsize (KiB) 1138028

[startup+642.316 s]
/proc/loadavg: 8.43 8.40 8.21 9/118 27793
/proc/meminfo: memFree=1454648/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1213556 CPUtime=1261.69
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 281703 0 0 0 125993 176 0 0 23 0 5 0 1122703600 1242681344 281089 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 303389 281089 95 177 0 303211 0
[pid=25869/tid=25895] ppid=25867 vsize=1213556 CPUtime=313.41
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 37421 0 0 0 31321 20 0 0 25 0 5 0 1122704049 1242681344 281089 4294967295 134512640 135237372 3219589568 3083062656 134562328 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1213556 CPUtime=312.51
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 72184 0 0 0 31213 38 0 0 25 0 5 0 1122704049 1242681344 281089 4294967295 134512640 135237372 3219589568 3074673040 134562420 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1213556 CPUtime=314.95
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 98470 0 0 0 31431 64 0 0 25 0 5 0 1122704058 1242681344 281089 4294967295 134512640 135237372 3219589568 3066261280 134562416 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1213556 CPUtime=318.23
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 63835 0 0 0 31772 51 0 0 25 0 5 0 1122704058 1242681344 281089 4294967295 134512640 135237372 3219589568 3057898800 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1261.69
Current children cumulated vsize (KiB) 1213556

[startup+702.311 s]
/proc/loadavg: 8.39 8.38 8.21 10/118 27980
/proc/meminfo: memFree=1382020/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1291004 CPUtime=1382.34
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 299827 0 0 0 138047 187 0 0 23 0 5 0 1122703600 1321988096 299213 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 322751 299213 95 177 0 322573 0
[pid=25869/tid=25895] ppid=25867 vsize=1291004 CPUtime=343.32
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 39406 0 0 0 34311 21 0 0 25 0 5 0 1122704049 1321988096 299213 4294967295 134512640 135237372 3219589568 3083054048 134751534 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1291004 CPUtime=342.01
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 76284 0 0 0 34161 40 0 0 25 0 5 0 1122704049 1321988096 299213 4294967295 134512640 135237372 3219589568 3074649472 134562317 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1291004 CPUtime=346.25
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 106586 0 0 0 34560 65 0 0 25 0 5 0 1122704058 1321988096 299213 4294967295 134512640 135237372 3219589568 3066224608 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1291004 CPUtime=348.17
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 67758 0 0 0 34759 58 0 0 25 0 5 0 1122704058 1321988096 299213 4294967295 134512640 135237372 3219589568 3057875504 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1382.34
Current children cumulated vsize (KiB) 1291004

[startup+762.316 s]
/proc/loadavg: 8.32 8.37 8.22 9/118 28168
/proc/meminfo: memFree=1312980/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1361720 CPUtime=1504.3
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 316601 0 0 0 150232 198 0 0 23 0 5 0 1122703600 1394401280 315987 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 340469 315987 95 177 0 340291 0
[pid=25869/tid=25895] ppid=25867 vsize=1361876 CPUtime=377.37
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 41920 0 0 0 37714 23 0 0 25 0 5 0 1122704049 1394561024 315988 4294967295 134512640 135237372 3219589568 3083059496 134732535 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1361876 CPUtime=371.81
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 80484 0 0 0 37140 41 0 0 25 0 5 0 1122704049 1394561024 315988 4294967295 134512640 135237372 3219589568 3074641616 134680364 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1361876 CPUtime=374.7
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 112424 0 0 0 37401 69 0 0 25 0 5 0 1122704058 1394561024 315988 4294967295 134512640 135237372 3219589568 3066258768 134546026 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1361876 CPUtime=377.83
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 71981 0 0 0 37721 62 0 0 25 0 5 0 1122704058 1394561024 315988 4294967295 134512640 135237372 3219589568 3057891696 134680320 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1504.3
Current children cumulated vsize (KiB) 1361720

[startup+822.302 s]
/proc/loadavg: 8.30 8.36 8.22 10/118 28356
/proc/meminfo: memFree=1248976/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1429592 CPUtime=1622.5
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 332568 0 0 0 162034 216 0 0 23 0 5 0 1122703600 1463902208 331954 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 357398 331954 95 177 0 357220 0
[pid=25869/tid=25895] ppid=25867 vsize=1429592 CPUtime=406.81
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 43783 0 0 0 40656 25 0 0 25 0 5 0 1122704049 1463902208 331954 4294967295 134512640 135237372 3219589568 3083056160 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1429592 CPUtime=401.11
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 83662 0 0 0 40070 41 0 0 25 0 5 0 1122704049 1463902208 331954 4294967295 134512640 135237372 3219589568 3074636768 134680302 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1429592 CPUtime=404.36
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 118545 0 0 0 40361 75 0 0 25 0 5 0 1122704058 1463902208 331954 4294967295 134512640 135237372 3219589568 3066261888 134546034 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1429592 CPUtime=407.63
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 76785 0 0 0 40691 72 0 0 25 0 5 0 1122704058 1463902208 331954 4294967295 134512640 135237372 3219589568 3057869104 134591086 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1622.5
Current children cumulated vsize (KiB) 1429592

[startup+882.317 s]
/proc/loadavg: 8.49 8.40 8.24 9/118 28545
/proc/meminfo: memFree=1190568/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1488768 CPUtime=1742.19
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 346630 0 0 0 173987 232 0 0 23 0 5 0 1122703600 1524498432 346016 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 372192 346016 95 177 0 372014 0
[pid=25869/tid=25895] ppid=25867 vsize=1488768 CPUtime=437.71
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 44944 0 0 0 43746 25 0 0 25 0 5 0 1122704049 1524498432 346016 4294967295 134512640 135237372 3219589568 3083053468 134562272 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1488768 CPUtime=430.41
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 87004 0 0 0 42999 42 0 0 25 0 5 0 1122704049 1524498432 346016 4294967295 134512640 135237372 3219589568 3074666448 134562380 0 0 0 0 0 0 0 -1 0 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1488768 CPUtime=434.15
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 123891 0 0 0 43333 82 0 0 25 0 5 0 1122704058 1524498432 346016 4294967295 134512640 135237372 3219589568 3066262112 134562338 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1488768 CPUtime=437.33
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 80998 0 0 0 43653 80 0 0 25 0 5 0 1122704058 1524498432 346016 4294967295 134512640 135237372 3219589568 3057877568 134562338 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1742.19
Current children cumulated vsize (KiB) 1488768

[startup+942.312 s]
/proc/loadavg: 8.47 8.41 8.25 9/118 28736
/proc/meminfo: memFree=1130036/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1551864 CPUtime=1860.52
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 361650 0 0 0 185801 251 0 0 23 0 5 0 1122703600 1589108736 361036 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 387966 361036 95 177 0 387788 0
[pid=25869/tid=25895] ppid=25867 vsize=1551864 CPUtime=467.61
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 46483 0 0 0 46735 26 0 0 25 0 5 0 1122704049 1589108736 361036 4294967295 134512640 135237372 3219589568 3083049440 134562338 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1551864 CPUtime=460.02
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 90166 0 0 0 45953 49 0 0 25 0 5 0 1122704049 1589108736 361036 4294967295 134512640 135237372 3219589568 3074665616 134680470 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1551864 CPUtime=463.65
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 129802 0 0 0 46279 86 0 0 25 0 5 0 1122704058 1589108736 361036 4294967295 134512640 135237372 3219589568 3066266656 134749343 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1551864 CPUtime=466.65
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 85406 0 0 0 46578 87 0 0 25 0 5 0 1122704058 1589108736 361036 4294967295 134512640 135237372 3219589568 3057899360 134680720 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1860.52
Current children cumulated vsize (KiB) 1551864



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1000.01 s]
/proc/loadavg: 8.35 8.39 8.26 10/118 28922
/proc/meminfo: memFree=1069528/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=1614128 CPUtime=1976.09
/proc/25869/stat : 25869 (ttsth-5-0) S 25867 25869 14986 0 -1 4202496 376327 0 0 0 197355 254 0 0 23 0 5 0 1122703600 1652867072 375713 4294967295 134512640 135237372 3219589568 3219589164 134629544 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 403532 375713 95 177 0 403354 0
[pid=25869/tid=25895] ppid=25867 vsize=1614128 CPUtime=496.35
/proc/25869/task/25895/stat : 25895 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 47753 0 0 0 49609 26 0 0 25 0 5 0 1122704049 1652867072 375713 4294967295 134512640 135237372 3219589568 3083054256 134562351 0 0 0 0 0 0 0 -1 2 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=1614128 CPUtime=489.5
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 93486 0 0 0 48900 50 0 0 25 0 5 0 1122704049 1652867072 375713 4294967295 134512640 135237372 3219589568 3074643840 134561428 0 0 0 0 0 0 0 -1 1 0 0 0
[pid=25869/tid=25897] ppid=25867 vsize=1614128 CPUtime=492.45
/proc/25869/task/25897/stat : 25897 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 134945 0 0 0 49158 87 0 0 25 0 5 0 1122704058 1652867072 375713 4294967295 134512640 135237372 3219589568 3066256080 134562355 0 0 0 0 0 0 0 -1 3 0 0 0
[pid=25869/tid=25898] ppid=25867 vsize=1614128 CPUtime=495.2
/proc/25869/task/25898/stat : 25898 (ttsth-5-0) R 25867 25869 14986 0 -1 4202560 90350 0 0 0 49432 88 0 0 25 0 5 0 1122704058 1652867072 375713 4294967295 134512640 135237372 3219589568 3057885984 134562338 0 0 0 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1976.09
Current children cumulated vsize (KiB) 1614128

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1000.21 s]
/proc/loadavg: 8.35 8.39 8.26 10/118 28922
/proc/meminfo: memFree=1069528/3368532 swapFree=1953000/1959888
[pid=25869] ppid=25867 vsize=0 CPUtime=1976.22
/proc/25869/stat : 25869 (ttsth-5-0) Z 25867 25869 14986 0 -1 4203524 376328 0 0 0 197359 263 0 0 18 0 2 0 1122703600 0 0 4294967295 0 0 0 0 0 0 0 0 0 4294967295 0 0 17 2 0 0 0
/proc/25869/statm: 0 0 0 0 0 0 0
[pid=25869/tid=25896] ppid=25867 vsize=0 CPUtime=489.6
/proc/25869/task/25896/stat : 25896 (ttsth-5-0) R 25867 25869 14986 0 -1 4203588 93487 0 0 0 48901 59 0 0 25 0 2 0 1122704049 0 0 4294967295 0 0 0 0 0 0 0 0 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1976.22
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1000.21
CPU time (s): 1976.22
CPU user time (s): 1973.59
CPU system time (s): 2.63
CPU usage (%): 197.581
Max. virtual memory (cumulated for all children) (KiB): 1614128

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1973.59
system time used= 2.63
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 376328
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 549
involuntary context switches= 56146

runsolver used 0 second user time and 0 second system time

The end

Launcher Data 

Begin job on tipi45 at 2009-05-30 14:14:23
IDJOB=1781113
IDBENCH=71097
IDSOLVER=632
FILE ID=tipi45/1781113-1243685663
PBS_JOBID=
Free space on /tmp= 15499 MiB

SOLVER NAME= ttsth-5-0 2009-03-19 (4 threads)
BENCH NAME= SAT09/APPLICATIONS/satComp09_BioInstances/rpoc_xits_14_UNKNOWN.cnf
COMMAND LINE= HOME/ttsth-5-0 -num-threads MAXNBTHREAD BENCHNAME
RUNSOLVER COMMAND LINE= runsolver --timestamp  -w /tmp/evaluation-result-1781113-1243685663/watcher-1781113-1243685663 -o /tmp/evaluation-result-1781113-1243685663/solver-1781113-1243685663 -C 10000 -W 1000 -M 2047  HOME/ttsth-5-0 -num-threads 4 HOME/instance-1781113-1243685663.cnf

TIME LIMIT= 10000 seconds
MEMORY LIMIT= 2047 MiB
MAX NB THREAD= 4

MD5SUM BENCH= 74b3bf25aa16f613ee15d6b4b9bc9486
RANDOM SEED= 236399483

Linux tipi45 2.6.21.5 #6 SMP Wed Jul 4 17:16:38 CEST 2007 i686 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3992.41
clflush size	: 64

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.10
clflush size	: 64

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 0
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.16
clflush size	: 64

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.081
cache size	: 4096 KB
physical id	: 3
siblings	: 2
core id		: 1
cpu cores	: 2
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips	: 3990.14
clflush size	: 64


/proc/meminfo:
MemTotal:      3368532 kB
MemFree:       2588248 kB
Buffers:         35184 kB
Cached:         183168 kB
SwapCached:       1856 kB
Active:         609860 kB
Inactive:        90796 kB
HighTotal:     2489216 kB
HighFree:      1838404 kB
LowTotal:       879316 kB
LowFree:        749844 kB
SwapTotal:     1959888 kB
SwapFree:      1953000 kB
Dirty:            5132 kB
Writeback:           0 kB
AnonPages:      481432 kB
Mapped:           6380 kB
Slab:            66264 kB
SReclaimable:    54724 kB
SUnreclaim:      11540 kB
PageTables:       1284 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   3644152 kB
Committed_AS:   569920 kB
VmallocTotal:   114680 kB
VmallocUsed:     17900 kB
VmallocChunk:    96244 kB

Free space on /tmp at the end= 15499 MiB
End job on tipi45 at 2009-05-30 14:31:06