Trace number 3265478

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.35 1207.3

General information on the benchmark

NameSAT05/crafted/lynce05/social-golfer-problem/
sgp_5-6-8.shuffled-as.sat05-2669.cnf
MD5SUM4229175ff4382a4625ce92a19d20dde4
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark135.322
Satisfiable
(Un)Satisfiability was proved
Number of variables58560
Number of clauses778800
Sum of the clauses size1766280
Maximum clause length48
Minimum clause length1
Number of clauses of size 196
Number of clauses of size 2599208
Number of clauses of size 3177096
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52400

Solver Data 

2.59/2.66	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265478-1304102254.cnf 
2.59/2.66	c Using SATTIMEOUT of 4800
2.59/2.66	c Using SATRAM of 15500
2.59/2.66	c Size of subset node = 32
2.59/2.66	c Variables = 58560
2.59/2.66	c Clauses = 778800
2.59/2.66	c Literals = 1766280
2.59/2.66	c Reduced size:
2.59/2.66	c Variables = 58560
2.59/2.66	c Clauses = 443760
2.59/2.66	c Literals = 989112
2.59/2.66	c a2
3.09/3.16	c a1
5.89/5.92	c Independent sub-problems = 1
5.89/5.92	c Sub-problem with 29376 variables and 443760 clauses
5.89/5.92	c using permutation method 4
5.89/5.92	c count = 1
5.89/5.92	c Ordering finished 5.840 seconds
5.89/5.92	c mapped 5.910 seconds
5.89/5.92	c permuted 5.910 seconds
5.89/5.92	c about to sort
5.89/5.96	c sorted
7.19/7.29	c clauses now 443760
7.19/7.29	c init finished 6.340 seconds
7.19/7.29	c tree built 7.280 seconds
7.19/7.29	c 
7.19/7.29	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
7.19/7.29	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
7.19/7.29	c ................................ | ..........                    |                                  7.3         0.0             
7.30/7.30	c ................................ | ..........                  1 |                                  7.3         0.1             
7.30/7.30	c ................................ | ..........                  2 |                                  7.3         0.3             
7.30/7.30	c ................................ | ..........                  3 |                                  7.3         0.4             
7.30/7.30	c ................................ | ..........                  4 |                                  7.3         0.5             
7.30/7.30	c ................................ | ..........                  5 |                                  7.3         0.7             
7.30/7.31	c ................................ | ..........               1000 |              184                 7.3       136.1             
95.97/29.51	c ................................ | ..........            2317000 | 27786        318089             96.0     24127.9      1040411
191.82/53.55	c ................................ | ..........            4527000 | 66572        608748            192.0     23576.9      2203728
287.69/77.57	c ................................ | ..........            6368000 | 100651       874512            288.0     22111.1      3331177
383.95/101.61	c ................................ | ..........            8453000 | 137272       1131389           384.0     22012.4      4293944
479.81/125.65	c ................................ | *.........           10300000 | 167159       1400875           480.0     21456.5      5064852
575.69/149.68	c ................................ | *.........           12334000 | 192281       1654966           576.0     21412.1      6124023
671.96/173.72	c ................................ | *.........           14054000 | 220764       1871478           672.0     20912.4      7071724
767.81/197.75	c ................................ | *.........           15670000 | 248990       2084558           768.0     20402.8      8402506
863.67/221.78	c ................................ | *.........           17003000 | 274775       2261900           864.0     19679.4      9538377
959.94/245.82	c ................................ | **........           18307000 | 304636       2439459           960.0     19069.2     10678667
1055.81/269.84	c ................................ | **........           19510000 | 338532       2578769          1056.0     18475.4     11681953
1151.67/293.89	c ................................ | **........           20726000 | 369098       2712626          1152.1     17990.2     12807464
1247.96/317.93	c ................................ | **........           21839000 | 395897       2831666          1248.1     17498.2     13409288
1343.81/341.96	c ................................ | **........           23226000 | 423954       2978251          1344.1     17280.5     14025714
1439.68/365.99	c ................................ | ***.......           24251000 | 451537       3100592          1440.0     16840.4     14511656
1535.94/390.03	c ................................ | ***.......           25260000 | 480844       3232408          1536.1     16444.3     15283761
1631.81/414.05	c ................................ | ***.......           26419000 | 511868       3388797          1632.0     16187.7     16403567
1727.68/438.08	c ................................ | ***.......           27589000 | 539830       3548093          1728.0     15965.7     17079230
1823.94/462.12	c ................................ | ***.......           28731000 | 562788       3699568          1824.0     15751.5     17516244
1919.83/486.17	c ................................ | ****......           29583000 | 583129       3811371          1920.1     15407.2     18166489
2015.68/510.19	c ................................ | ****......           30543000 | 612247       3931066          2016.0     15150.0     18653268
2111.95/534.22	c ................................ | ****......           31439000 | 638713       4051078          2112.1     14885.5     19101825
2207.82/558.25	c ................................ | ****......           32495000 | 665920       4187714          2208.0     14716.8     19680741
2304.08/582.30	c ................................ | ****......           33892000 | 689610       4380813          2304.1     14709.6     20176763
2399.95/606.32	c ................................ | *****.....           35155000 | 710314       4549780          2400.1     14647.6     20606543
2495.83/630.35	c ................................ | *****.....           36202000 | 735436       4696264          2496.0     14503.9     21204052
2592.10/654.44	c ................................ | *****.....           37467000 | 758909       4868367          2592.3     14453.4     21665218
2687.95/678.45	c ................................ | *****.....           38537000 | 784502       5002864          2688.2     14335.9     22035927
2783.82/702.45	c ................................ | *****.....           39334000 | 808063       5105827          2784.0     14128.4     22339741
2880.09/726.51	c ................................ | ******....           40367000 | 828361       5236952          2880.1     14015.7     22682281
2975.95/750.51	c ................................ | ******....           42723000 | 852599       5458232          2976.0     14355.8     23160912
3071.81/774.59	c ................................ | ******....           46794000 | 882876       5789600          3072.2     15231.4     23870051
3167.70/798.59	c ................................ | ******....           47505000 | 903660       5889881          3168.1     14994.9     24266037
3263.95/822.62	c ................................ | ******....           48266000 | 924776       5969985          3264.0     14787.2     24684694
3359.82/846.65	c ................................ | *******...           49424000 | 950586       6088979          3360.0     14709.3     25414652
3456.08/870.73	c ................................ | *******...           50284000 | 972298       6177040          3456.2     14548.9     25994858
3551.95/894.71	c ................................ | *******...           51113000 | 994262       6275765          3552.0     14389.8     26937429
3647.81/918.75	c ................................ | *******...           52278000 | 1022433      6394190          3648.0     14330.4     27953692
3743.70/942.78	c ................................ | *******...           53538000 | 1041947      6515701          3744.0     14299.6     28702874
3839.97/966.82	c ................................ | ********..           54879000 | 1060418      6650632          3840.1     14291.2     29608598
3935.83/990.84	c ................................ | ********..           56030000 | 1077043      6763777          3936.0     14235.3     30177901
4032.08/1014.90	c ................................ | ********..           57325000 | 1096021      6893784          4032.1     14217.1     31049486
4127.95/1038.95	c ................................ | ********..           58005000 | 1111411      6987224          4128.2     14051.0     31633910
4223.82/1062.97	c ................................ | ********..           58872000 | 1126363      7079783          4224.1     13937.1     32050912
4320.09/1087.05	c ................................ | *********.           59682000 | 1143448      7163902          4320.3     13814.4     32559320
4415.98/1111.05	c ................................ | *********.           60401000 | 1161466      7273876          4416.2     13677.2     33280745
4511.82/1135.06	c ................................ | *********.           61508000 | 1176919      7389326          4512.1     13631.9     34005371
4607.68/1159.09	c ................................ | *********.           63141000 | 1198572      7544899          4608.1     13702.3     34811260
4703.95/1183.16	c ................................ | *********.           64369000 | 1219963      7682142          4704.2     13683.2     35894126

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265478-1304102254/watcher-3265478-1304102254 -o /tmp/evaluation-result-3265478-1304102254/solver-3265478-1304102254 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265478-1304102254.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.13 7.87 7.86 2/177 9731
/proc/meminfo: memFree=29712804/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=9540 CPUtime=0 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 443 0 0 0 0 0 0 0 21 0 1 0 502640788 9768960 385 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 2385 385 147 36 0 290 0

[startup+0.0922251 s]
/proc/loadavg: 7.13 7.87 7.86 2/177 9731
/proc/meminfo: memFree=29712804/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=23512 CPUtime=0.08 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 3925 0 0 0 8 0 0 0 21 0 1 0 502640788 24076288 3867 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 5878 3868 148 36 0 3783 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 23512

[startup+0.100219 s]
/proc/loadavg: 7.13 7.87 7.86 2/177 9731
/proc/meminfo: memFree=29712804/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=24700 CPUtime=0.09 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 4246 0 0 0 8 1 0 0 21 0 1 0 502640788 25292800 4188 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 6175 4189 148 36 0 4080 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 24700

[startup+0.300189 s]
/proc/loadavg: 7.13 7.87 7.86 2/177 9731
/proc/meminfo: memFree=29712804/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=57492 CPUtime=0.29 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 12322 0 0 0 27 2 0 0 22 0 1 0 502640788 58871808 12264 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261616384 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 14373 12266 148 36 0 12278 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 57492

[startup+0.701147 s]
/proc/loadavg: 7.13 7.87 7.86 2/177 9731
/proc/meminfo: memFree=29712804/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=104628 CPUtime=0.69 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 23577 0 0 0 66 3 0 0 24 0 1 0 502640788 107139072 23519 33554432000 4194304 4338100 140737005027584 18446744073709551615 4221388 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 26157 23519 150 36 0 24062 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 104628

[startup+1.50099 s]
/proc/loadavg: 7.13 7.87 7.86 7/182 9736
/proc/meminfo: memFree=29575008/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=104628 CPUtime=1.49 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 23577 0 0 0 146 3 0 0 25 0 1 0 502640788 107139072 23519 33554432000 4194304 4338100 140737005027584 18446744073709551615 4221512 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 26157 23519 150 36 0 24062 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 104628

[startup+3.10069 s]
/proc/loadavg: 7.13 7.87 7.86 6/182 9736
/proc/meminfo: memFree=29536812/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=148816 CPUtime=3.09 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 34625 0 0 0 304 5 0 0 25 0 1 0 502640788 152387584 34566 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261011598 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 37204 34566 153 36 0 35109 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 148816

[startup+6.30113 s]
/proc/loadavg: 6.96 7.82 7.84 6/182 9737
/proc/meminfo: memFree=29370644/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=426788 CPUtime=6.29 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) R 9729 9731 9165 0 -1 4202496 106599 0 0 0 615 14 0 0 25 0 1 0 502640788 437030912 103052 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/9731/statm: 106697 103053 163 36 0 104602 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 426788

[startup+12.7009 s]
/proc/loadavg: 7.05 7.83 7.84 9/186 9741
/proc/meminfo: memFree=28676140/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=882204 CPUtime=28.85 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 163206 0 0 0 2748 137 0 0 25 0 5 0 502640788 903376896 159658 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 220551 159658 176 36 0 218456 0
[pid=9731/tid=9738] ppid=9729 vsize=882204 CPUtime=5.39 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) S 9729 9731 9165 0 -1 4202560 2146 0 0 0 510 29 0 0 25 0 5 0 502641517 903376896 159658 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442164 0 0 4096 0 18446604471117034432 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=882204 CPUtime=5.39 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 2018 0 0 0 509 30 0 0 25 0 5 0 502641517 903376896 159658 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=882204 CPUtime=5.36 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 2081 0 0 0 512 24 0 0 25 0 5 0 502641517 903376896 159658 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273429198 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=882204 CPUtime=5.39 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 2187 0 0 0 508 31 0 0 25 0 5 0 502641517 903376896 159658 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 28.85
Current children cumulated vsize (KiB) 882204

[startup+25.5007 s]
/proc/loadavg: 7.40 7.87 7.85 9/186 9741
/proc/meminfo: memFree=28287004/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=891248 CPUtime=79.98 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 169026 0 0 0 7581 417 0 0 25 0 5 0 502640788 912637952 165478 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 222812 165478 176 36 0 220717 0
[pid=9731/tid=9738] ppid=9729 vsize=891248 CPUtime=18.2 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 3077 0 0 0 1712 108 0 0 25 0 5 0 502641517 912637952 165478 33554432000 4194304 4338100 140737005027584 18446744073709551615 4284123 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=891248 CPUtime=18.19 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 3663 0 0 0 1725 94 0 0 25 0 5 0 502641517 912637952 165478 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=891248 CPUtime=18.09 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 2951 0 0 0 1716 93 0 0 25 0 5 0 502641517 912637952 165478 33554432000 4194304 4338100 140737005027584 18446744073709551615 4283712 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=891248 CPUtime=18.19 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 4561 0 0 0 1719 100 0 0 25 0 5 0 502641517 912637952 165478 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 79.98
Current children cumulated vsize (KiB) 891248

[startup+51.1011 s]
/proc/loadavg: 7.60 7.88 7.86 9/186 9742
/proc/meminfo: memFree=27684256/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=897100 CPUtime=182.23 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 180390 0 0 0 17284 939 0 0 25 0 5 0 502640788 918630400 176842 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 224275 176842 176 36 0 222180 0
[pid=9731/tid=9738] ppid=9729 vsize=897100 CPUtime=43.8 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) S 9729 9731 9165 0 -1 4202560 6714 0 0 0 4136 244 0 0 25 0 5 0 502641517 918630400 176842 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442164 0 0 4096 0 18446604471117034432 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=897100 CPUtime=43.78 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 5625 0 0 0 4156 222 0 0 25 0 5 0 502641517 918630400 176842 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=897100 CPUtime=43.55 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 6099 0 0 0 4134 221 0 0 25 0 5 0 502641517 918630400 176842 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273429198 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=897100 CPUtime=43.78 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 7178 0 0 0 4149 229 0 0 25 0 5 0 502641517 918630400 176842 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 182.23
Current children cumulated vsize (KiB) 897100

[startup+102.307 s]
/proc/loadavg: 7.99 7.94 7.88 9/186 9743
/proc/meminfo: memFree=26858012/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=974384 CPUtime=386.77 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 201430 0 0 0 36700 1977 0 0 25 0 5 0 502640788 997769216 197882 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 243596 197882 176 36 0 241501 0
[pid=9731/tid=9738] ppid=9729 vsize=974384 CPUtime=94.85 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 11905 0 0 0 8958 527 0 0 25 0 5 0 502641517 997769216 197882 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261011686 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=974384 CPUtime=94.98 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 10118 0 0 0 9037 461 0 0 25 0 5 0 502641517 997769216 197882 33554432000 4194304 4338100 140737005027584 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=974384 CPUtime=94.68 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 11589 0 0 0 8973 495 0 0 25 0 5 0 502641517 997769216 197882 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261011583 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=974384 CPUtime=94.94 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 13044 0 0 0 9023 471 0 0 25 0 5 0 502641517 997769216 197882 33554432000 4194304 4338100 140737005027584 18446744073709551615 4284433 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 386.77
Current children cumulated vsize (KiB) 974384

[startup+162.3 s]
/proc/loadavg: 8.08 7.98 7.89 9/186 9745
/proc/meminfo: memFree=26112716/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1017864 CPUtime=626.41 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 223653 0 0 0 59446 3195 0 0 25 0 5 0 502640788 1042292736 220105 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 254466 220105 176 36 0 252371 0
[pid=9731/tid=9738] ppid=9729 vsize=1017864 CPUtime=154.84 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 16988 0 0 0 14622 862 0 0 25 0 5 0 502641517 1042292736 220105 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1017864 CPUtime=154.87 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 15426 0 0 0 14751 736 0 0 25 0 5 0 502641517 1042292736 220105 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1017864 CPUtime=154.67 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 16627 0 0 0 14643 824 0 0 19 0 5 0 502641517 1042292736 220105 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1017864 CPUtime=154.7 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 19838 0 0 0 14720 750 0 0 25 0 5 0 502641517 1042292736 220105 33554432000 4194304 4338100 140737005027584 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 626.41
Current children cumulated vsize (KiB) 1017864

[startup+222.301 s]
/proc/loadavg: 8.12 8.01 7.91 9/186 9746
/proc/meminfo: memFree=25515228/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1178196 CPUtime=866.07 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 250257 0 0 0 82417 4190 0 0 25 0 5 0 502640788 1206472704 246709 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 294549 246709 176 36 0 292454 0
[pid=9731/tid=9738] ppid=9729 vsize=1178196 CPUtime=214.84 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 20819 0 0 0 20340 1144 0 0 25 0 5 0 502641517 1206472704 246709 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1178196 CPUtime=214.81 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 23426 0 0 0 20523 958 0 0 25 0 5 0 502641517 1206472704 246709 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1178196 CPUtime=214.67 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 21058 0 0 0 20378 1089 0 0 25 0 5 0 502641517 1206472704 246709 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1178196 CPUtime=214.43 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 30180 0 0 0 20467 976 0 0 25 0 5 0 502641517 1206472704 246709 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 866.07
Current children cumulated vsize (KiB) 1178196

[startup+282.301 s]
/proc/loadavg: 8.09 8.02 7.91 9/186 9748
/proc/meminfo: memFree=24885732/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1262800 CPUtime=1105.74 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 273926 0 0 0 105545 5029 0 0 25 0 5 0 502640788 1293107200 270378 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 315700 270378 176 36 0 313605 0
[pid=9731/tid=9738] ppid=9729 vsize=1262800 CPUtime=274.84 cores=1,3,5,7

################
# More data... #
################

[pid=9731/tid=9739] ppid=9729 vsize=1600528 CPUtime=694.54 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 52221 0 0 0 67172 2282 0 0 25 0 5 0 502641517 1638940672 366496 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1600528 CPUtime=693.46 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 47232 0 0 0 66712 2634 0 0 25 0 5 0 502641517 1638940672 366496 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1600528 CPUtime=693.46 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 65658 0 0 0 67053 2293 0 0 25 0 5 0 502641517 1638940672 366496 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2783.42
Current children cumulated vsize (KiB) 1600528

[startup+762.306 s]
/proc/loadavg: 8.03 8.04 7.95 9/186 9761
/proc/meminfo: memFree=22136152/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1690732 CPUtime=3023.1 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 384148 0 0 0 291634 10676 0 0 25 0 5 0 502640788 1731309568 380600 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 422683 380600 176 36 0 420588 0
[pid=9731/tid=9738] ppid=9729 vsize=1690732 CPUtime=754.66 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 54491 0 0 0 72541 2925 0 0 25 0 5 0 502641517 1731309568 380600 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273429286 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1690732 CPUtime=754.54 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 55411 0 0 0 72994 2460 0 0 25 0 5 0 502641517 1731309568 380600 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1690732 CPUtime=753.13 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 51391 0 0 0 72511 2802 0 0 25 0 5 0 502641517 1731309568 380600 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1690732 CPUtime=753.47 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 68081 0 0 0 72880 2467 0 0 25 0 5 0 502641517 1731309568 380600 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442164 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3023.1
Current children cumulated vsize (KiB) 1690732

[startup+822.301 s]
/proc/loadavg: 8.13 8.07 7.96 9/186 9762
/proc/meminfo: memFree=21610580/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1770756 CPUtime=3262.75 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 394869 0 0 0 315040 11235 0 0 25 0 5 0 502640788 1813254144 391321 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 442689 391321 176 36 0 440594 0
[pid=9731/tid=9738] ppid=9729 vsize=1770756 CPUtime=814.65 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 57374 0 0 0 78411 3054 0 0 25 0 5 0 502641517 1813254144 391321 33554432000 4194304 4338100 140737005027584 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1770756 CPUtime=814.52 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 58567 0 0 0 78833 2619 0 0 25 0 5 0 502641517 1813254144 391321 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1770756 CPUtime=812.8 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 53274 0 0 0 78356 2924 0 0 25 0 5 0 502641517 1813254144 391321 33554432000 4194304 4338100 140737005027584 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1770756 CPUtime=813.45 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 70880 0 0 0 78730 2615 0 0 25 0 5 0 502641517 1813254144 391321 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3262.75
Current children cumulated vsize (KiB) 1770756

[startup+882.301 s]
/proc/loadavg: 8.11 8.07 7.97 9/186 9764
/proc/meminfo: memFree=21066756/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1854364 CPUtime=3502.42 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 410470 0 0 0 338476 11766 0 0 25 0 5 0 502640788 1898868736 406922 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 463591 406922 176 36 0 461496 0
[pid=9731/tid=9738] ppid=9729 vsize=1854364 CPUtime=874.65 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 61526 0 0 0 84271 3194 0 0 25 0 5 0 502641517 1898868736 406922 33554432000 4194304 4338100 140737005027584 18446744073709551615 4234400 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1854364 CPUtime=874.52 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 61310 0 0 0 84702 2750 0 0 25 0 5 0 502641517 1898868736 406922 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261012206 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1854364 CPUtime=872.47 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 56214 0 0 0 84190 3057 0 0 25 0 5 0 502641517 1898868736 406922 33554432000 4194304 4338100 140737005027584 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1854364 CPUtime=873.46 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 76646 0 0 0 84604 2742 0 0 25 0 5 0 502641517 1898868736 406923 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261011681 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3502.42
Current children cumulated vsize (KiB) 1854364

[startup+942.3 s]
/proc/loadavg: 8.14 8.08 7.98 9/186 9765
/proc/meminfo: memFree=20605688/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1873160 CPUtime=3742.08 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 429657 0 0 0 361796 12412 0 0 25 0 5 0 502640788 1918115840 426109 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 468290 426109 176 36 0 466195 0
[pid=9731/tid=9738] ppid=9729 vsize=1873160 CPUtime=934.65 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 68118 0 0 0 90100 3365 0 0 25 0 5 0 502641517 1918115840 426109 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1873160 CPUtime=934.51 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 65873 0 0 0 90546 2905 0 0 25 0 5 0 502641517 1918115840 426109 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1873160 CPUtime=932.15 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 59379 0 0 0 89992 3223 0 0 25 0 5 0 502641517 1918115840 426109 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1873160 CPUtime=933.46 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 81513 0 0 0 90450 2896 0 0 25 0 5 0 502641517 1918115840 426109 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3742.08
Current children cumulated vsize (KiB) 1873160

[startup+1002.3 s]
/proc/loadavg: 8.14 8.09 7.99 9/186 9767
/proc/meminfo: memFree=20192436/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1894280 CPUtime=3981.75 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 445817 0 0 0 385142 13033 0 0 25 0 5 0 502640788 1939742720 442269 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 473570 442269 176 36 0 471475 0
[pid=9731/tid=9738] ppid=9729 vsize=1894280 CPUtime=994.65 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 71718 0 0 0 95944 3521 0 0 25 0 5 0 502641517 1939742720 442269 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1894280 CPUtime=994.5 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 69513 0 0 0 96381 3069 0 0 25 0 5 0 502641517 1939742720 442269 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261313014 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1894280 CPUtime=991.82 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 60745 0 0 0 95816 3366 0 0 25 0 5 0 502641517 1939742720 442270 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1894280 CPUtime=993.46 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 89067 0 0 0 96292 3054 0 0 25 0 5 0 502641517 1939742720 442270 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3981.75
Current children cumulated vsize (KiB) 1894280

[startup+1062.3 s]
/proc/loadavg: 8.14 8.10 8.00 9/186 9769
/proc/meminfo: memFree=19849772/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=1968640 CPUtime=4221.42 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 459447 0 0 0 408578 13564 0 0 25 0 5 0 502640788 2015887360 455899 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 492160 455899 176 36 0 490065 0
[pid=9731/tid=9738] ppid=9729 vsize=1968640 CPUtime=1054.66 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 74457 0 0 0 101822 3644 0 0 25 0 5 0 502641517 2015887360 455899 33554432000 4194304 4338100 140737005027584 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=1968640 CPUtime=1054.5 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 72971 0 0 0 102227 3223 0 0 25 0 5 0 502641517 2015887360 455899 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=1968640 CPUtime=1051.49 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 62811 0 0 0 101676 3473 0 0 25 0 5 0 502641517 2015887360 455899 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=1968640 CPUtime=1053.45 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 94434 0 0 0 102144 3201 0 0 25 0 5 0 502641517 2015887360 455899 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4221.42
Current children cumulated vsize (KiB) 1968640

[startup+1122.3 s]
/proc/loadavg: 8.10 8.10 8.00 9/186 9770
/proc/meminfo: memFree=19362388/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=2057992 CPUtime=4461.09 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 473825 0 0 0 432022 14087 0 0 25 0 5 0 502640788 2107383808 470277 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 514498 470277 176 36 0 512403 0
[pid=9731/tid=9738] ppid=9729 vsize=2057992 CPUtime=1114.66 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 77074 0 0 0 107718 3748 0 0 25 0 5 0 502641517 2107383808 470277 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=2057992 CPUtime=1114.49 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 76386 0 0 0 108069 3380 0 0 25 0 5 0 502641517 2107383808 470277 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=2057992 CPUtime=1111.16 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 64699 0 0 0 107540 3576 0 0 25 0 5 0 502641517 2107383808 470277 33554432000 4194304 4338100 140737005027584 18446744073709551615 254261011898 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=2057992 CPUtime=1113.46 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 100892 0 0 0 107986 3360 0 0 25 0 5 0 502641517 2107383808 470277 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4461.09
Current children cumulated vsize (KiB) 2057992

[startup+1182.3 s]
/proc/loadavg: 8.11 8.11 8.01 9/186 9772
/proc/meminfo: memFree=18984128/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=2141856 CPUtime=4700.76 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 492841 0 0 0 455422 14654 0 0 25 0 5 0 502640788 2193260544 489293 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 535464 489293 176 36 0 533369 0
[pid=9731/tid=9738] ppid=9729 vsize=2141856 CPUtime=1174.61 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 82702 0 0 0 113585 3876 0 0 25 0 5 0 502641517 2193260544 489293 33554432000 4194304 4338100 140737005027584 18446744073709551615 4256288 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=2141856 CPUtime=1174.5 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 80167 0 0 0 113911 3539 0 0 25 0 5 0 502641517 2193260544 489293 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=2141856 CPUtime=1170.88 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 68889 0 0 0 113393 3695 0 0 25 0 5 0 502641517 2193260544 489293 33554432000 4194304 4338100 140737005027584 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=2141856 CPUtime=1173.45 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 106309 0 0 0 113824 3521 0 0 25 0 5 0 502641517 2193260544 489293 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4700.76
Current children cumulated vsize (KiB) 2141856



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1207.2 s]
/proc/loadavg: 7.75 8.03 7.98 5/176 9791
/proc/meminfo: memFree=27740524/32951124 swapFree=67111348/67111528
[pid=9731] ppid=9729 vsize=2214928 CPUtime=4800.24 cores=1,3,5,7
/proc/9731/stat : 9731 (ttsth-5-4) S 9729 9731 9165 0 -1 4202496 500405 0 0 0 465153 14871 0 0 25 0 5 0 502640788 2268086272 496857 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/9731/statm: 553732 496857 176 36 0 551637 0
[pid=9731/tid=9738] ppid=9729 vsize=2214928 CPUtime=1199.53 cores=1,3,5,7
/proc/9731/task/9738/stat : 9738 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 84377 0 0 0 116032 3921 0 0 25 0 5 0 502641517 2268086272 496857 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=9731/tid=9739] ppid=9729 vsize=2214928 CPUtime=1199.39 cores=1,3,5,7
/proc/9731/task/9739/stat : 9739 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 82193 0 0 0 116335 3604 0 0 25 0 5 0 502641517 2268086272 496857 33554432000 4194304 4338100 140737005027584 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=9731/tid=9740] ppid=9729 vsize=2214928 CPUtime=1195.64 cores=1,3,5,7
/proc/9731/task/9740/stat : 9740 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 70030 0 0 0 115829 3735 0 0 25 0 5 0 502641517 2268086272 496857 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=9731/tid=9741] ppid=9729 vsize=2214928 CPUtime=1198.37 cores=1,3,5,7
/proc/9731/task/9741/stat : 9741 (ttsth-5-4) R 9729 9731 9165 0 -1 4202560 109031 0 0 0 116248 3589 0 0 25 0 5 0 502641517 2268086272 496857 33554432000 4194304 4338100 140737005027584 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 2214928

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1207.3
CPU time (s): 4800.35
CPU user time (s): 4651.56
CPU system time (s): 148.8
CPU usage (%): 397.611
Max. virtual memory (cumulated for all children) (KiB): 2214928

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4651.55
system time used= 148.8
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 500405
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 20166
involuntary context switches= 32364

runsolver used 1.86172 second user time and 4.73428 second system time

The end

Launcher Data 

Begin job on node140 at 2011-04-29 20:37:35
IDJOB=3265478
IDBENCH=85640
IDSOLVER=1654
FILE ID=node140/3265478-1304102254
RUNJOBID= node140-1304102254-9712
PBS_JOBID= 13170605
Free space on /tmp= 72652 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/lynce05/social-golfer-problem/sgp_5-6-8.shuffled-as.sat05-2669.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265478-1304102254/watcher-3265478-1304102254 -o /tmp/evaluation-result-3265478-1304102254/solver-3265478-1304102254 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265478-1304102254.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 4229175ff4382a4625ce92a19d20dde4
RANDOM SEED=247380789

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29713208 kB
Buffers:       1006012 kB
Cached:        1660164 kB
SwapCached:        180 kB
Active:         293820 kB
Inactive:      2453268 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29713208 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:           13296 kB
Writeback:           0 kB
AnonPages:       80432 kB
Mapped:          15204 kB
Slab:           427312 kB
PageTables:       4752 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   222012 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72652 MiB
End job on node140 at 2011-04-29 20:57:44