Trace number 3250296

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.16 1205.18

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-5-5_34308.cnf
MD5SUM1ef198d3f2d717e33f4f2ba124eb7eb2
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables34306
Number of clauses708168
Sum of the clauses size3540840
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 5708168
Number of clauses of size over 50

Solver Data 

0.89/0.98	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250296-1304089698.cnf 
0.89/0.98	c Using SATTIMEOUT of 4800
0.89/0.98	c Using SATRAM of 15500
0.89/0.98	c Size of subset node = 32
0.89/0.98	c Variables = 34306
0.89/0.98	c Clauses = 708168
0.89/0.98	c Literals = 3540840
0.89/0.98	c Reduced size:
0.89/0.98	c Variables = 34306
0.89/0.98	c Clauses = 708168
0.89/0.98	c Literals = 3540840
0.89/0.98	c a2
3.38/3.41	c a1
4.09/4.12	c Independent sub-problems = 5
4.09/4.12	c Sub-problem with 4257 variables and 85800 clauses
4.09/4.12	c using permutation method 4
4.09/4.12	c count = 1
4.09/4.12	c Ordering finished 4.080 seconds
4.09/4.12	c mapped 4.110 seconds
4.09/4.12	c permuted 4.110 seconds
4.09/4.12	c about to sort
4.09/4.13	c sorted
4.49/4.52	c clauses now 85800
4.49/4.52	c init finished 4.170 seconds
4.49/4.52	c tree built 4.510 seconds
4.49/4.52	c 
4.49/4.52	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
4.49/4.52	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
4.49/4.52	c ................................ | ..........                    |                                  4.5         0.0             
4.49/4.52	c ................................ | ..........                  1 |                                  4.5         0.2             
4.49/4.52	c ................................ | ..........                  2 |                                  4.5         0.4             
4.49/4.52	c ................................ | ..........                  3 |                                  4.5         0.7             
4.49/4.52	c ................................ | ..........                  4 |                                  4.5         0.9             
4.49/4.52	c ................................ | ..........                  5 |                                  4.5         1.1             
4.49/4.54	c ................................ | ..........               1000 |              149                 4.6       218.8           13
95.89/27.44	c ................................ | ..........            1431000 | 35707        1052663            96.0     14901.6       716525
191.73/51.47	c ................................ | ..........            2837000 | 80565        1953053           192.0     14775.3      2170169
288.00/75.50	c ................................ | ..........            4111000 | 131633       2820923           288.0     14273.8      3708232
383.86/99.54	c ................................ | ..........            5330000 | 178437       3637762           384.0     13879.1      5024571
479.73/123.57	c ................................ | *.........            6441000 | 219812       4346779           480.0     13417.9      6224156
575.99/147.62	c ................................ | *.........            7468000 | 270257       4916080           576.1     12963.7      8159496
671.87/171.67	c ................................ | *.........            8508000 | 315335       5560294           672.1     12658.5      9471533
767.72/195.69	c ................................ | *.........            9325000 | 353700       6161832           768.1     12140.7     11103257
863.99/219.73	c ................................ | *.........            9981000 | 387896       6671973           864.1     11550.7     13317298
959.85/243.74	c ................................ | **........           10603000 | 420598       7105258           960.0     11044.3     15324388
1055.72/267.78	c ................................ | **........           11353000 | 458757       7561579          1056.0     10750.4     16784366
1151.99/291.81	c ................................ | **........           12082000 | 493478       8060753          1152.1     10487.3     18267204
1247.87/315.84	c ................................ | **........           12806000 | 522904       8552201          1248.0     10261.0     19705587
1343.75/339.89	c ................................ | **........           13676000 | 567287       9064943          1344.1     10174.8     21016239
1440.00/363.90	c ................................ | ***.......           14438000 | 609147       9515502          1440.0     10026.3     23234354
1535.86/387.94	c ................................ | ***.......           14966000 | 640309       9818681          1536.0      9743.3     26599341
1631.73/411.98	c ................................ | ***.......           15692000 | 671678       10237270         1632.1      9614.8     27665010
1728.00/436.02	c ................................ | ***.......           16345000 | 701210       10644244         1728.1      9458.4     29249906
1823.86/460.04	c ................................ | ***.......           16900000 | 725817       10998845         1824.0      9265.1     30461600
1920.13/484.11	c ................................ | ****......           17452000 | 753398       11365816         1920.2      9088.6     31999195
2016.02/508.12	c ................................ | ****......           18081000 | 780803       11695658         2016.1      8968.4     33416640
2111.88/532.15	c ................................ | ****......           18745000 | 809913       12003009         2112.1      8875.1     35530511
2208.14/556.24	c ................................ | ****......           19233000 | 837297       12292627         2208.3      8709.4     37537052
2304.02/580.21	c ................................ | ****......           19759000 | 861702       12597373         2304.1      8575.7     39357572
2399.88/604.23	c ................................ | *****.....           20199000 | 882649       12849276         2400.0      8416.1     40830115
2496.15/628.37	c ................................ | *****.....           20572000 | 903603       13080562         2496.4      8240.5     42486561
2592.04/652.32	c ................................ | *****.....           20929000 | 925466       13282720         2592.1      8074.1     44612107
2687.89/676.35	c ................................ | *****.....           21241000 | 943705       13463730         2688.1      7901.9     46204927
2783.76/700.36	c ................................ | *****.....           21522000 | 961247       13642229         2784.0      7730.5     47982347
2880.03/724.44	c ................................ | ******....           21760000 | 975360       13801353         2880.2      7555.0     49571121
2975.89/748.46	c ................................ | ******....           21973000 | 986476       13944963         2976.2      7383.0     51195751
3072.16/772.52	c ................................ | ******....           22166000 | 998183       14107495         3072.3      7214.9     52476625
3168.02/796.53	c ................................ | ******....           22356000 | 1009837      14249582         3168.2      7056.5     53737318
3263.92/820.55	c ................................ | ******....           22630000 | 1026097      14432594         3264.1      6933.0     54902489
3360.17/844.62	c ................................ | *******...           22933000 | 1047071      14617326         3360.3      6824.8     56704518
3456.05/868.63	c ................................ | *******...           23195000 | 1063725      14758660         3456.2      6711.2     58498420
3551.91/892.67	c ................................ | *******...           23454000 | 1080482      14928934         3552.2      6602.6     60114389
3648.18/916.77	c ................................ | *******...           23686000 | 1096323      15085251         3648.5      6492.0     61785397
3744.05/940.79	c ................................ | *******...           23921000 | 1110991      15256304         3744.4      6388.5     62963031
3839.95/964.73	c ................................ | ********..           24130000 | 1126494      15405076         3840.1      6283.7     64503826
3935.80/988.76	c ................................ | ********..           24348000 | 1141472      15554196         3936.1      6185.9     65909348
4032.06/1012.85	c ................................ | ********..           24488000 | 1153062      15681698         4032.3      6073.0     66877474
4128.34/1036.93	c ................................ | ********..           24615000 | 1163023      15794407         4128.5      5962.3     67624159
4223.80/1060.85	c ................................ | ********..           24854000 | 1181427      15930821         4224.0      5884.0     69277679
4319.67/1084.89	c ................................ | *********.           25082000 | 1195927      16067644         4320.1      5805.9     70596187
4415.94/1108.98	c ................................ | *********.           25555000 | 1218844      16281620         4416.3      5786.6     71939591
4511.83/1132.97	c ................................ | *********.           25996000 | 1238031      16523710         4512.1      5761.4     73290319
4607.68/1156.98	c ................................ | *********.           26437000 | 1260500      16751099         4608.0      5737.2     74260979
4703.95/1181.02	c ................................ | *********.           26985000 | 1288859      17009289         4704.1      5736.5     75403727
4799.81/1205.05	c CPU limit (4800) reached
4799.81/1205.05	c Time = 4800.03 seconds
4799.81/1205.05	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250296-1304089698/watcher-3250296-1304089698 -o /tmp/evaluation-result-3250296-1304089698/solver-3250296-1304089698 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250296-1304089698.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.73 7.93 7.85 5/181 21004
/proc/meminfo: memFree=28434252/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=9244 CPUtime=0 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) R 21002 21004 20765 0 -1 4202496 368 0 0 0 0 0 0 0 22 0 1 0 599200499 9465856 310 33554432000 4194304 4338100 140735894165312 18446744073709551615 260020395776 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/21004/statm: 2311 311 146 36 0 216 0

[startup+0.0273321 s]
/proc/loadavg: 7.73 7.93 7.85 5/181 21004
/proc/meminfo: memFree=28434252/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=11424 CPUtime=0.02 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) R 21002 21004 20765 0 -1 4202496 929 0 0 0 2 0 0 0 22 0 1 0 599200499 11698176 871 33554432000 4194304 4338100 140735894165312 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/21004/statm: 2856 872 147 36 0 761 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 11424

[startup+0.10032 s]
/proc/loadavg: 7.73 7.93 7.85 5/181 21004
/proc/meminfo: memFree=28434252/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=18928 CPUtime=0.09 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) R 21002 21004 20765 0 -1 4202496 2780 0 0 0 9 0 0 0 22 0 1 0 599200499 19382272 2722 33554432000 4194304 4338100 140735894165312 18446744073709551615 260020395776 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/21004/statm: 4732 2723 147 36 0 2637 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18928

[startup+0.300283 s]
/proc/loadavg: 7.73 7.93 7.85 5/181 21004
/proc/meminfo: memFree=28434252/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=38852 CPUtime=0.29 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) R 21002 21004 20765 0 -1 4202496 7678 0 0 0 29 0 0 0 23 0 1 0 599200499 39784448 7620 33554432000 4194304 4338100 140735894165312 18446744073709551615 260020395776 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/21004/statm: 9713 7621 147 36 0 7618 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 38852

[startup+0.700221 s]
/proc/loadavg: 7.73 7.93 7.85 5/181 21004
/proc/meminfo: memFree=28434252/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=78456 CPUtime=0.69 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) R 21002 21004 20765 0 -1 4202496 17476 0 0 0 68 1 0 0 25 0 1 0 599200499 80338944 17418 33554432000 4194304 4338100 140735894165312 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/21004/statm: 19614 17419 148 36 0 17519 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 78456

[startup+1.50107 s]
/proc/loadavg: 7.73 7.93 7.85 6/182 21005
/proc/meminfo: memFree=28338896/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=317468 CPUtime=1.49 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) R 21002 21004 20765 0 -1 4202496 76684 0 0 0 139 10 0 0 25 0 1 0 599200499 325087232 76626 33554432000 4194304 4338100 140735894165312 18446744073709551615 260020056773 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/21004/statm: 79367 76633 151 36 0 77272 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 317468

[startup+3.10078 s]
/proc/loadavg: 7.59 7.90 7.84 6/182 21005
/proc/meminfo: memFree=28089408/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=350804 CPUtime=3.08 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) R 21002 21004 20765 0 -1 4202496 84943 0 0 0 298 10 0 0 25 0 1 0 599200499 359223296 84884 33554432000 4194304 4338100 140735894165312 18446744073709551615 260019791569 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/21004/statm: 87701 84884 153 36 0 85606 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 350804

[startup+6.30115 s]
/proc/loadavg: 7.59 7.90 7.84 10/186 21009
/proc/meminfo: memFree=28075736/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=529700 CPUtime=11.59 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 89108 0 0 0 1115 44 0 0 25 0 5 0 599200499 542412800 86761 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 132425 86761 176 36 0 130330 0
[pid=21004/tid=21006] ppid=21002 vsize=529700 CPUtime=1.76 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 169 0 0 0 159 17 0 0 20 0 5 0 599200952 542412800 86761 33554432000 4194304 4338100 140735894165312 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=529700 CPUtime=1.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 138 0 0 0 176 0 0 0 20 0 5 0 599200952 542412800 86761 33554432000 4194304 4338100 140735894165312 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=529700 CPUtime=1.75 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 222 0 0 0 161 14 0 0 20 0 5 0 599200952 542412800 86761 33554432000 4194304 4338100 140735894165312 18446744073709551615 4288180 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=529700 CPUtime=1.76 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 148 0 0 0 176 0 0 0 20 0 5 0 599200952 542412800 86761 33554432000 4194304 4338100 140735894165312 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.59
Current children cumulated vsize (KiB) 529700

[startup+12.701 s]
/proc/loadavg: 7.65 7.91 7.84 9/186 21009
/proc/meminfo: memFree=28053952/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=529700 CPUtime=37.15 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 90376 0 0 0 3595 120 0 0 25 0 5 0 599200499 542412800 88029 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 132425 88029 176 36 0 130330 0
[pid=21004/tid=21006] ppid=21002 vsize=529700 CPUtime=8.13 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 502 0 0 0 756 57 0 0 25 0 5 0 599200952 542412800 88029 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=529700 CPUtime=8.16 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 264 0 0 0 816 0 0 0 25 0 5 0 599200952 542412800 88029 33554432000 4194304 4338100 140735894165312 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=529700 CPUtime=8.13 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 770 0 0 0 764 49 0 0 25 0 5 0 599200952 542412800 88029 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=529700 CPUtime=8.16 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 409 0 0 0 816 0 0 0 25 0 5 0 599200952 542412800 88029 33554432000 4194304 4338100 140735894165312 18446744073709551615 260019791294 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37.15
Current children cumulated vsize (KiB) 529700

[startup+25.5006 s]
/proc/loadavg: 7.71 7.91 7.84 9/186 21010
/proc/meminfo: memFree=28008484/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=595236 CPUtime=88.27 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 94429 0 0 0 8563 264 0 0 25 0 5 0 599200499 609521664 92082 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 148809 92082 176 36 0 146714 0
[pid=21004/tid=21006] ppid=21002 vsize=595236 CPUtime=20.87 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 1189 0 0 0 1953 134 0 0 25 0 5 0 599200952 609521664 92082 33554432000 4194304 4338100 140735894165312 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=595236 CPUtime=20.96 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 865 0 0 0 2095 1 0 0 25 0 5 0 599200952 609521664 92082 33554432000 4194304 4338100 140735894165312 18446744073709551615 4287724 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=595236 CPUtime=20.93 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 1837 0 0 0 1978 115 0 0 25 0 5 0 599200952 609521664 92082 33554432000 4194304 4338100 140735894165312 18446744073709551615 4256374 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=595236 CPUtime=20.96 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 2107 0 0 0 2095 1 0 0 25 0 5 0 599200952 609521664 92082 33554432000 4194304 4338100 140735894165312 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 88.27
Current children cumulated vsize (KiB) 595236

[startup+51.1009 s]
/proc/loadavg: 7.80 7.91 7.85 9/186 21010
/proc/meminfo: memFree=27901976/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=595236 CPUtime=190.54 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 106872 0 0 0 18450 604 0 0 25 0 5 0 599200499 609521664 104525 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 148809 104525 176 36 0 146714 0
[pid=21004/tid=21006] ppid=21002 vsize=595236 CPUtime=46.35 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 4075 0 0 0 4325 310 0 0 25 0 5 0 599200952 609521664 104525 33554432000 4194304 4338100 140735894165312 18446744073709551615 4285837 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=595236 CPUtime=46.56 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 4274 0 0 0 4653 3 0 0 25 0 5 0 599200952 609521664 104525 33554432000 4194304 4338100 140735894165312 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=595236 CPUtime=46.5 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 4197 0 0 0 4377 273 0 0 25 0 5 0 599200952 609521664 104525 33554432000 4194304 4338100 140735894165312 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=595236 CPUtime=46.56 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 5895 0 0 0 4652 4 0 0 25 0 5 0 599200952 609521664 104525 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 190.54
Current children cumulated vsize (KiB) 595236

[startup+102.306 s]
/proc/loadavg: 7.98 7.94 7.86 9/186 21012
/proc/meminfo: memFree=27670620/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=678104 CPUtime=395.06 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 136690 0 0 0 38285 1221 0 0 25 0 5 0 599200499 694378496 134343 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 169526 134343 176 36 0 167431 0
[pid=21004/tid=21006] ppid=21002 vsize=678104 CPUtime=97.5 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 9959 0 0 0 9113 637 0 0 25 0 5 0 599200952 694378496 134343 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=678104 CPUtime=97.77 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 15358 0 0 0 9767 10 0 0 25 0 5 0 599200952 694378496 134343 33554432000 4194304 4338100 140735894165312 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=678104 CPUtime=97.48 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 8377 0 0 0 9195 553 0 0 25 0 5 0 599200952 694378496 134343 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=678104 CPUtime=97.76 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 14565 0 0 0 9768 8 0 0 25 0 5 0 599200952 694378496 134343 33554432000 4194304 4338100 140735894165312 18446744073709551615 4285833 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 395.06
Current children cumulated vsize (KiB) 678104

[startup+162.3 s]
/proc/loadavg: 8.17 8.00 7.88 9/186 21013
/proc/meminfo: memFree=27347892/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=861420 CPUtime=634.71 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 179549 0 0 0 61522 1949 0 0 25 0 5 0 599200499 882094080 177202 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 215355 177202 176 36 0 213260 0
[pid=21004/tid=21006] ppid=21002 vsize=861420 CPUtime=157.34 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 18856 0 0 0 14723 1011 0 0 25 0 5 0 599200952 882094080 177202 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=861420 CPUtime=157.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 27452 0 0 0 15762 14 0 0 25 0 5 0 599200952 882094080 177202 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=861420 CPUtime=157.38 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 17590 0 0 0 14841 897 0 0 25 0 5 0 599200952 882094080 177202 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032221877 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=861420 CPUtime=157.67 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 27220 0 0 0 15754 13 0 0 25 0 5 0 599200952 882094080 177202 33554432000 4194304 4338100 140735894165312 18446744073709551615 4230408 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 634.71
Current children cumulated vsize (KiB) 861420

[startup+222.3 s]
/proc/loadavg: 8.11 8.01 7.89 9/186 21015
/proc/meminfo: memFree=27057820/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=1111724 CPUtime=874.36 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 225497 0 0 0 84934 2502 0 0 25 0 5 0 599200499 1138405376 223150 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 277931 223153 176 36 0 275836 0
[pid=21004/tid=21006] ppid=21002 vsize=1111724 CPUtime=217.31 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 33718 0 0 0 20430 1301 0 0 25 0 5 0 599200952 1138405376 223153 33554432000 4194304 4338100 140735894165312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=1111724 CPUtime=217.77 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 38235 0 0 0 21758 19 0 0 25 0 5 0 599200952 1138405376 223153 33554432000 4194304 4338100 140735894165312 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=1111724 CPUtime=217.11 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 30687 0 0 0 20559 1152 0 0 25 0 5 0 599200952 1138405376 223153 33554432000 4194304 4338100 140735894165312 18446744073709551615 4206909 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=1111724 CPUtime=217.63 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 34429 0 0 0 21745 18 0 0 25 0 5 0 599200952 1138405376 223153 33554432000 4194304 4338100 140735894165312 18446744073709551615 4256051 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 874.36
Current children cumulated vsize (KiB) 1111724

################
# More data... #
################

/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 113510 0 0 0 69716 48 0 0 25 0 5 0 599200952 2353520640 547907 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2791.75
Current children cumulated vsize (KiB) 2298360

[startup+762.307 s]
/proc/loadavg: 8.21 8.11 7.97 9/186 21029
/proc/meminfo: memFree=24686792/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=2532660 CPUtime=3031.44 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 582888 0 0 0 297589 5555 0 0 25 0 5 0 599200499 2593443840 580541 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 633165 580541 176 36 0 631070 0
[pid=21004/tid=21006] ppid=21002 vsize=2532660 CPUtime=755.93 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 123934 0 0 0 72728 2865 0 0 25 0 5 0 599200952 2593443840 580541 33554432000 4194304 4338100 140735894165312 18446744073709551615 260020092416 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=2532660 CPUtime=757.77 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 125426 0 0 0 75726 51 0 0 25 0 5 0 599200952 2593443840 580541 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=2532660 CPUtime=755.54 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 121943 0 0 0 72980 2574 0 0 25 0 5 0 599200952 2593443840 580541 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=2532660 CPUtime=757.66 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 123154 0 0 0 75714 52 0 0 25 0 5 0 599200952 2593443840 580541 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3031.44
Current children cumulated vsize (KiB) 2532660

[startup+822.301 s]
/proc/loadavg: 8.17 8.12 7.98 9/186 21063
/proc/meminfo: memFree=27340456/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=2555928 CPUtime=3271.09 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 610672 0 0 0 321400 5709 0 0 25 0 5 0 599200499 2617270272 608325 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 638982 608325 176 36 0 636887 0
[pid=21004/tid=21006] ppid=21002 vsize=2555928 CPUtime=815.6 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 129310 0 0 0 78619 2941 0 0 25 0 5 0 599200952 2617270272 608325 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=2555928 CPUtime=817.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 129436 0 0 0 81723 53 0 0 25 0 5 0 599200952 2617270272 608325 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=2555928 CPUtime=815.53 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 133044 0 0 0 78905 2648 0 0 25 0 5 0 599200952 2617270272 608325 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=2555928 CPUtime=817.64 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 130451 0 0 0 81710 54 0 0 25 0 5 0 599200952 2617270272 608325 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3271.09
Current children cumulated vsize (KiB) 2555928

[startup+882.301 s]
/proc/loadavg: 8.17 8.12 7.99 9/186 21064
/proc/meminfo: memFree=26631732/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=2787144 CPUtime=3510.76 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 646931 0 0 0 345143 5933 0 0 25 0 5 0 599200499 2854035456 644584 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 696786 644584 176 36 0 694691 0
[pid=21004/tid=21006] ppid=21002 vsize=2787144 CPUtime=875.26 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 137371 0 0 0 84470 3056 0 0 25 0 5 0 599200952 2854035456 644584 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=2787144 CPUtime=877.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 135645 0 0 0 87721 55 0 0 25 0 5 0 599200952 2854035456 644584 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=2787144 CPUtime=875.53 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 146467 0 0 0 84803 2750 0 0 25 0 5 0 599200952 2854035456 644584 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=2787144 CPUtime=877.65 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 139017 0 0 0 87707 58 0 0 25 0 5 0 599200952 2854035456 644584 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3510.76
Current children cumulated vsize (KiB) 2787144

[startup+942.301 s]
/proc/loadavg: 8.18 8.13 8.00 9/186 21066
/proc/meminfo: memFree=26242928/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=2821752 CPUtime=3750.44 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 679131 0 0 0 368914 6130 0 0 25 0 5 0 599200499 2889474048 676784 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 705438 676784 176 36 0 703343 0
[pid=21004/tid=21006] ppid=21002 vsize=2821752 CPUtime=934.96 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 144544 0 0 0 90340 3156 0 0 25 0 5 0 599200952 2889474048 676784 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=2821752 CPUtime=937.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 141711 0 0 0 93719 57 0 0 25 0 5 0 599200952 2889474048 676784 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=2821752 CPUtime=935.51 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 157860 0 0 0 90709 2842 0 0 25 0 5 0 599200952 2889474048 676784 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=2821752 CPUtime=937.65 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 146585 0 0 0 93704 61 0 0 25 0 5 0 599200952 2889474048 676784 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3750.44
Current children cumulated vsize (KiB) 2821752

[startup+1002.3 s]
/proc/loadavg: 8.12 8.12 8.00 9/186 21068
/proc/meminfo: memFree=25862784/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=3039944 CPUtime=3990.12 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 708151 0 0 0 392735 6277 0 0 25 0 5 0 599200499 3112902656 705804 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 759986 705804 176 36 0 757891 0
[pid=21004/tid=21006] ppid=21002 vsize=3039944 CPUtime=994.96 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 156264 0 0 0 96267 3229 0 0 25 0 5 0 599200952 3112902656 705804 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=3039944 CPUtime=997.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 145943 0 0 0 99717 59 0 0 25 0 5 0 599200952 3112902656 705804 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=3039944 CPUtime=995.19 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 163946 0 0 0 96606 2913 0 0 25 0 5 0 599200952 3112902656 705804 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=3039944 CPUtime=997.65 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 153567 0 0 0 99702 63 0 0 25 0 5 0 599200952 3112902656 705804 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3990.12
Current children cumulated vsize (KiB) 3039944

[startup+1062.3 s]
/proc/loadavg: 8.11 8.11 8.00 9/186 21069
/proc/meminfo: memFree=25441644/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=3064784 CPUtime=4229.8 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 733494 0 0 0 416574 6406 0 0 25 0 5 0 599200499 3138338816 731147 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 766196 731147 176 36 0 764101 0
[pid=21004/tid=21006] ppid=21002 vsize=3064784 CPUtime=1054.96 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 165685 0 0 0 102204 3292 0 0 25 0 5 0 599200952 3138338816 731147 33554432000 4194304 4338100 140735894165312 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=3064784 CPUtime=1057.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 150337 0 0 0 105715 61 0 0 25 0 5 0 599200952 3138338816 731147 33554432000 4194304 4338100 140735894165312 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=3064784 CPUtime=1054.86 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 168656 0 0 0 102512 2974 0 0 25 0 5 0 599200952 3138338816 731147 33554432000 4194304 4338100 140735894165312 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=3064784 CPUtime=1057.65 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 160385 0 0 0 105700 65 0 0 25 0 5 0 599200952 3138338816 731147 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4229.8
Current children cumulated vsize (KiB) 3064784

[startup+1122.3 s]
/proc/loadavg: 8.09 8.10 8.00 9/186 21071
/proc/meminfo: memFree=25012784/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=3238000 CPUtime=4469.47 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 763485 0 0 0 440220 6727 0 0 25 0 5 0 599200499 3315712000 761138 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 809500 761138 176 36 0 807405 0
[pid=21004/tid=21006] ppid=21002 vsize=3238000 CPUtime=1114.96 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 175537 0 0 0 108040 3456 0 0 25 0 5 0 599200952 3315712000 761138 33554432000 4194304 4338100 140735894165312 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=3238000 CPUtime=1117.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 155147 0 0 0 111713 63 0 0 25 0 5 0 599200952 3315712000 761138 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=3238000 CPUtime=1114.54 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 175502 0 0 0 108327 3127 0 0 25 0 5 0 599200952 3315712000 761138 33554432000 4194304 4338100 140735894165312 18446744073709551615 4256374 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=3238000 CPUtime=1117.66 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 168868 0 0 0 111698 68 0 0 25 0 5 0 599200952 3315712000 761138 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4469.47
Current children cumulated vsize (KiB) 3238000

[startup+1182.3 s]
/proc/loadavg: 8.09 8.10 8.00 9/186 21072
/proc/meminfo: memFree=24565188/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=3337960 CPUtime=4709.15 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) S 21002 21004 20765 0 -1 4202496 789946 0 0 0 463797 7118 0 0 25 0 5 0 599200499 3418071040 787599 33554432000 4194304 4338100 140735894165312 18446744073709551615 260032198629 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 834490 787599 176 36 0 832395 0
[pid=21004/tid=21006] ppid=21002 vsize=3337960 CPUtime=1174.88 cores=1,3,5,7
/proc/21004/task/21006/stat : 21006 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 183258 0 0 0 113831 3657 0 0 25 0 5 0 599200952 3418071040 787599 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=3337960 CPUtime=1177.76 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 160471 0 0 0 117710 66 0 0 25 0 5 0 599200952 3418071040 787599 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=21004/tid=21008] ppid=21002 vsize=3337960 CPUtime=1174.29 cores=1,3,5,7
/proc/21004/task/21008/stat : 21008 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 182510 0 0 0 114117 3312 0 0 25 0 5 0 599200952 3418071040 787599 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=21004/tid=21009] ppid=21002 vsize=3337960 CPUtime=1177.66 cores=1,3,5,7
/proc/21004/task/21009/stat : 21009 (ttsth-5-4) R 21002 21004 20765 0 -1 4202560 175276 0 0 0 117696 70 0 0 25 0 5 0 599200952 3418071040 787599 33554432000 4194304 4338100 140735894165312 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4709.15
Current children cumulated vsize (KiB) 3337960



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.1 s]
/proc/loadavg: 8.06 8.09 8.00 6/183 21073
/proc/meminfo: memFree=25499288/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=0 CPUtime=4800.08 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) Z 21002 21004 20765 0 -1 4203548 803877 0 0 0 472723 7285 0 0 18 0 2 0 599200499 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 0 0 0 0 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=0 CPUtime=1200.57 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4203588 162453 0 0 0 119985 72 0 0 25 0 2 0 599200952 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.1 s]
/proc/loadavg: 8.06 8.09 8.00 6/183 21073
/proc/meminfo: memFree=25499288/32951124 swapFree=67111464/67111528
[pid=21004] ppid=21002 vsize=0 CPUtime=4800.08 cores=1,3,5,7
/proc/21004/stat : 21004 (ttsth-5-4) Z 21002 21004 20765 0 -1 4203548 803877 0 0 0 472723 7285 0 0 18 0 2 0 599200499 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/21004/statm: 0 0 0 0 0 0 0
[pid=21004/tid=21007] ppid=21002 vsize=0 CPUtime=1200.57 cores=1,3,5,7
/proc/21004/task/21007/stat : 21007 (ttsth-5-4) R 21002 21004 20765 0 -1 4203588 162453 0 0 0 119985 72 0 0 25 0 2 0 599200952 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.08
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1205.18
CPU time (s): 4800.16
CPU user time (s): 4727.23
CPU system time (s): 72.9309
CPU usage (%): 398.294
Max. virtual memory (cumulated for all children) (KiB): 3345496

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4727.23
system time used= 72.9309
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 803877
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 14564
involuntary context switches= 34330

runsolver used 1.88071 second user time and 4.70129 second system time

The end

Launcher Data 

Begin job on node110 at 2011-04-29 17:08:19
IDJOB=3250296
IDBENCH=83249
IDSOLVER=1654
FILE ID=node110/3250296-1304089698
RUNJOBID= node110-1304088491-20784
PBS_JOBID= 13170568
Free space on /tmp= 72812 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-5-5_34308.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250296-1304089698/watcher-3250296-1304089698 -o /tmp/evaluation-result-3250296-1304089698/solver-3250296-1304089698 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250296-1304089698.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1ef198d3f2d717e33f4f2ba124eb7eb2
RANDOM SEED=233618333

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28434532 kB
Buffers:        896024 kB
Cached:         719440 kB
SwapCached:         64 kB
Active:        2570948 kB
Inactive:      1535592 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28434532 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:           22804 kB
Writeback:           0 kB
AnonPages:     2490176 kB
Mapped:          15268 kB
Slab:           342584 kB
PageTables:       9512 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2681468 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72816 MiB
End job on node110 at 2011-04-29 17:28:26