Trace number 3250164

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.44 1202.4

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-3-6_2072.cnf
MD5SUMd2ef996f95bbf68a3c980c797dfdc364
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2071
Number of clauses159039
Sum of the clauses size479052
Maximum clause length6
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3158394
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 5645

Solver Data 

0.09/0.14	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250164-1304089422.cnf 
0.09/0.14	c Using SATTIMEOUT of 4800
0.09/0.14	c Using SATRAM of 15500
0.09/0.14	c Size of subset node = 32
0.09/0.14	c Variables = 2071
0.09/0.14	c Clauses = 159039
0.09/0.14	c Literals = 479052
0.09/0.14	c Reduced size:
0.09/0.14	c Variables = 2071
0.09/0.14	c Clauses = 90486
0.09/0.14	c Literals = 273393
0.09/0.14	c a2
0.19/0.25	c a1
0.29/0.37	c Independent sub-problems = 2
0.29/0.37	c Sub-problem with 819 variables and 42300 clauses
0.29/0.37	c using permutation method 4
0.29/0.37	c count = 1
0.29/0.37	c Ordering finished 0.360 seconds
0.29/0.37	c mapped 0.360 seconds
0.29/0.37	c permuted 0.360 seconds
0.29/0.37	c about to sort
0.29/0.37	c sorted
0.39/0.48	c clauses now 42300
0.39/0.48	c init finished 0.380 seconds
0.39/0.48	c tree built 0.470 seconds
0.39/0.48	c 
0.39/0.48	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.48	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.48	c ................................ | ..........                    |                                  0.5         0.0             
0.39/0.48	c ................................ | ..........                  1 |                                  0.5         2.1             
0.39/0.48	c ................................ | ..........                  2 |                                  0.5         4.3             
0.39/0.48	c ................................ | ..........                  3 |                                  0.5         6.4             
0.39/0.48	c ................................ | ..........                  4 |                                  0.5         8.5             
0.39/0.48	c ................................ | ..........                  5 |                                  0.5        10.6             
0.39/0.48	c ................................ | ..........               1000 | 2            355                 0.5      2040.8         2399
96.01/24.40	c ................................ | ..........            1603000 | 49336        773148             96.0     16696.2     27704841
191.90/48.43	c ................................ | ..........            2548000 | 85911        1254647           192.0     13268.8     47834010
287.75/72.48	c ................................ | ..........            3242000 | 119996       1574527           288.1     11253.0     66018865
384.01/96.54	c ................................ | ..........            3923000 | 147419       1872629           384.2     10210.8     78353401
479.89/120.54	c ................................ | *.........            4491000 | 170079       2135454           480.1      9355.1     91221421
575.75/144.58	c ................................ | *.........            4992000 | 191995       2392928           576.1      8665.5    102184219
672.02/168.65	c ................................ | *.........            5395000 | 215309       2625007           672.2      8025.3    114826133
767.88/192.67	c ................................ | *.........            5753000 | 234454       2837613           768.2      7489.1    125707339
864.18/216.71	c ................................ | *.........            6201000 | 252168       3059163           864.2      7175.4    135282220
960.03/240.73	c ................................ | **........            6563000 | 268982       3242386           960.2      6835.3    143560768
1055.89/264.78	c ................................ | **........            7062000 | 288980       3437415          1056.2      6686.0    153389332
1151.76/288.76	c ................................ | **........            7444000 | 307049       3599753          1152.0      6461.7    159976241
1248.03/312.87	c ................................ | **........            7774000 | 324449       3749480          1248.3      6227.6    167764172
1344.30/336.92	c ................................ | **........            8091000 | 340132       3920224          1344.4      6018.3    175456114
1440.16/360.91	c ................................ | ***.......            8397000 | 357860       4073555          1440.2      5830.3    184880041
1536.05/384.92	c ................................ | ***.......            8663000 | 370964       4224167          1536.1      5639.5    191519377
1632.31/409.06	c ................................ | ***.......            8911000 | 383859       4381817          1632.5      5458.3    199103932
1728.18/433.01	c ................................ | ***.......            9137000 | 397976       4510421          1728.2      5286.9    206756366
1824.05/457.01	c ................................ | ***.......            9415000 | 414211       4650472          1824.1      5161.5    211558883
1919.92/481.04	c ................................ | ****......            9653000 | 428789       4789432          1920.1      5027.4    218785674
2016.19/505.11	c ................................ | ****......            9985000 | 447087       4933969          2016.3      4952.2    226073472
2112.09/529.12	c ................................ | ****......           10268000 | 462624       5068630          2112.2      4861.3    231574659
2207.94/553.15	c ................................ | ****......           10562000 | 478631       5191589          2208.2      4783.2    238461769
2304.21/577.22	c ................................ | ****......           10823000 | 494175       5308314          2304.3      4696.9    243505411
2400.08/601.20	c ................................ | *****.....           11111000 | 508443       5447591          2400.1      4629.4    245152445
2495.96/625.25	c ................................ | *****.....           11405000 | 521067       5576453          2496.2      4569.0    251464735
2591.82/649.27	c ................................ | *****.....           11717000 | 534451       5706568          2592.1      4520.3    257837410
2687.72/673.28	c ................................ | *****.....           12162000 | 552292       5860858          2688.0      4524.5    265446535
2783.93/697.36	c ................................ | *****.....           12644000 | 574292       6020740          2784.2      4541.4    273376657
2879.79/721.40	c ................................ | ******....           13076000 | 594936       6162743          2880.2      4540.0    277530321
2975.66/745.38	c ................................ | ******....           13404000 | 613669       6269433          2976.0      4504.0    279195326
3071.92/769.44	c ................................ | ******....           13779000 | 635900       6394546          3072.1      4485.2    279386671
3167.80/793.48	c ................................ | ******....           14080000 | 649867       6506290          3168.2      4444.2    281773680
3264.06/817.57	c ................................ | ******....           14332000 | 664995       6614397          3264.3      4390.5    286621584
3359.96/841.55	c ................................ | *******...           14595000 | 679504       6723194          3360.1      4343.6    292735469
3456.22/865.61	c ................................ | *******...           14858000 | 693710       6831108          3456.3      4298.9    298671923
3552.08/889.62	c ................................ | *******...           15140000 | 708517       6943191          3552.2      4262.2    303372634
3647.95/913.68	c ................................ | *******...           15374000 | 722144       7036748          3648.3      4214.0    306766958
3743.82/937.69	c ................................ | *******...           15607000 | 737620       7137386          3744.2      4168.3    311887975
3840.09/961.72	c ................................ | ********..           15852000 | 753397       7233067          3840.2      4127.9    312159904
3935.96/985.72	c ................................ | ********..           16114000 | 767936       7326400          3936.1      4093.9    312807532
4031.85/1009.75	c ................................ | ********..           16491000 | 786559       7429597          4032.0      4090.0    313994796
4128.10/1033.83	c ................................ | ********..           16910000 | 809226       7549420          4128.2      4096.2    315597913
4223.98/1057.82	c ................................ | ********..           17271000 | 828287       7664301          4224.1      4088.7    318575007
4319.84/1081.86	c ................................ | *********.           17601000 | 845486       7777878          4320.1      4074.2    324499014
4415.71/1105.87	c ................................ | *********.           17897000 | 860873       7877587          4416.0      4052.8    330326001
4511.98/1129.96	c ................................ | *********.           18262000 | 879723       7985214          4512.2      4047.2    336214994
4607.88/1153.96	c ................................ | *********.           18644000 | 896256       8099743          4608.1      4045.9    342943457
4703.73/1177.99	c ................................ | *********.           18945000 | 911270       8193934          4704.1      4027.3    348509984

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250164-1304089422/watcher-3250164-1304089422 -o /tmp/evaluation-result-3250164-1304089422/solver-3250164-1304089422 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250164-1304089422.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.71 1.85 2.32 5/181 2604
/proc/meminfo: memFree=24794484/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=9112 CPUtime=0 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) R 2602 2604 1375 0 -1 4202496 361 0 0 0 0 0 0 0 21 0 1 0 599172666 9330688 303 33554432000 4194304 4338100 140737093730224 18446744073709551615 252458065664 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2604/statm: 2278 304 146 36 0 183 0

[startup+0.0433639 s]
/proc/loadavg: 1.71 1.85 2.32 5/181 2604
/proc/meminfo: memFree=24794484/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=15388 CPUtime=0.04 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) R 2602 2604 1375 0 -1 4202496 1921 0 0 0 4 0 0 0 21 0 1 0 599172666 15757312 1863 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457726661 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2604/statm: 3847 1865 147 36 0 1752 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 15388

[startup+0.100354 s]
/proc/loadavg: 1.71 1.85 2.32 5/181 2604
/proc/meminfo: memFree=24794484/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=24172 CPUtime=0.09 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) R 2602 2604 1375 0 -1 4202496 4112 0 0 0 9 0 0 0 21 0 1 0 599172666 24752128 4054 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457726661 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2604/statm: 6043 4055 147 36 0 3948 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 24172

[startup+0.300322 s]
/proc/loadavg: 1.71 1.85 2.32 5/181 2604
/proc/meminfo: memFree=24794484/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=40552 CPUtime=0.29 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) R 2602 2604 1375 0 -1 4202496 8512 0 0 0 28 1 0 0 22 0 1 0 599172666 41525248 8059 33554432000 4194304 4338100 140737093730224 18446744073709551615 4245241 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2604/statm: 10138 8059 155 36 0 8043 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 40552

[startup+0.700209 s]
/proc/loadavg: 1.71 1.85 2.32 5/181 2604
/proc/meminfo: memFree=24794484/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=177068 CPUtime=1.34 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 15860 0 0 0 131 3 0 0 23 0 5 0 599172666 181317632 15323 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 44267 15323 176 36 0 42172 0
Current children cumulated CPU time (s) 1.34
Current children cumulated vsize (KiB) 177068

[startup+1.50106 s]
/proc/loadavg: 1.71 1.85 2.32 9/186 2609
/proc/meminfo: memFree=24645156/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=284720 CPUtime=4.53 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 25717 0 0 0 449 4 0 0 23 0 5 0 599172666 291553280 25180 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 71180 25180 176 36 0 69085 0
[pid=2604/tid=2606] ppid=2602 vsize=284720 CPUtime=1.01 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 3596 0 0 0 100 1 0 0 19 0 5 0 599172714 291553280 25180 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457731608 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=284720 CPUtime=1.01 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 3095 0 0 0 101 0 0 0 19 0 5 0 599172714 291553280 25180 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457460788 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=284720 CPUtime=1 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 3065 0 0 0 100 0 0 0 19 0 5 0 599172714 291553280 25180 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457460816 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=284720 CPUtime=1 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 3272 0 0 0 100 0 0 0 25 0 5 0 599172714 291553280 25180 33554432000 4194304 4338100 140737093730224 18446744073709551615 4256374 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4.53
Current children cumulated vsize (KiB) 284720

[startup+3.10076 s]
/proc/loadavg: 1.71 1.85 2.32 9/186 2609
/proc/meminfo: memFree=24530140/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=364240 CPUtime=10.93 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 45236 0 0 0 1085 8 0 0 23 0 5 0 599172666 372981760 44699 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 91060 44699 176 36 0 88965 0
[pid=2604/tid=2606] ppid=2602 vsize=364240 CPUtime=2.61 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 9281 0 0 0 259 2 0 0 23 0 5 0 599172714 372981760 44699 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=364240 CPUtime=2.61 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 7604 0 0 0 261 0 0 0 22 0 5 0 599172714 372981760 44700 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=364240 CPUtime=2.59 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 8411 0 0 0 258 1 0 0 23 0 5 0 599172714 372981760 44700 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457461112 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=364240 CPUtime=2.6 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 7252 0 0 0 259 1 0 0 25 0 5 0 599172714 372981760 44700 33554432000 4194304 4338100 140737093730224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 10.93
Current children cumulated vsize (KiB) 364240

[startup+6.30117 s]
/proc/loadavg: 2.22 1.95 2.35 9/186 2609
/proc/meminfo: memFree=24231424/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=506848 CPUtime=23.71 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 80601 0 0 0 2357 14 0 0 23 0 5 0 599172666 519012352 80064 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 126712 80065 176 36 0 124617 0
[pid=2604/tid=2606] ppid=2602 vsize=506848 CPUtime=5.8 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 18735 0 0 0 577 3 0 0 25 0 5 0 599172714 519012352 80065 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457461498 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=506848 CPUtime=5.81 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 17081 0 0 0 580 1 0 0 25 0 5 0 599172714 519012352 80065 33554432000 4194304 4338100 140737093730224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=506848 CPUtime=5.79 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 17129 0 0 0 576 3 0 0 25 0 5 0 599172714 519012352 80065 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=506848 CPUtime=5.8 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 14968 0 0 0 577 3 0 0 25 0 5 0 599172714 519012352 80065 33554432000 4194304 4338100 140737093730224 18446744073709551615 4284188 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.71
Current children cumulated vsize (KiB) 506848

[startup+12.701 s]
/proc/loadavg: 2.68 2.05 2.38 9/186 2609
/proc/meminfo: memFree=23784236/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=747612 CPUtime=49.27 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 141858 0 0 0 4902 25 0 0 23 0 5 0 599172666 765554688 141321 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 186903 141321 176 36 0 184808 0
[pid=2604/tid=2606] ppid=2602 vsize=747612 CPUtime=12.2 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 32361 0 0 0 1214 6 0 0 25 0 5 0 599172714 765554688 141321 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=747612 CPUtime=12.21 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 32220 0 0 0 1218 3 0 0 25 0 5 0 599172714 765554688 141321 33554432000 4194304 4338100 140737093730224 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=747612 CPUtime=12.19 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 33330 0 0 0 1213 6 0 0 25 0 5 0 599172714 765554688 141321 33554432000 4194304 4338100 140737093730224 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=747612 CPUtime=12.16 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 31258 0 0 0 1210 6 0 0 25 0 5 0 599172714 765554688 141321 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 49.27
Current children cumulated vsize (KiB) 747612

[startup+25.5006 s]
/proc/loadavg: 3.86 2.34 2.47 9/186 2610
/proc/meminfo: memFree=23053500/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=1158856 CPUtime=100.41 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 244696 0 0 0 9995 46 0 0 23 0 5 0 599172666 1186668544 244159 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 289714 244159 176 36 0 287619 0
[pid=2604/tid=2606] ppid=2602 vsize=1158856 CPUtime=25.01 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 60221 0 0 0 2488 13 0 0 25 0 5 0 599172714 1186668544 244160 33554432000 4194304 4338100 140737093730224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=1158856 CPUtime=25.01 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 56767 0 0 0 2493 8 0 0 25 0 5 0 599172714 1186668544 244160 33554432000 4194304 4338100 140737093730224 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=1158856 CPUtime=24.99 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 61038 0 0 0 2487 12 0 0 25 0 5 0 599172714 1186668544 244160 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=1158856 CPUtime=24.88 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 53982 0 0 0 2479 9 0 0 25 0 5 0 599172714 1186668544 244160 33554432000 4194304 4338100 140737093730224 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.41
Current children cumulated vsize (KiB) 1158856

[startup+51.1008 s]
/proc/loadavg: 5.34 2.81 2.62 9/186 2610
/proc/meminfo: memFree=21959140/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=1827740 CPUtime=202.66 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 414844 0 0 0 20189 77 0 0 23 0 5 0 599172666 1871605760 414307 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 456935 414307 176 36 0 454840 0
[pid=2604/tid=2606] ppid=2602 vsize=1827740 CPUtime=50.61 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 105724 0 0 0 5037 24 0 0 25 0 5 0 599172714 1871605760 414307 33554432000 4194304 4338100 140737093730224 18446744073709551615 4256067 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=1827740 CPUtime=50.61 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 96415 0 0 0 5047 14 0 0 25 0 5 0 599172714 1871605760 414307 33554432000 4194304 4338100 140737093730224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=1827740 CPUtime=50.59 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 102885 0 0 0 5040 19 0 0 25 0 5 0 599172714 1871605760 414307 33554432000 4194304 4338100 140737093730224 18446744073709551615 4230424 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=1827740 CPUtime=50.34 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 97131 0 0 0 5018 16 0 0 25 0 5 0 599172714 1871605760 414307 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 202.66
Current children cumulated vsize (KiB) 1827740

[startup+102.306 s]
/proc/loadavg: 6.92 3.63 2.90 9/186 2612
/proc/meminfo: memFree=20352580/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=2851440 CPUtime=407.2 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 670267 0 0 0 40589 131 0 0 23 0 5 0 599172666 2919874560 669730 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 712860 669730 176 36 0 710765 0
[pid=2604/tid=2606] ppid=2602 vsize=2851440 CPUtime=101.82 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 169615 0 0 0 10143 39 0 0 25 0 5 0 599172714 2919874560 669730 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=2851440 CPUtime=101.56 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 155629 0 0 0 10130 26 0 0 25 0 5 0 599172714 2919874560 669730 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457762491 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=2851440 CPUtime=101.79 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 171871 0 0 0 10146 33 0 0 25 0 5 0 599172714 2919874560 669730 33554432000 4194304 4338100 140737093730224 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=2851440 CPUtime=101.52 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 160463 0 0 0 10123 29 0 0 25 0 5 0 599172714 2919874560 669730 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457460961 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.2
Current children cumulated vsize (KiB) 2851440

[startup+162.301 s]
/proc/loadavg: 7.67 4.44 3.22 9/186 2613
/proc/meminfo: memFree=18897932/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=3790288 CPUtime=646.85 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 907084 0 0 0 64503 182 0 0 23 0 5 0 599172666 3881254912 906547 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 947572 906547 176 36 0 945477 0
[pid=2604/tid=2606] ppid=2602 vsize=3790288 CPUtime=161.81 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 224789 0 0 0 16129 52 0 0 25 0 5 0 599172714 3881254912 906547 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=3790288 CPUtime=161.23 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 197850 0 0 0 16087 36 0 0 25 0 5 0 599172714 3881254912 906547 33554432000 4194304 4338100 140737093730224 18446744073709551615 4263552 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=3790288 CPUtime=161.78 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 230439 0 0 0 16132 46 0 0 25 0 5 0 599172714 3881254912 906547 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=3790288 CPUtime=161.52 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 241317 0 0 0 16107 45 0 0 25 0 5 0 599172714 3881254912 906547 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 646.85
Current children cumulated vsize (KiB) 3790288


################
# More data... #
################

/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 590600 0 0 0 76026 127 0 0 25 0 5 0 599172714 9416052736 2263217 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3043.59
Current children cumulated vsize (KiB) 9195364

[startup+822.3 s]
/proc/loadavg: 8.07 7.69 5.68 9/186 2631
/proc/meminfo: memFree=10043028/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=9473004 CPUtime=3283.24 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2334735 0 0 0 327790 534 0 0 23 0 5 0 599172666 9700356096 2334198 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2368251 2334198 176 36 0 2366156 0
[pid=2604/tid=2606] ppid=2602 vsize=9473004 CPUtime=821.78 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 672064 0 0 0 82009 169 0 0 25 0 5 0 599172714 9700356096 2334198 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=9473004 CPUtime=817.63 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 474426 0 0 0 81659 104 0 0 25 0 5 0 599172714 9700356096 2334199 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=9473004 CPUtime=821.8 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 565528 0 0 0 82055 125 0 0 25 0 5 0 599172714 9700356096 2334199 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=9473004 CPUtime=821.51 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 610028 0 0 0 82019 132 0 0 25 0 5 0 599172714 9700356096 2334199 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3283.24
Current children cumulated vsize (KiB) 9473004

[startup+882.3 s]
/proc/loadavg: 8.06 7.76 5.84 9/186 2632
/proc/meminfo: memFree=9365968/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=9951264 CPUtime=3522.91 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2448286 0 0 0 351728 563 0 0 23 0 5 0 599172666 10190094336 2447749 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2487816 2447749 176 36 0 2485721 0
[pid=2604/tid=2606] ppid=2602 vsize=9951264 CPUtime=881.78 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 717554 0 0 0 87998 180 0 0 25 0 5 0 599172714 10190094336 2447749 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=9951264 CPUtime=877.31 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 477131 0 0 0 87625 106 0 0 25 0 5 0 599172714 10190094336 2447749 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=9951264 CPUtime=881.8 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 611054 0 0 0 88042 138 0 0 25 0 5 0 599172714 10190094336 2447749 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=9951264 CPUtime=881.52 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 629858 0 0 0 88016 136 0 0 25 0 5 0 599172714 10190094336 2447749 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3522.91
Current children cumulated vsize (KiB) 9951264

[startup+942.301 s]
/proc/loadavg: 8.06 7.81 5.97 9/186 2634
/proc/meminfo: memFree=8789968/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=10272168 CPUtime=3762.6 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2530777 0 0 0 375675 585 0 0 23 0 5 0 599172666 10518700032 2530240 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2568042 2530240 176 36 0 2565947 0
[pid=2604/tid=2606] ppid=2602 vsize=10272168 CPUtime=941.78 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 761125 0 0 0 93987 191 0 0 25 0 5 0 599172714 10518700032 2530240 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=10272168 CPUtime=936.98 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 477360 0 0 0 93592 106 0 0 25 0 5 0 599172714 10518700032 2530240 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=10272168 CPUtime=941.8 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 635011 0 0 0 94036 144 0 0 25 0 5 0 599172714 10518700032 2530240 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=10272168 CPUtime=941.52 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 644592 0 0 0 94012 140 0 0 25 0 5 0 599172714 10518700032 2530240 33554432000 4194304 4338100 140737093730224 18446744073709551615 4287088 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3762.6
Current children cumulated vsize (KiB) 10272168

[startup+1002.3 s]
/proc/loadavg: 8.05 7.86 6.10 9/186 2636
/proc/meminfo: memFree=8473256/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=10312212 CPUtime=4002.27 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2542290 0 0 0 399637 590 0 0 23 0 5 0 599172666 10559705088 2541753 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2578053 2541753 176 36 0 2575958 0
[pid=2604/tid=2606] ppid=2602 vsize=10312212 CPUtime=1001.78 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 761529 0 0 0 99986 192 0 0 25 0 5 0 599172714 10559705088 2541753 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=10312212 CPUtime=996.65 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 484101 0 0 0 99558 107 0 0 25 0 5 0 599172714 10559705088 2541753 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=10312212 CPUtime=1001.81 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 635481 0 0 0 100035 146 0 0 25 0 5 0 599172714 10559705088 2541753 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=10312212 CPUtime=1001.52 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 648490 0 0 0 100011 141 0 0 25 0 5 0 599172714 10559705088 2541753 33554432000 4194304 4338100 140737093730224 18446744073709551615 4256288 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4002.27
Current children cumulated vsize (KiB) 10312212

[startup+1062.3 s]
/proc/loadavg: 8.12 7.91 6.23 9/186 2637
/proc/meminfo: memFree=8008616/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=10505660 CPUtime=4241.95 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2591792 0 0 0 423591 604 0 0 23 0 5 0 599172666 10757795840 2591255 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2626415 2591255 176 36 0 2624320 0
[pid=2604/tid=2606] ppid=2602 vsize=10505660 CPUtime=1061.78 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 793676 0 0 0 105977 201 0 0 25 0 5 0 599172714 10757795840 2591255 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=10505660 CPUtime=1056.54 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 484353 0 0 0 105546 108 0 0 25 0 5 0 599172714 10757795840 2591255 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=10505660 CPUtime=1061.59 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 635899 0 0 0 106013 146 0 0 25 0 5 0 599172714 10757795840 2591256 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=10505660 CPUtime=1061.52 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 665176 0 0 0 106007 145 0 0 25 0 5 0 599172714 10757795840 2591256 33554432000 4194304 4338100 140737093730224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4241.95
Current children cumulated vsize (KiB) 10505660

[startup+1122.3 s]
/proc/loadavg: 8.10 7.94 6.34 9/186 2639
/proc/meminfo: memFree=7284068/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=10980496 CPUtime=4481.63 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2711438 0 0 0 447529 634 0 0 23 0 5 0 599172666 11244027904 2710901 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2745124 2710901 176 36 0 2743029 0
[pid=2604/tid=2606] ppid=2602 vsize=10980496 CPUtime=1121.78 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 842434 0 0 0 111964 214 0 0 25 0 5 0 599172714 11244027904 2710901 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=10980496 CPUtime=1116.55 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 484615 0 0 0 111546 109 0 0 25 0 5 0 599172714 11244027904 2710901 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457461170 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=10980496 CPUtime=1121.26 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 668296 0 0 0 111972 154 0 0 25 0 5 0 599172714 11244027904 2710901 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=10980496 CPUtime=1121.53 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 703404 0 0 0 112000 153 0 0 25 0 5 0 599172714 11244027904 2710901 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4481.63
Current children cumulated vsize (KiB) 10980496

[startup+1182.3 s]
/proc/loadavg: 8.16 7.98 6.45 9/186 2640
/proc/meminfo: memFree=6583204/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=11448944 CPUtime=4721.3 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2829257 0 0 0 471465 665 0 0 23 0 5 0 599172666 11723718656 2828720 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2862236 2828721 176 36 0 2860141 0
[pid=2604/tid=2606] ppid=2602 vsize=11448944 CPUtime=1181.79 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 881595 0 0 0 117953 226 0 0 25 0 5 0 599172714 11723718656 2828721 33554432000 4194304 4338100 140737093730224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=11448944 CPUtime=1176.54 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 509090 0 0 0 117541 113 0 0 25 0 5 0 599172714 11723718656 2828721 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=11448944 CPUtime=1180.93 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 695246 0 0 0 117931 162 0 0 25 0 5 0 599172714 11723718656 2828721 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469880067 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=11448944 CPUtime=1181.53 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 730638 0 0 0 117993 160 0 0 25 0 5 0 599172714 11723718656 2828721 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4721.3
Current children cumulated vsize (KiB) 11448944



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 8.11 7.98 6.48 5/180 2642
/proc/meminfo: memFree=13322892/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=11584240 CPUtime=4800.01 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) S 2602 2604 1375 0 -1 4202496 2864356 0 0 0 479325 676 0 0 23 0 5 0 599172666 11862261760 2863819 33554432000 4194304 4338100 140737093730224 18446744073709551615 252469868517 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 2896060 2863819 176 36 0 2893965 0
[pid=2604/tid=2606] ppid=2602 vsize=11584240 CPUtime=1201.5 cores=1,3,5,7
/proc/2604/task/2606/stat : 2606 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 890371 0 0 0 119921 229 0 0 25 0 5 0 599172714 11862261760 2863825 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2604/tid=2607] ppid=2602 vsize=11584240 CPUtime=1196.25 cores=1,3,5,7
/proc/2604/task/2607/stat : 2607 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 517157 0 0 0 119509 116 0 0 25 0 5 0 599172714 11862261760 2863825 33554432000 4194304 4338100 140737093730224 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=11584240 CPUtime=1200.52 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 704216 0 0 0 119887 165 0 0 25 0 5 0 599172714 11862261760 2863825 33554432000 4194304 4338100 140737093730224 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2604/tid=2609] ppid=2602 vsize=11584240 CPUtime=1201.23 cores=1,3,5,7
/proc/2604/task/2609/stat : 2609 (ttsth-5-4) R 2602 2604 1375 0 -1 4202560 739929 0 0 0 119961 162 0 0 25 0 5 0 599172714 11862261760 2863825 33554432000 4194304 4338100 140737093730224 18446744073709551615 252457461170 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.01
Current children cumulated vsize (KiB) 11584240

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.2 s]
/proc/loadavg: 7.54 7.86 6.45 2/178 2643
/proc/meminfo: memFree=18554208/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=0 CPUtime=4800.23 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) Z 2602 2604 1375 0 -1 4203548 2864375 0 0 0 479328 695 0 0 18 0 2 0 599172666 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 0 0 0 0 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=0 CPUtime=1200.73 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4203588 704222 0 0 0 119888 185 0 0 25 0 2 0 599172714 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.23
Current children cumulated vsize (KiB) 0

[startup+1202.4 s]
/proc/loadavg: 7.54 7.86 6.45 2/178 2643
/proc/meminfo: memFree=18554208/32951124 swapFree=67111348/67111528
[pid=2604] ppid=2602 vsize=0 CPUtime=4800.43 cores=1,3,5,7
/proc/2604/stat : 2604 (ttsth-5-4) Z 2602 2604 1375 0 -1 4203548 2864375 0 0 0 479328 715 0 0 18 0 2 0 599172666 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2604/statm: 0 0 0 0 0 0 0
[pid=2604/tid=2608] ppid=2602 vsize=0 CPUtime=1200.92 cores=1,3,5,7
/proc/2604/task/2608/stat : 2608 (ttsth-5-4) R 2602 2604 1375 0 -1 4203588 704222 0 0 0 119888 204 0 0 25 0 2 0 599172714 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.43
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.4
CPU time (s): 4800.44
CPU user time (s): 4793.28
CPU system time (s): 7.15691
CPU usage (%): 399.237
Max. virtual memory (cumulated for all children) (KiB): 11584240

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4793.28
system time used= 7.15691
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2864375
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2288
involuntary context switches= 31031

runsolver used 2.00069 second user time and 4.5723 second system time

The end

Launcher Data 

Begin job on node111 at 2011-04-29 17:03:42
IDJOB=3250164
IDBENCH=83245
IDSOLVER=1654
FILE ID=node111/3250164-1304089422
RUNJOBID= node111-1304089422-2581
PBS_JOBID= 13170507
Free space on /tmp= 70896 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-3-6_2072.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250164-1304089422/watcher-3250164-1304089422 -o /tmp/evaluation-result-3250164-1304089422/solver-3250164-1304089422 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250164-1304089422.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d2ef996f95bbf68a3c980c797dfdc364
RANDOM SEED=830383153

node111.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5428.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24794640 kB
Buffers:       1280836 kB
Cached:        6315732 kB
SwapCached:        124 kB
Active:        3277176 kB
Inactive:      4391260 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24794640 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:            2700 kB
Writeback:           0 kB
AnonPages:       70980 kB
Mapped:          15144 kB
Slab:           425128 kB
PageTables:       4668 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   268500 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 70896 MiB
End job on node111 at 2011-04-29 17:23:46