Trace number 3250692

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.5 1202.65

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/
VanderWaerden3k/VanDerWaerden_2-3-19_349.cnf
MD5SUMfaed4f95ee485d18812f5d39a28df3e5
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables349
Number of clauses33487
Sum of the clauses size151837
Maximum clause length19
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 330276
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 53211

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250692-1304093293.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 349
0.00/0.04	c Clauses = 33487
0.00/0.04	c Literals = 151837
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 349
0.00/0.04	c Clauses = 33487
0.00/0.04	c Literals = 151837
0.00/0.04	c a2
0.19/0.20	c a1
0.49/0.51	c Independent sub-problems = 1
0.49/0.51	c Sub-problem with 349 variables and 33487 clauses
0.49/0.51	c using permutation method 4
0.49/0.51	c count = 1
0.49/0.51	c Ordering finished 0.500 seconds
0.49/0.51	c mapped 0.500 seconds
0.49/0.51	c permuted 0.500 seconds
0.49/0.51	c about to sort
0.49/0.52	c sorted
0.59/0.65	c clauses now 33487
0.59/0.65	c init finished 0.510 seconds
0.59/0.65	c tree built 0.640 seconds
0.59/0.65	c 
0.59/0.65	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.59/0.65	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.59/0.65	c ................................ | ..........                    |                                  0.6         0.0             
0.59/0.65	c ................................ | ..........                  1 |                                  0.6         1.6             
0.59/0.65	c ................................ | ..........                  2 |                                  0.6         3.1             
0.59/0.65	c ................................ | ..........                  3 |                                  0.6         4.7             
0.59/0.65	c ................................ | ..........                  4 |                                  0.6         6.2             
0.59/0.65	c ................................ | ..........                  5 |                                  0.6         7.8             
0.59/0.69	c ................................ | ..........               1000 | 44           339                 0.8      1282.1        35878
96.28/24.60	c ................................ | ..........             358000 | 51825        74353              96.3      3718.3     27998932
191.74/48.59	c ................................ | ..........             610000 | 87174        122626            192.1      3175.4     48140997
288.03/72.62	c ................................ | ..........             822000 | 114256       162387            288.1      2853.0     62127674
383.87/96.67	c ................................ | ..........            1005000 | 137946       200666            384.2      2615.9     77137006
480.54/120.80	c ................................ | *.........            1154000 | 157541       234373            480.6      2401.3     89249851
576.41/144.87	c ................................ | *.........            1296000 | 178150       264279            576.7      2247.2    101344548
672.28/168.81	c ................................ | *.........            1419000 | 195427       293475            672.3      2110.6    112883885
768.14/192.89	c ................................ | *.........            1551000 | 214221       319955            768.5      2018.2    123721889
864.42/216.93	c ................................ | *.........            1697000 | 235271       344997            864.6      1962.8    134884400
960.31/240.94	c ................................ | **........            1809000 | 252141       370640            960.5      1883.5    145145503
1056.56/265.01	c ................................ | **........            1917000 | 266943       391693           1056.6      1814.3    152820828
1152.01/288.93	c ................................ | **........            2021000 | 281590       412712           1152.1      1754.1    160180947
1248.26/313.05	c ................................ | **........            2117000 | 296244       432256           1248.5      1695.7    163658404
1343.72/336.97	c ................................ | **........            2198000 | 308496       450589           1344.0      1635.4    171528503
1439.98/361.09	c ................................ | ***.......            2298000 | 323724       467043           1440.4      1595.4    180038868
1536.66/385.25	c ................................ | ***.......            2373000 | 333918       485295           1536.9      1544.0    185748881
1632.13/409.20	c ................................ | ***.......            2438000 | 344524       501541           1632.5      1493.4    192850515
1728.38/433.28	c ................................ | ***.......            2505000 | 355338       520213           1728.7      1449.0    199053521
1825.46/457.56	c ................................ | ***.......            2568000 | 366168       537946           1825.7      1406.6    206111070
1920.92/481.49	c ................................ | ****......            2621000 | 375299       556390           1921.3      1364.2    212233842
2016.38/505.31	c ................................ | ****......            2677000 | 385004       573549           2016.4      1327.6    218570927
2113.05/529.59	c ................................ | ****......            2736000 | 395674       589086           2113.4      1294.6    225136775
2209.72/553.71	c ................................ | ****......            2787000 | 405083       605473           2209.8      1261.2    230894523
2304.41/577.41	c ................................ | ****......            2858000 | 416902       620468           2304.4      1240.2    237598329
2401.07/601.64	c ................................ | *****.....            2932000 | 428462       637338           2401.2      1221.0    243912001
2496.90/625.60	c ................................ | *****.....            3000000 | 440472       651421           2496.9      1201.5    249338631
2593.17/649.72	c ................................ | *****.....            3066000 | 452131       665294           2593.3      1182.3    254476894
2688.64/673.68	c ................................ | *****.....            3123000 | 461492       679230           2689.0      1161.4    259845714
2784.91/697.74	c ................................ | *****.....            3185000 | 471105       694071           2785.1      1143.6    264987224
2881.57/721.94	c ................................ | ******....            3245000 | 480399       708386           2881.7      1126.1    269352611
2976.66/745.72	c ................................ | ******....            3305000 | 489900       722135           2976.7      1110.3    274685602
3072.51/769.75	c ................................ | ******....            3360000 | 499628       734033           3072.7      1093.5    279304009
3168.77/793.85	c ................................ | ******....            3414000 | 509017       744748           3169.0      1077.3    284343611
3265.44/818.08	c ................................ | ******....            3462000 | 517399       753962           3265.8      1060.1    288249362
3359.70/841.68	c ................................ | *******...            3508000 | 525517       762555           3360.0      1044.0    292471372
3456.36/865.89	c ................................ | *******...            3552000 | 533540       774171           3456.7      1027.6    297124397
3552.64/889.99	c ................................ | *******...            3610000 | 543845       784535           3553.0      1016.1    298475406
3648.49/913.97	c ................................ | *******...            3654000 | 550536       795292           3648.8      1001.4    298475406
3744.28/937.92	c ................................ | *******...            3695000 | 556256       806794           3744.4       986.8    299902088
3840.14/961.97	c ................................ | ********..            3743000 | 562922       818060           3840.4       974.6    299902088
3936.81/986.11	c ................................ | ********..            3790000 | 570070       828937           3936.8       962.7    299902088
4032.27/1010.05	c ................................ | ********..            3841000 | 578994       838440           4032.5       952.5    299902088
4129.74/1034.42	c ................................ | ********..            3894000 | 586810       847955           4129.8       942.9    299902088
4224.43/1058.19	c ................................ | ********..            3948000 | 595455       857913           4224.8       934.5    301283918
4319.88/1082.04	c ................................ | *********.            4001000 | 603372       868254           4320.1       926.1    303944161
4416.55/1106.29	c ................................ | *********.            4053000 | 611442       880117           4416.9       917.6    306511117
4512.00/1130.18	c ................................ | *********.            4101000 | 619346       891541           4512.3       908.8    308349060
4609.47/1154.51	c ................................ | *********.            4148000 | 627069       901872           4609.5       899.9    309248448
4704.53/1178.34	c ................................ | *********.            4197000 | 633940       912736           4704.7       892.1    312304579

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250692-1304093293/watcher-3250692-1304093293 -o /tmp/evaluation-result-3250692-1304093293/solver-3250692-1304093293 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250692-1304093293.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.17 7.90 8.00 1/172 28607
/proc/meminfo: memFree=17460296/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) R 28605 28607 27480 0 -1 4202496 440 0 0 0 0 0 0 0 22 0 1 0 599560120 9768960 383 33554432000 4194304 4338100 140734734827056 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28607/statm: 2385 383 146 36 0 290 0

[startup+0.0932619 s]
/proc/loadavg: 7.17 7.90 8.00 1/172 28607
/proc/meminfo: memFree=17460296/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=32820 CPUtime=0.08 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) R 28605 28607 27480 0 -1 4202496 5105 0 0 0 8 0 0 0 22 0 1 0 599560120 33607680 4928 33554432000 4194304 4338100 140734734827056 18446744073709551615 215988289028 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28607/statm: 8205 4928 421 36 0 6110 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 32820

[startup+0.101254 s]
/proc/loadavg: 7.17 7.90 8.00 1/172 28607
/proc/meminfo: memFree=17460296/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=32820 CPUtime=0.09 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) R 28605 28607 27480 0 -1 4202496 5105 0 0 0 9 0 0 0 22 0 1 0 599560120 33607680 4928 33554432000 4194304 4338100 140734734827056 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28607/statm: 8205 4928 421 36 0 6110 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 32820

[startup+0.300223 s]
/proc/loadavg: 7.17 7.90 8.00 1/172 28607
/proc/meminfo: memFree=17460296/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=17092 CPUtime=0.29 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) R 28605 28607 27480 0 -1 4202496 5689 0 0 0 29 0 0 0 23 0 1 0 599560120 17502208 2292 33554432000 4194304 4338100 140734734827056 18446744073709551615 4296782 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28607/statm: 4273 2292 155 36 0 2178 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 17092

[startup+0.701042 s]
/proc/loadavg: 7.17 7.90 8.00 1/172 28607
/proc/meminfo: memFree=17460296/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=241840 CPUtime=0.82 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 16427 0 0 0 80 2 0 0 25 0 5 0 599560120 247644160 12835 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 60460 12835 176 36 0 58365 0
Current children cumulated CPU time (s) 0.82
Current children cumulated vsize (KiB) 241840

[startup+1.5009 s]
/proc/loadavg: 7.17 7.90 8.00 9/186 28626
/proc/meminfo: memFree=17352924/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=281488 CPUtime=4.01 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 27603 0 0 0 397 4 0 0 25 0 5 0 599560120 288243712 24011 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 70372 24012 176 36 0 68277 0
[pid=28607/tid=28619] ppid=28605 vsize=281488 CPUtime=0.83 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 3093 0 0 0 83 0 0 0 19 0 5 0 599560185 288243712 24012 33554432000 4194304 4338100 140734734827056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=281488 CPUtime=0.83 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 2763 0 0 0 83 0 0 0 19 0 5 0 599560185 288243712 24012 33554432000 4194304 4338100 140734734827056 18446744073709551615 4256993 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=281488 CPUtime=0.83 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 3068 0 0 0 82 1 0 0 19 0 5 0 599560185 288243712 24012 33554432000 4194304 4338100 140734734827056 18446744073709551615 4256490 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=281488 CPUtime=0.83 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 3265 0 0 0 83 0 0 0 18 0 5 0 599560185 288243712 24013 33554432000 4194304 4338100 140734734827056 18446744073709551615 4256989 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4.01
Current children cumulated vsize (KiB) 281488

[startup+3.10062 s]
/proc/loadavg: 7.17 7.90 8.00 9/186 28626
/proc/meminfo: memFree=17232412/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=360832 CPUtime=10.4 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 47366 0 0 0 1032 8 0 0 25 0 5 0 599560120 369491968 43774 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 90208 43774 176 36 0 88113 0
[pid=28607/tid=28619] ppid=28605 vsize=360832 CPUtime=2.42 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 7828 0 0 0 242 0 0 0 22 0 5 0 599560185 369491968 43774 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=360832 CPUtime=2.43 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 7930 0 0 0 242 1 0 0 22 0 5 0 599560185 369491968 43775 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000405198 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=360832 CPUtime=2.42 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 7669 0 0 0 240 2 0 0 22 0 5 0 599560185 369491968 43775 33554432000 4194304 4338100 140734734827056 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=360832 CPUtime=2.43 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 8524 0 0 0 242 1 0 0 22 0 5 0 599560185 369491968 43775 33554432000 4194304 4338100 140734734827056 18446744073709551615 4256993 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 10.4
Current children cumulated vsize (KiB) 360832

[startup+6.30106 s]
/proc/loadavg: 7.24 7.90 8.00 9/186 28626
/proc/meminfo: memFree=16957196/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=499448 CPUtime=23.19 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 82328 0 0 0 2305 14 0 0 25 0 5 0 599560120 511434752 78736 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 124862 78737 176 36 0 122767 0
[pid=28607/tid=28619] ppid=28605 vsize=499448 CPUtime=5.63 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 16586 0 0 0 561 2 0 0 25 0 5 0 599560185 511434752 78737 33554432000 4194304 4338100 140734734827056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=499448 CPUtime=5.64 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 15824 0 0 0 561 3 0 0 25 0 5 0 599560185 511434752 78737 33554432000 4194304 4338100 140734734827056 18446744073709551615 4288141 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=499448 CPUtime=5.6 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 17186 0 0 0 557 3 0 0 25 0 5 0 599560185 511434752 78737 33554432000 4194304 4338100 140734734827056 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=499448 CPUtime=5.64 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 17318 0 0 0 561 3 0 0 25 0 5 0 599560185 511434752 78737 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 23.19
Current children cumulated vsize (KiB) 499448

[startup+12.7009 s]
/proc/loadavg: 7.30 7.90 8.00 9/186 28627
/proc/meminfo: memFree=16516120/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=741232 CPUtime=48.74 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 143059 0 0 0 4850 24 0 0 25 0 5 0 599560120 759021568 139467 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 185308 139468 176 36 0 183213 0
[pid=28607/tid=28619] ppid=28605 vsize=741232 CPUtime=12.03 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 30879 0 0 0 1199 4 0 0 25 0 5 0 599560185 759021568 139469 33554432000 4194304 4338100 140734734827056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=741232 CPUtime=12.03 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 30174 0 0 0 1198 5 0 0 25 0 5 0 599560185 759021568 139469 33554432000 4194304 4338100 140734734827056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=741232 CPUtime=11.97 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 33595 0 0 0 1191 6 0 0 18 0 5 0 599560185 759021568 139471 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=741232 CPUtime=12.03 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 32998 0 0 0 1198 5 0 0 25 0 5 0 599560185 759021568 139472 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 48.74
Current children cumulated vsize (KiB) 741232

[startup+25.5007 s]
/proc/loadavg: 7.52 7.92 8.00 9/186 28627
/proc/meminfo: memFree=15860648/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=1148952 CPUtime=99.87 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 244724 0 0 0 9945 42 0 0 25 0 5 0 599560120 1176526848 241132 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 287238 241132 176 36 0 285143 0
[pid=28607/tid=28619] ppid=28605 vsize=1148952 CPUtime=24.83 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 58475 0 0 0 2474 9 0 0 25 0 5 0 599560185 1176526848 241132 33554432000 4194304 4338100 140734734827056 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=1148952 CPUtime=24.83 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 52855 0 0 0 2474 9 0 0 25 0 5 0 599560185 1176526848 241132 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000406787 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=1148952 CPUtime=24.69 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 60318 0 0 0 2459 10 0 0 25 0 5 0 599560185 1176526848 241132 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=1148952 CPUtime=24.83 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 57661 0 0 0 2473 10 0 0 25 0 5 0 599560185 1176526848 241132 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 99.87
Current children cumulated vsize (KiB) 1148952

[startup+51.1002 s]
/proc/loadavg: 7.68 7.93 8.00 9/186 28628
/proc/meminfo: memFree=14840720/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=1814240 CPUtime=202.13 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 411421 0 0 0 20141 72 0 0 25 0 5 0 599560120 1857781760 407829 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 453560 407829 176 36 0 451465 0
[pid=28607/tid=28619] ppid=28605 vsize=1814240 CPUtime=50.43 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 102896 0 0 0 5027 16 0 0 25 0 5 0 599560185 1857781760 407830 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=1814240 CPUtime=50.43 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 87281 0 0 0 5028 15 0 0 25 0 5 0 599560185 1857781760 407830 33554432000 4194304 4338100 140734734827056 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=1814240 CPUtime=50.16 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 103030 0 0 0 4997 19 0 0 25 0 5 0 599560185 1857781760 407830 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=1814240 CPUtime=50.43 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 102800 0 0 0 5024 19 0 0 25 0 5 0 599560185 1857781760 407830 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.13
Current children cumulated vsize (KiB) 1814240

[startup+102.306 s]
/proc/loadavg: 7.90 7.95 8.00 9/186 28629
/proc/meminfo: memFree=13395356/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=2790412 CPUtime=406.66 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 655020 0 0 0 40551 115 0 0 25 0 5 0 599560120 2857381888 651428 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 697603 651428 176 36 0 695508 0
[pid=28607/tid=28619] ppid=28605 vsize=2790412 CPUtime=101.63 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 171586 0 0 0 10135 28 0 0 25 0 5 0 599560185 2857381888 651428 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254231 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=2790412 CPUtime=101.63 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 143362 0 0 0 10138 25 0 0 25 0 5 0 599560185 2857381888 651428 33554432000 4194304 4338100 140734734827056 18446744073709551615 4256318 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=2790412 CPUtime=101.08 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 162804 0 0 0 10079 29 0 0 25 0 5 0 599560185 2857381888 651428 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=2790412 CPUtime=101.64 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 161853 0 0 0 10134 30 0 0 25 0 5 0 599560185 2857381888 651428 33554432000 4194304 4338100 140734734827056 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 406.66
Current children cumulated vsize (KiB) 2790412

[startup+162.301 s]
/proc/loadavg: 7.99 7.97 8.00 9/186 28631
/proc/meminfo: memFree=12032564/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=3714188 CPUtime=646.31 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 885579 0 0 0 64477 154 0 0 25 0 5 0 599560120 3803328512 881987 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 928547 881987 176 36 0 926452 0
[pid=28607/tid=28619] ppid=28605 vsize=3714188 CPUtime=161.63 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 229070 0 0 0 16125 38 0 0 25 0 5 0 599560185 3803328512 881987 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=3714188 CPUtime=161.61 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 198637 0 0 0 16126 35 0 0 25 0 5 0 599560185 3803328512 881987 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=3714188 CPUtime=160.75 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 223844 0 0 0 16036 39 0 0 25 0 5 0 599560185 3803328512 881987 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=3714188 CPUtime=161.64 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 218613 0 0 0 16125 39 0 0 25 0 5 0 599560185 3803328512 881987 33554432000 4194304 4338100 140734734827056 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 646.31
Current children cumulated vsize (KiB) 3714188


################
# More data... #
################

/proc/meminfo: memFree=4025196/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=9030556 CPUtime=3042.97 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2211615 0 0 0 303849 448 0 0 25 0 5 0 599560120 9247289344 2208023 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2257639 2208023 176 36 0 2255544 0
[pid=28607/tid=28619] ppid=28605 vsize=9030556 CPUtime=761.64 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 560621 0 0 0 76052 112 0 0 25 0 5 0 599560185 9247289344 2208023 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=9030556 CPUtime=761.51 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 549825 0 0 0 76037 114 0 0 25 0 5 0 599560185 9247289344 2208023 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=9030556 CPUtime=759.72 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 549224 0 0 0 75866 106 0 0 25 0 5 0 599560185 9247289344 2208023 33554432000 4194304 4338100 140734734827056 18446744073709551615 4239479 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=9030556 CPUtime=759.4 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 536530 0 0 0 75829 111 0 0 25 0 5 0 599560185 9247289344 2208023 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3042.97
Current children cumulated vsize (KiB) 9030556

[startup+822.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/186 28648
/proc/meminfo: memFree=3496196/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=9378028 CPUtime=3282.61 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2297906 0 0 0 327789 472 0 0 25 0 5 0 599560120 9603100672 2294314 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2344507 2294314 176 36 0 2342412 0
[pid=28607/tid=28619] ppid=28605 vsize=9378028 CPUtime=821.63 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 585517 0 0 0 82043 120 0 0 25 0 5 0 599560185 9603100672 2294314 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=9378028 CPUtime=821.5 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 566819 0 0 0 82032 118 0 0 25 0 5 0 599560185 9603100672 2294314 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=9378028 CPUtime=819.72 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 568796 0 0 0 81860 112 0 0 25 0 5 0 599560185 9603100672 2294314 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=9378028 CPUtime=819.07 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 561359 0 0 0 81790 117 0 0 25 0 5 0 599560185 9603100672 2294314 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3282.61
Current children cumulated vsize (KiB) 9378028

[startup+882.301 s]
/proc/loadavg: 8.06 8.05 8.00 9/186 28650
/proc/meminfo: memFree=2963780/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=9729912 CPUtime=3522.26 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2383730 0 0 0 351732 494 0 0 25 0 5 0 599560120 9963429888 2380138 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2432478 2380138 176 36 0 2430383 0
[pid=28607/tid=28619] ppid=28605 vsize=9729912 CPUtime=881.63 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 609857 0 0 0 88038 125 0 0 25 0 5 0 599560185 9963429888 2380138 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=9729912 CPUtime=881.49 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 585180 0 0 0 88025 124 0 0 25 0 5 0 599560185 9963429888 2380138 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=9729912 CPUtime=879.72 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 587948 0 0 0 87854 118 0 0 25 0 5 0 599560185 9963429888 2380138 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=9729912 CPUtime=878.74 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 585330 0 0 0 87750 124 0 0 25 0 5 0 599560185 9963429888 2380138 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3522.26
Current children cumulated vsize (KiB) 9729912

[startup+942.3 s]
/proc/loadavg: 8.09 8.07 8.01 9/186 28651
/proc/meminfo: memFree=2547968/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=9978900 CPUtime=3761.85 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2431540 0 0 0 375676 509 0 0 25 0 5 0 599560120 10218393600 2427948 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2494725 2427948 176 36 0 2492630 0
[pid=28607/tid=28619] ppid=28605 vsize=9978900 CPUtime=941.61 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 620811 0 0 0 94032 129 0 0 25 0 5 0 599560185 10218393600 2427948 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=9978900 CPUtime=941.45 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 608773 0 0 0 94016 129 0 0 25 0 5 0 599560185 10218393600 2427948 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=9978900 CPUtime=939.69 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 595885 0 0 0 93848 121 0 0 25 0 5 0 599560185 10218393600 2427948 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=9978900 CPUtime=938.4 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 590656 0 0 0 93714 126 0 0 25 0 5 0 599560185 10218393600 2427948 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254413 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3761.85
Current children cumulated vsize (KiB) 9978900

[startup+1002.3 s]
/proc/loadavg: 8.18 8.10 8.02 9/186 28653
/proc/meminfo: memFree=2370028/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=9979296 CPUtime=4001.52 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2431750 0 0 0 399643 509 0 0 25 0 5 0 599560120 10218799104 2428158 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2494824 2428158 176 36 0 2492729 0
[pid=28607/tid=28619] ppid=28605 vsize=9979296 CPUtime=1001.61 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 620865 0 0 0 100032 129 0 0 25 0 5 0 599560185 10218799104 2428158 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=9979296 CPUtime=1001.45 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 608817 0 0 0 100016 129 0 0 25 0 5 0 599560185 10218799104 2428158 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=9979296 CPUtime=999.69 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 595924 0 0 0 99848 121 0 0 25 0 5 0 599560185 10218799104 2428158 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=9979296 CPUtime=998.07 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 590729 0 0 0 99681 126 0 0 25 0 5 0 599560185 10218799104 2428158 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4001.52
Current children cumulated vsize (KiB) 9979296

[startup+1062.3 s]
/proc/loadavg: 8.25 8.13 8.03 9/186 28655
/proc/meminfo: memFree=2199584/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=10033000 CPUtime=4241.19 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2445087 0 0 0 423606 513 0 0 25 0 5 0 599560120 10273792000 2441495 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2508250 2441495 176 36 0 2506155 0
[pid=28607/tid=28619] ppid=28605 vsize=10033000 CPUtime=1061.61 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 627633 0 0 0 106030 131 0 0 25 0 5 0 599560185 10273792000 2441495 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=10033000 CPUtime=1061.44 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 608856 0 0 0 106015 129 0 0 25 0 5 0 599560185 10273792000 2441495 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=10033000 CPUtime=1059.69 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 595958 0 0 0 105848 121 0 0 25 0 5 0 599560185 10273792000 2441495 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=10033000 CPUtime=1057.74 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 597225 0 0 0 105647 127 0 0 25 0 5 0 599560185 10273792000 2441495 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4241.19
Current children cumulated vsize (KiB) 10033000

[startup+1122.3 s]
/proc/loadavg: 8.09 8.10 8.02 9/186 28656
/proc/meminfo: memFree=1960604/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=10239712 CPUtime=4480.85 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2497044 0 0 0 447561 524 0 0 25 0 5 0 599560120 10485465088 2493452 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2559928 2493452 176 36 0 2557833 0
[pid=28607/tid=28619] ppid=28605 vsize=10239712 CPUtime=1121.62 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 651787 0 0 0 112025 137 0 0 25 0 5 0 599560185 10485465088 2493452 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=10239712 CPUtime=1121.43 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 608918 0 0 0 112014 129 0 0 25 0 5 0 599560185 10485465088 2493452 33554432000 4194304 4338100 140734734827056 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=10239712 CPUtime=1119.69 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 595996 0 0 0 111848 121 0 0 25 0 5 0 599560185 10485465088 2493452 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=10239712 CPUtime=1117.42 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 624928 0 0 0 111609 133 0 0 25 0 5 0 599560185 10485465088 2493452 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4480.85
Current children cumulated vsize (KiB) 10239712

[startup+1182.3 s]
/proc/loadavg: 8.06 8.09 8.02 9/186 28658
/proc/meminfo: memFree=1726112/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=10390252 CPUtime=4720.51 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2534624 0 0 0 471516 535 0 0 25 0 5 0 599560120 10639618048 2531032 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2597563 2531032 176 36 0 2595468 0
[pid=28607/tid=28619] ppid=28605 vsize=10390252 CPUtime=1181.62 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 670385 0 0 0 118020 142 0 0 25 0 5 0 599560185 10639618048 2531032 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=10390252 CPUtime=1181.43 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 617237 0 0 0 118011 132 0 0 25 0 5 0 599560185 10639618048 2531032 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=10390252 CPUtime=1179.69 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 596040 0 0 0 117848 121 0 0 25 0 5 0 599560185 10639618048 2531032 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=10390252 CPUtime=1177.08 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 635547 0 0 0 117573 135 0 0 25 0 5 0 599560185 10639618048 2531032 33554432000 4194304 4338100 140734734827056 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4720.51
Current children cumulated vsize (KiB) 10390252



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.04 8.09 8.02 9/186 28658
/proc/meminfo: memFree=1593204/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=10477968 CPUtime=4800.04 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) S 28605 28607 27480 0 -1 4202496 2556246 0 0 0 479464 540 0 0 25 0 5 0 599560120 10729439232 2552654 33554432000 4194304 4338100 140734734827056 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 2619492 2552654 176 36 0 2617397 0
[pid=28607/tid=28619] ppid=28605 vsize=10477968 CPUtime=1201.53 cores=0,2,4,6
/proc/28607/task/28619/stat : 28619 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 677234 0 0 0 120009 144 0 0 25 0 5 0 599560185 10729439232 2552654 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28607/tid=28620] ppid=28605 vsize=10477968 CPUtime=1201.34 cores=0,2,4,6
/proc/28607/task/28620/stat : 28620 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 620304 0 0 0 120001 133 0 0 25 0 5 0 599560185 10729439232 2552654 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28607/tid=28621] ppid=28605 vsize=10477968 CPUtime=1199.61 cores=0,2,4,6
/proc/28607/task/28621/stat : 28621 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 602395 0 0 0 119838 123 0 0 25 0 5 0 599560185 10729439232 2552654 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=10477968 CPUtime=1196.88 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4202560 640898 0 0 0 119551 137 0 0 25 0 5 0 599560185 10729439232 2552654 33554432000 4194304 4338100 140734734827056 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 10477968

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.6 s]
/proc/loadavg: 8.04 8.09 8.02 9/186 28658
/proc/meminfo: memFree=1593204/32950940 swapFree=67111528/67111528
[pid=28607] ppid=28605 vsize=0 CPUtime=4800.44 cores=0,2,4,6
/proc/28607/stat : 28607 (ttsth-5-4) Z 28605 28607 27480 0 -1 4203548 2556256 0 0 0 479466 578 0 0 18 0 2 0 599560120 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/28607/statm: 0 0 0 0 0 0 0
[pid=28607/tid=28622] ppid=28605 vsize=0 CPUtime=1197.26 cores=0,2,4,6
/proc/28607/task/28622/stat : 28622 (ttsth-5-4) R 28605 28607 27480 0 -1 4203588 640898 0 0 0 119551 175 0 0 25 0 2 0 599560185 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.44
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.65
CPU time (s): 4800.5
CPU user time (s): 4794.66
CPU system time (s): 5.83511
CPU usage (%): 399.161
Max. virtual memory (cumulated for all children) (KiB): 10477968

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4794.66
system time used= 5.83511
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2556256
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 972
involuntary context switches= 20504

runsolver used 1.9817 second user time and 4.5873 second system time

The end

Launcher Data 

Begin job on node117 at 2011-04-29 18:08:13
IDJOB=3250692
IDBENCH=83336
IDSOLVER=1654
FILE ID=node117/3250692-1304093293
RUNJOBID= node117-1304093293-28597
PBS_JOBID= 13170554
Free space on /tmp= 71680 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden3k/VanDerWaerden_2-3-19_349.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250692-1304093293/watcher-3250692-1304093293 -o /tmp/evaluation-result-3250692-1304093293/solver-3250692-1304093293 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250692-1304093293.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= faed4f95ee485d18812f5d39a28df3e5
RANDOM SEED=1453078770

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      17460576 kB
Buffers:       2379972 kB
Cached:       12463496 kB
SwapCached:          0 kB
Active:        8140172 kB
Inactive:      6767572 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      17460576 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             892 kB
Writeback:           0 kB
AnonPages:       63732 kB
Mapped:          14496 kB
Slab:           518780 kB
PageTables:       4832 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS:   186376 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71680 MiB
End job on node117 at 2011-04-29 18:28:18