Trace number 3262178

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.49 1202.54

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-23_506.cnf
MD5SUM3c65ff80757b12edd41f8ee6da874877
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark89.0965
Satisfiable
(Un)Satisfiability was proved
Number of variables253
Number of clauses34386
Sum of the clauses size155873
Maximum clause length23
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 284
Number of clauses of size 331656
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52646

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262178-1304091609.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 253
0.00/0.04	c Clauses = 34386
0.00/0.04	c Literals = 155873
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 253
0.00/0.04	c Clauses = 34386
0.00/0.04	c Literals = 155873
0.00/0.04	c a2
0.19/0.23	c a1
0.59/0.60	c Independent sub-problems = 1
0.59/0.60	c Sub-problem with 253 variables and 34386 clauses
0.59/0.60	c using permutation method 4
0.59/0.60	c count = 1
0.59/0.60	c Ordering finished 0.590 seconds
0.59/0.60	c mapped 0.590 seconds
0.59/0.60	c permuted 0.590 seconds
0.59/0.60	c about to sort
0.59/0.60	c sorted
0.69/0.73	c clauses now 34386
0.69/0.73	c init finished 0.600 seconds
0.69/0.73	c tree built 0.720 seconds
0.69/0.73	c 
0.69/0.73	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.69/0.73	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.69/0.73	c ................................ | ..........                    |                                  0.7         0.0             
0.69/0.73	c ................................ | ..........                  1 |                                  0.7         1.4             
0.69/0.73	c ................................ | ..........                  2 |                                  0.7         2.8             
0.69/0.73	c ................................ | ..........                  3 |                                  0.7         4.2             
0.69/0.73	c ................................ | ..........                  4 |                                  0.7         5.6             
0.69/0.73	c ................................ | ..........                  5 |                                  0.7         6.9             
0.69/0.76	c ................................ | ..........               1000 | 18           531                 0.8      1204.8        30343
96.05/24.68	c ................................ | ..........             353000 | 32123        164914             96.4      3662.2     24162790
191.92/48.67	c ................................ | ..........             551000 | 54304        248803            192.2      2866.5     37897012
288.60/72.85	c ................................ | ..........             696000 | 67457        324153            288.8      2409.7     51111801
383.67/96.69	c ................................ | ..........             817000 | 79547        383564            384.1      2127.2     62268256
479.94/120.72	c ................................ | *.........             927000 | 90231        430874            480.1      1931.0     72313754
576.61/144.92	c ................................ | *.........            1027000 | 101219       477852            576.7      1780.8     80329168
673.26/169.15	c ................................ | *.........            1120000 | 111227       519321            673.5      1663.0     88412546
768.73/193.01	c ................................ | *.........            1202000 | 122363       561097            768.8      1563.5     90964020
864.20/216.91	c ................................ | *.........            1278000 | 131805       595146            864.2      1478.8     97225790
960.87/241.17	c ................................ | **........            1364000 | 142291       632522            961.2      1419.1    104281528
1057.16/265.28	c ................................ | **........            1443000 | 151666       667956           1057.5      1364.6    111619849
1153.43/289.30	c ................................ | **........            1515000 | 160364       698830           1153.4      1313.5    119939908
1248.49/313.14	c ................................ | **........            1582000 | 168464       735873           1248.7      1266.9    124721524
1345.16/337.31	c ................................ | **........            1645000 | 176479       769101           1345.2      1222.8    129753932
1440.62/361.29	c ................................ | ***.......            1699000 | 183835       797780           1441.0      1179.0    136183568
1536.90/385.30	c ................................ | ***.......            1762000 | 192578       825368           1536.9      1146.5    143049206
1632.39/409.29	c ................................ | ***.......            1821000 | 200946       852408           1632.7      1115.3    147389534
1729.47/433.55	c ................................ | ***.......            1875000 | 208566       877871           1729.7      1084.0    152981518
1824.92/457.48	c ................................ | ***.......            1931000 | 216709       902431           1825.3      1057.9    159724100
1919.99/481.23	c ................................ | ****......            1990000 | 224556       927789           1920.1      1036.4    163919149
2016.66/505.48	c ................................ | ****......            2044000 | 232628       954842           2017.0      1013.4    167896619
2112.12/529.37	c ................................ | ****......            2097000 | 239813       978389           2112.4       992.7    172864677
2208.42/553.49	c ................................ | ****......            2149000 | 246385       1006738          2208.8       972.9    176705923
2306.26/577.92	c ................................ | ****......            2196000 | 253434       1029932          2306.3       952.2    180152842
2400.14/601.43	c ................................ | *****.....            2244000 | 260880       1049466          2400.3       934.9    184787610
2496.41/625.54	c ................................ | *****.....            2291000 | 267902       1071330          2496.6       917.6    189555358
2593.08/649.75	c ................................ | *****.....            2339000 | 274912       1092948          2593.3       901.9    193343227
2688.15/673.55	c ................................ | *****.....            2395000 | 282816       1113713          2688.4       890.9    195625138
2784.05/697.55	c ................................ | *****.....            2455000 | 290862       1134438          2784.2       881.7    197741360
2881.89/722.08	c ................................ | ******....            2509000 | 298434       1156498          2882.2       870.5    200482638
2976.17/745.61	c ................................ | ******....            2563000 | 306487       1177625          2976.2       861.2    203227914
3073.66/770.02	c ................................ | ******....            2609000 | 312483       1199637          3073.7       848.8    206744324
3168.71/793.84	c ................................ | ******....            2651000 | 317947       1218008          3168.9       836.6    210796778
3263.78/817.69	c ................................ | ******....            2692000 | 324241       1234307          3264.2       824.7    215221180
3360.05/841.77	c ................................ | *******...            2735000 | 330243       1253529          3360.3       813.9    218810362
3455.94/865.74	c ................................ | *******...            2777000 | 336248       1271626          3456.1       803.5    223496001
3552.20/889.81	c ................................ | *******...            2820000 | 342525       1291443          3552.2       793.9    227196595
3648.07/913.89	c ................................ | *******...            2863000 | 349504       1310895          3648.4       784.7    230626257
3745.96/938.32	c ................................ | *******...            2909000 | 356015       1331066          3746.1       776.6    233948305
3842.21/962.41	c ................................ | ********..            2944000 | 361798       1349205          3842.3       766.2    237089489
3937.28/986.24	c ................................ | ********..            2977000 | 367257       1367607          3937.5       756.1    240004203
4033.15/1010.28	c ................................ | ********..            3014000 | 373100       1384803          4033.5       747.2    243345261
4128.25/1034.10	c ................................ | ********..            3052000 | 379231       1401979          4128.6       739.2    246348431
4224.50/1058.15	c ................................ | ********..            3097000 | 386123       1417210          4224.7       733.1    250169253
4321.99/1082.58	c ................................ | *********.            3141000 | 393128       1435256          4322.3       726.7    253485901
4417.44/1106.40	c ................................ | *********.            3182000 | 399798       1450976          4417.4       720.3    257055677
4513.72/1130.52	c ................................ | *********.            3223000 | 406530       1467242          4513.8       714.0    260073323
4607.98/1154.11	c ................................ | *********.            3268000 | 413258       1483919          4608.0       709.2    263588326
4703.88/1178.17	c ................................ | *********.            3310000 | 419416       1500501          4704.1       703.6    266784392

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262178-1304091609/watcher-3262178-1304091609 -o /tmp/evaluation-result-3262178-1304091609/solver-3262178-1304091609 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262178-1304091609.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.00 8.03 8.00 2/177 17454
/proc/meminfo: memFree=19964400/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) R 17452 17454 16471 0 -1 4202496 427 0 0 0 0 0 0 0 25 0 1 0 501576529 9633792 369 33554432000 4194304 4338100 140735883423392 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17454/statm: 2352 370 146 36 0 257 0

[startup+0.106531 s]
/proc/loadavg: 8.00 8.03 8.00 2/177 17454
/proc/meminfo: memFree=19964400/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=39756 CPUtime=0.1 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) R 17452 17454 16471 0 -1 4202496 6340 0 0 0 10 0 0 0 25 0 1 0 501576529 40710144 6079 33554432000 4194304 4338100 140735883423392 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17454/statm: 9939 6079 682 36 0 7844 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 39756

[startup+0.200514 s]
/proc/loadavg: 8.00 8.03 8.00 2/177 17454
/proc/meminfo: memFree=19964400/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=39756 CPUtime=0.19 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) R 17452 17454 16471 0 -1 4202496 6349 0 0 0 19 0 0 0 25 0 1 0 501576529 40710144 6079 33554432000 4194304 4338100 140735883423392 18446744073709551615 264725800657 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17454/statm: 9939 6079 673 36 0 7844 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 39756

[startup+0.300508 s]
/proc/loadavg: 8.00 8.03 8.00 2/177 17454
/proc/meminfo: memFree=19964400/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=16416 CPUtime=0.29 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) R 17452 17454 16471 0 -1 4202496 6551 0 0 0 29 0 0 0 25 0 1 0 501576529 16809984 2142 33554432000 4194304 4338100 140735883423392 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17454/statm: 4104 2142 155 36 0 2009 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 16416

[startup+0.700419 s]
/proc/loadavg: 8.00 8.03 8.00 2/177 17454
/proc/meminfo: memFree=19964400/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=38252 CPUtime=0.69 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) R 17452 17454 16471 0 -1 4202496 12088 0 0 0 69 0 0 0 25 0 1 0 501576529 39170048 7611 33554432000 4194304 4338100 140735883423392 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17454/statm: 9596 7615 164 36 0 7501 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 38252

[startup+1.50024 s]
/proc/loadavg: 8.08 8.05 8.01 10/186 17463
/proc/meminfo: memFree=19840028/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=194140 CPUtime=3.78 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 22871 0 0 0 375 3 0 0 25 0 5 0 501576529 198799360 18394 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 48535 18395 176 36 0 46440 0
[pid=17454/tid=17460] ppid=17452 vsize=194140 CPUtime=0.76 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 2373 0 0 0 76 0 0 0 18 0 5 0 501576602 198799360 18395 33554432000 4194304 4338100 140735883423392 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=194140 CPUtime=0.75 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 1797 0 0 0 75 0 0 0 18 0 5 0 501576602 198799360 18395 33554432000 4194304 4338100 140735883423392 18446744073709551615 4239539 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=194140 CPUtime=0.75 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 2006 0 0 0 75 0 0 0 25 0 5 0 501576602 198799360 18396 33554432000 4194304 4338100 140735883423392 18446744073709551615 4255933 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=194140 CPUtime=0.75 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 3186 0 0 0 75 0 0 0 18 0 5 0 501576602 198799360 18396 33554432000 4194304 4338100 140735883423392 18446744073709551615 264725800686 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3.78
Current children cumulated vsize (KiB) 194140

[startup+3.10093 s]
/proc/loadavg: 8.08 8.05 8.01 9/186 17463
/proc/meminfo: memFree=19736488/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=263884 CPUtime=10.18 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 40585 0 0 0 1012 6 0 0 25 0 5 0 501576529 270217216 36108 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 65971 36108 176 36 0 63876 0
[pid=17454/tid=17460] ppid=17452 vsize=263884 CPUtime=2.36 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 6799 0 0 0 235 1 0 0 22 0 5 0 501576602 270217216 36108 33554432000 4194304 4338100 140735883423392 18446744073709551615 4239528 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=263884 CPUtime=2.36 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 5138 0 0 0 235 1 0 0 22 0 5 0 501576602 270217216 36108 33554432000 4194304 4338100 140735883423392 18446744073709551615 4256960 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=263884 CPUtime=2.36 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 6267 0 0 0 235 1 0 0 25 0 5 0 501576602 270217216 36108 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=263884 CPUtime=2.35 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 8871 0 0 0 234 1 0 0 22 0 5 0 501576602 270217216 36108 33554432000 4194304 4338100 140735883423392 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 10.18
Current children cumulated vsize (KiB) 263884

[startup+6.30033 s]
/proc/loadavg: 8.08 8.05 8.01 9/186 17463
/proc/meminfo: memFree=19444948/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=414952 CPUtime=22.96 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 77176 0 0 0 2283 13 0 0 25 0 5 0 501576529 424910848 72699 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 103738 72702 176 36 0 101643 0
[pid=17454/tid=17460] ppid=17452 vsize=414952 CPUtime=5.55 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 12124 0 0 0 553 2 0 0 25 0 5 0 501576602 424910848 72702 33554432000 4194304 4338100 140735883423392 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=414952 CPUtime=5.56 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 20881 0 0 0 552 4 0 0 25 0 5 0 501576602 424910848 72702 33554432000 4194304 4338100 140735883423392 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=414952 CPUtime=5.56 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 14607 0 0 0 554 2 0 0 25 0 5 0 501576602 424910848 72703 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=414952 CPUtime=5.53 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 16057 0 0 0 551 2 0 0 25 0 5 0 501576602 424910848 72703 33554432000 4194304 4338100 140735883423392 18446744073709551615 4284198 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 22.96
Current children cumulated vsize (KiB) 414952

[startup+12.7011 s]
/proc/loadavg: 8.14 8.06 8.01 9/186 17463
/proc/meminfo: memFree=19015540/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=678608 CPUtime=48.52 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 127909 0 0 0 4829 23 0 0 25 0 5 0 501576529 694894592 123432 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 169652 123432 176 36 0 167557 0
[pid=17454/tid=17460] ppid=17452 vsize=678608 CPUtime=11.96 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 22294 0 0 0 1191 5 0 0 25 0 5 0 501576602 694894592 123432 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=678608 CPUtime=11.96 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 32722 0 0 0 1189 7 0 0 25 0 5 0 501576602 694894592 123432 33554432000 4194304 4338100 140735883423392 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=678608 CPUtime=11.96 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 26932 0 0 0 1192 4 0 0 25 0 5 0 501576602 694894592 123432 33554432000 4194304 4338100 140735883423392 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=678608 CPUtime=11.89 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 32451 0 0 0 1184 5 0 0 25 0 5 0 501576602 694894592 123432 33554432000 4194304 4338100 140735883423392 18446744073709551615 4239543 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 48.52
Current children cumulated vsize (KiB) 678608

[startup+25.5007 s]
/proc/loadavg: 8.12 8.06 8.01 9/186 17463
/proc/meminfo: memFree=18332688/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=1001716 CPUtime=99.65 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 209704 0 0 0 9926 39 0 0 25 0 5 0 501576529 1025757184 205227 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 250429 205227 176 36 0 248334 0
[pid=17454/tid=17460] ppid=17452 vsize=1001716 CPUtime=24.76 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 37775 0 0 0 2468 8 0 0 25 0 5 0 501576602 1025757184 205227 33554432000 4194304 4338100 140735883423392 18446744073709551615 4265603 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=1001716 CPUtime=24.75 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 61381 0 0 0 2464 11 0 0 25 0 5 0 501576602 1025757184 205227 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=1001716 CPUtime=24.75 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 42577 0 0 0 2468 7 0 0 25 0 5 0 501576602 1025757184 205227 33554432000 4194304 4338100 140735883423392 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=1001716 CPUtime=24.62 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 54461 0 0 0 2453 9 0 0 25 0 5 0 501576602 1025757184 205227 33554432000 4194304 4338100 140735883423392 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 99.65
Current children cumulated vsize (KiB) 1001716

[startup+51.1009 s]
/proc/loadavg: 8.08 8.06 8.01 9/186 17464
/proc/meminfo: memFree=17368792/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=1466244 CPUtime=201.92 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 324617 0 0 0 20128 64 0 0 25 0 5 0 501576529 1501433856 320140 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 366561 320140 176 36 0 364466 0
[pid=17454/tid=17460] ppid=17452 vsize=1466244 CPUtime=50.36 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 60845 0 0 0 5022 14 0 0 25 0 5 0 501576602 1501433856 320140 33554432000 4194304 4338100 140735883423392 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=1466244 CPUtime=50.36 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 93706 0 0 0 5018 18 0 0 25 0 5 0 501576602 1501433856 320140 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=1466244 CPUtime=50.36 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 66533 0 0 0 5023 13 0 0 25 0 5 0 501576602 1501433856 320140 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=1466244 CPUtime=50.09 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 90023 0 0 0 4992 17 0 0 25 0 5 0 501576602 1501433856 320140 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 201.92
Current children cumulated vsize (KiB) 1466244

[startup+102.301 s]
/proc/loadavg: 8.13 8.07 8.01 10/186 17465
/proc/meminfo: memFree=15791256/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=2253488 CPUtime=406.46 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 522075 0 0 0 40539 107 0 0 25 0 5 0 501576529 2307571712 517598 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 563372 517598 176 36 0 561277 0
[pid=17454/tid=17460] ppid=17452 vsize=2253488 CPUtime=101.57 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 93907 0 0 0 10135 22 0 0 25 0 5 0 501576602 2307571712 517598 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=2253488 CPUtime=101.56 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 135052 0 0 0 10129 27 0 0 25 0 5 0 501576602 2307571712 517598 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=2253488 CPUtime=101.56 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 110760 0 0 0 10133 23 0 0 25 0 5 0 501576602 2307571712 517598 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=2253488 CPUtime=101.01 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 168846 0 0 0 10069 32 0 0 25 0 5 0 501576602 2307571712 517598 33554432000 4194304 4338100 140735883423392 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 406.46
Current children cumulated vsize (KiB) 2253488

[startup+162.301 s]
/proc/loadavg: 8.10 8.07 8.01 9/186 17467
/proc/meminfo: memFree=14402204/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=2945184 CPUtime=646.1 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 694481 0 0 0 64466 144 0 0 25 0 5 0 501576529 3015868416 689876 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 736296 689878 176 36 0 734201 0
[pid=17454/tid=17460] ppid=17452 vsize=2945184 CPUtime=161.55 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 126681 0 0 0 16126 29 0 0 25 0 5 0 501576602 3015868416 689878 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=2945184 CPUtime=161.56 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 171904 0 0 0 16120 36 0 0 25 0 5 0 501576602 3015868416 689878 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=2945184 CPUtime=161.55 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 155315 0 0 0 16123 32 0 0 25 0 5 0 501576602 3015868416 689878 33554432000 4194304 4338100 140735883423392 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=2945184 CPUtime=160.67 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 227073 0 0 0 16022 45 0 0 25 0 5 0 501576602 3015868416 689878 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 646.1
Current children cumulated vsize (KiB) 2945184


################
# More data... #
################

/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 534876 0 0 0 75860 111 0 0 25 0 5 0 501576602 6916734976 1638732 33554432000 4194304 4338100 140735883423392 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3042.9
Current children cumulated vsize (KiB) 6754624

[startup+822.3 s]
/proc/loadavg: 8.12 8.10 8.02 9/186 17484
/proc/meminfo: memFree=5918356/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=7083460 CPUtime=3282.56 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 1726079 0 0 0 327878 378 0 0 25 0 5 0 501576529 7253463040 1721474 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 1770865 1721474 176 36 0 1768770 0
[pid=17454/tid=17460] ppid=17452 vsize=7083460 CPUtime=819.86 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 311692 0 0 0 81912 74 0 0 25 0 5 0 501576602 7253463040 1721474 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=7083460 CPUtime=821.58 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 462499 0 0 0 82058 100 0 0 25 0 5 0 501576602 7253463040 1721474 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=7083460 CPUtime=820.65 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 381263 0 0 0 81978 87 0 0 25 0 5 0 501576602 7253463040 1721474 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=7083460 CPUtime=819.7 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 557115 0 0 0 81856 114 0 0 25 0 5 0 501576602 7253463040 1721474 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3282.56
Current children cumulated vsize (KiB) 7083460

[startup+882.307 s]
/proc/loadavg: 8.11 8.09 8.02 9/186 17486
/proc/meminfo: memFree=5275728/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=7406040 CPUtime=3522.26 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 1806036 0 0 0 351830 396 0 0 25 0 5 0 501576529 7583784960 1801431 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 1851510 1801431 176 36 0 1849415 0
[pid=17454/tid=17460] ppid=17452 vsize=7406040 CPUtime=879.54 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 325688 0 0 0 87877 77 0 0 25 0 5 0 501576602 7583784960 1801431 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=7406040 CPUtime=881.59 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 481179 0 0 0 88053 106 0 0 25 0 5 0 501576602 7583784960 1801431 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=7406040 CPUtime=880.66 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 392694 0 0 0 87977 89 0 0 25 0 5 0 501576602 7583784960 1801431 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=7406040 CPUtime=879.71 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 592965 0 0 0 87850 121 0 0 25 0 5 0 501576602 7583784960 1801431 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3522.26
Current children cumulated vsize (KiB) 7406040

[startup+942.301 s]
/proc/loadavg: 8.04 8.07 8.02 9/186 17488
/proc/meminfo: memFree=4933432/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=7659804 CPUtime=3761.92 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 1870394 0 0 0 375778 414 0 0 25 0 5 0 501576529 7843639296 1865789 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 1914951 1865789 176 36 0 1912856 0
[pid=17454/tid=17460] ppid=17452 vsize=7659804 CPUtime=939.21 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 341366 0 0 0 93840 81 0 0 25 0 5 0 501576602 7843639296 1865789 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=7659804 CPUtime=941.59 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 497233 0 0 0 94048 111 0 0 25 0 5 0 501576602 7843639296 1865789 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=7659804 CPUtime=940.66 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 405045 0 0 0 93973 93 0 0 25 0 5 0 501576602 7843639296 1865789 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=7659804 CPUtime=939.71 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 613240 0 0 0 93844 127 0 0 25 0 5 0 501576602 7843639296 1865789 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3761.92
Current children cumulated vsize (KiB) 7659804

[startup+1002.3 s]
/proc/loadavg: 8.05 8.07 8.02 9/186 17489
/proc/meminfo: memFree=4571656/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=7891804 CPUtime=4001.59 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 1927949 0 0 0 399731 428 0 0 25 0 5 0 501576529 8081207296 1923344 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 1972951 1923344 176 36 0 1970856 0
[pid=17454/tid=17460] ppid=17452 vsize=7891804 CPUtime=998.88 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 347779 0 0 0 99805 83 0 0 25 0 5 0 501576602 8081207296 1923344 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=7891804 CPUtime=1001.59 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 516988 0 0 0 100044 115 0 0 25 0 5 0 501576602 8081207296 1923344 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=7891804 CPUtime=1000.66 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 420025 0 0 0 99970 96 0 0 25 0 5 0 501576602 8081207296 1923344 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=7891804 CPUtime=999.7 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 629647 0 0 0 99839 131 0 0 25 0 5 0 501576602 8081207296 1923344 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4001.59
Current children cumulated vsize (KiB) 7891804

[startup+1062.3 s]
/proc/loadavg: 8.06 8.07 8.01 9/186 17491
/proc/meminfo: memFree=4039380/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=8167596 CPUtime=4241.27 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 1996639 0 0 0 423684 443 0 0 25 0 5 0 501576529 8363618304 1992034 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 2041899 1992034 176 36 0 2039804 0
[pid=17454/tid=17460] ppid=17452 vsize=8167596 CPUtime=1058.56 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 362997 0 0 0 105769 87 0 0 25 0 5 0 501576602 8363618304 1992034 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=8167596 CPUtime=1061.59 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 535231 0 0 0 106039 120 0 0 25 0 5 0 501576602 8363618304 1992034 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=8167596 CPUtime=1060.66 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 435040 0 0 0 105968 98 0 0 25 0 5 0 501576602 8363618304 1992034 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=8167596 CPUtime=1059.71 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 649861 0 0 0 105835 136 0 0 25 0 5 0 501576602 8363618304 1992034 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4241.27
Current children cumulated vsize (KiB) 8167596

[startup+1122.3 s]
/proc/loadavg: 8.05 8.07 8.01 9/186 17492
/proc/meminfo: memFree=3464716/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=8427616 CPUtime=4480.95 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 2062544 0 0 0 447638 457 0 0 25 0 5 0 501576529 8629878784 2057939 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 2106904 2057939 176 36 0 2104809 0
[pid=17454/tid=17460] ppid=17452 vsize=8427616 CPUtime=1118.23 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 377294 0 0 0 111733 90 0 0 25 0 5 0 501576602 8629878784 2057939 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=8427616 CPUtime=1121.59 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 549543 0 0 0 112036 123 0 0 25 0 5 0 501576602 8629878784 2057939 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=8427616 CPUtime=1120.66 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 454586 0 0 0 111964 102 0 0 25 0 5 0 501576602 8629878784 2057939 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=8427616 CPUtime=1119.71 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 667611 0 0 0 111831 140 0 0 25 0 5 0 501576602 8629878784 2057939 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4480.95
Current children cumulated vsize (KiB) 8427616

[startup+1182.3 s]
/proc/loadavg: 8.05 8.07 8.01 9/186 17494
/proc/meminfo: memFree=2886904/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=8697020 CPUtime=4720.64 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 2129393 0 0 0 471592 472 0 0 25 0 5 0 501576529 8905748480 2124788 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 2174255 2124788 176 36 0 2172160 0
[pid=17454/tid=17460] ppid=17452 vsize=8697020 CPUtime=1177.91 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 397371 0 0 0 117697 94 0 0 25 0 5 0 501576602 8905748480 2124788 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=8697020 CPUtime=1181.59 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 561542 0 0 0 118033 126 0 0 25 0 5 0 501576602 8905748480 2124788 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=8697020 CPUtime=1180.65 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 474308 0 0 0 117959 106 0 0 25 0 5 0 501576602 8905748480 2124788 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=8697020 CPUtime=1179.71 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 682662 0 0 0 117829 142 0 0 25 0 5 0 501576602 8905748480 2124788 33554432000 4194304 4338100 140735883423392 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4720.64
Current children cumulated vsize (KiB) 8697020



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.03 8.06 8.01 5/180 17496
/proc/meminfo: memFree=11292992/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=8860960 CPUtime=4800.13 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) S 17452 17454 16471 0 -1 4202496 2172072 0 0 0 479531 482 0 0 25 0 5 0 501576529 9073623040 2167467 33554432000 4194304 4338100 140735883423392 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 2215240 2167467 176 36 0 2213145 0
[pid=17454/tid=17460] ppid=17452 vsize=8860960 CPUtime=1197.7 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 403966 0 0 0 119674 96 0 0 25 0 5 0 501576602 9073623040 2167467 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17454/tid=17461] ppid=17452 vsize=8860960 CPUtime=1201.5 cores=1,3,5,7
/proc/17454/task/17461/stat : 17461 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 572392 0 0 0 120021 129 0 0 25 0 5 0 501576602 9073623040 2167467 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17454/tid=17462] ppid=17452 vsize=8860960 CPUtime=1200.56 cores=1,3,5,7
/proc/17454/task/17462/stat : 17462 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 481797 0 0 0 119948 108 0 0 25 0 5 0 501576602 9073623040 2167467 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17454/tid=17463] ppid=17452 vsize=8860960 CPUtime=1199.62 cores=1,3,5,7
/proc/17454/task/17463/stat : 17463 (ttsth-5-4) R 17452 17454 16471 0 -1 4202560 700407 0 0 0 119815 147 0 0 25 0 5 0 501576602 9073623040 2167467 33554432000 4194304 4338100 140735883423392 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 8860960

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.31 s]
/proc/loadavg: 8.03 8.06 8.01 2/178 17497
/proc/meminfo: memFree=13613544/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=0 CPUtime=4800.25 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) Z 17452 17454 16471 0 -1 4203548 2172072 0 0 0 479533 492 0 0 18 0 2 0 501576529 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 0 0 0 0 0 0 0
[pid=17454/tid=17460] ppid=17452 vsize=0 CPUtime=1197.82 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4203588 403966 0 0 0 119675 107 0 0 25 0 2 0 501576602 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

[startup+1202.5 s]
/proc/loadavg: 8.03 8.06 8.01 2/178 17497
/proc/meminfo: memFree=13613544/32951124 swapFree=67111524/67111528
[pid=17454] ppid=17452 vsize=0 CPUtime=4800.44 cores=1,3,5,7
/proc/17454/stat : 17454 (ttsth-5-4) Z 17452 17454 16471 0 -1 4203548 2172072 0 0 0 479533 511 0 0 18 0 2 0 501576529 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17454/statm: 0 0 0 0 0 0 0
[pid=17454/tid=17460] ppid=17452 vsize=0 CPUtime=1198 cores=1,3,5,7
/proc/17454/task/17460/stat : 17460 (ttsth-5-4) R 17452 17454 16471 0 -1 4203588 403966 0 0 0 119675 125 0 0 25 0 2 0 501576602 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.44
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.54
CPU time (s): 4800.49
CPU user time (s): 4795.34
CPU system time (s): 5.15022
CPU usage (%): 399.196
Max. virtual memory (cumulated for all children) (KiB): 8860960

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.34
system time used= 5.15022
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2172072
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1177
involuntary context switches= 30345

runsolver used 1.92071 second user time and 4.65129 second system time

The end

Launcher Data 

Begin job on node141 at 2011-04-29 17:40:09
IDJOB=3262178
IDBENCH=83280
IDSOLVER=1654
FILE ID=node141/3262178-1304091609
RUNJOBID= node141-1304090404-17335
PBS_JOBID= 13170540
Free space on /tmp= 71424 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-23_506.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262178-1304091609/watcher-3262178-1304091609 -o /tmp/evaluation-result-3262178-1304091609/solver-3262178-1304091609 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262178-1304091609.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3c65ff80757b12edd41f8ee6da874877
RANDOM SEED=2112449253

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19965052 kB
Buffers:       1543632 kB
Cached:       10780808 kB
SwapCached:          4 kB
Active:        3748856 kB
Inactive:      8656860 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19965052 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            1588 kB
Writeback:           0 kB
AnonPages:       80636 kB
Mapped:          15068 kB
Slab:           516252 kB
PageTables:       4060 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   224788 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71424 MiB
End job on node141 at 2011-04-29 18:00:13