Trace number 3265038

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.22 1211.52

General information on the benchmark

NameSAT05/crafted/jarvisalo05/mod2c-rand3bip-sat/
mod2c-rand3bip-sat-250-1.shuffled-as.sat05-2533.cnf
MD5SUMdb05e30798a5a29f39775d55b01c4246
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark156.638
Satisfiable
(Un)Satisfiability was proved
Number of variables349
Number of clauses2416
Sum of the clauses size10560
Maximum clause length6
Minimum clause length4
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 41656
Number of clauses of size 5624
Number of clauses of size over 5136

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265038-1304101229.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 349
0.00/0.00	c Clauses = 2416
0.00/0.00	c Literals = 10560
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 349
0.00/0.00	c Clauses = 2416
0.00/0.00	c Literals = 10560
0.00/0.00	c a2
0.00/0.00	c a1
0.01/0.07	c Independent sub-problems = 1
0.01/0.07	c Sub-problem with 349 variables and 2416 clauses
0.01/0.07	c using permutation method 4
0.01/0.07	c count = 2
0.01/0.07	c Ordering finished 0.070 seconds
0.01/0.07	c mapped 0.070 seconds
0.01/0.07	c permuted 0.070 seconds
0.01/0.07	c about to sort
0.01/0.07	c sorted
0.01/0.08	c clauses now 2416
0.01/0.08	c init finished 0.070 seconds
0.01/0.08	c tree built 0.070 seconds
0.01/0.08	c 
0.01/0.08	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.01/0.08	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.01/0.08	c ................................ | ..........                    |                                  0.1         0.0             
0.01/0.08	c ................................ | ..........                  1 |                                  0.1        10.0             
0.01/0.08	c ................................ | ..........                  2 |                                  0.1        20.0             
0.01/0.08	c ................................ | ..........                  3 |                                  0.1        30.0             
0.01/0.08	c ................................ | ..........                  4 |              1                   0.1        40.0             
0.01/0.08	c ................................ | ..........                  5 |              1                   0.1        50.0             
0.01/0.08	c ................................ | ..........               1000 | 86           129                 0.1     10000.0         1532
95.94/24.12	c ................................ | ..........            1313000 | 316073       538168             96.0     13677.1     19775070
191.69/48.18	c ................................ | ..........            2045000 | 508575       857796            192.0     10648.8     32090783
287.75/72.29	c ................................ | ..........            2761000 | 695843       1152402           288.1      9582.8     42481166
384.14/96.40	c ................................ | ..........            3390000 | 862732       1411428           384.1      8825.1     50668700
479.81/120.45	c ................................ | *.........            4010000 | 1030352      1644957           480.0      8354.0     57510881
575.76/144.58	c ................................ | *.........            4529000 | 1173138      1847190           576.1      7861.6     64049067
671.80/168.65	c ................................ | *.........            4967000 | 1297029      2038025           672.0      7391.0     68285774
767.76/192.76	c ................................ | *.........            5398000 | 1416157      2227678           768.0      7028.6     74683072
863.95/216.95	c ................................ | *.........            5845000 | 1539872      2424151           864.2      6763.8     78333947
959.86/241.07	c ................................ | **........            6346000 | 1677210      2632697           960.2      6609.2     83229481
1056.04/265.22	c ................................ | **........            6835000 | 1813325      2838022          1056.1      6471.8     85106486
1151.72/289.38	c ................................ | **........            7322000 | 1949785      3039358          1152.0      6355.7     86709888
1247.96/313.52	c ................................ | **........            7809000 | 2086530      3238212          1248.0      6257.1     88365255
1343.82/337.65	c ................................ | **........            8254000 | 2216200      3424519          1344.0      6141.2     91936368
1439.86/361.87	c ................................ | ***.......            8715000 | 2347076      3620182          1440.1      6051.6     95084546
1535.96/386.02	c ................................ | ***.......            9151000 | 2469421      3801588          1536.0      5957.5     97536965
1631.73/410.27	c ................................ | ***.......            9568000 | 2585835      3965627          1632.0      5862.7     98472160
1727.85/434.48	c ................................ | ***.......           10029000 | 2715419      4138881          1728.2      5803.3    100054172
1823.66/458.69	c ................................ | ***.......           10508000 | 2847928      4314613          1824.0      5760.9    102824137
1919.99/482.92	c ................................ | ****......           10967000 | 2976385      4494712          1920.1      5711.7    103874258
2015.64/507.10	c ................................ | ****......           11455000 | 3112947      4685135          2016.0      5682.0    104058427
2111.80/531.38	c ................................ | ****......           11921000 | 3245842      4867310          2112.1      5644.0    106418738
2208.24/555.60	c ................................ | ****......           12279000 | 3348478      5067980          2208.2      5560.5    110158577
2303.82/579.68	c ................................ | ****......           12597000 | 3440224      5231752          2304.2      5467.1    114163666
2399.69/603.88	c ................................ | *****.....           12933000 | 3536842      5383871          2400.0      5388.7    114736124
2496.10/628.10	c ................................ | *****.....           13272000 | 3632939      5554072          2496.1      5317.1    114736124
2591.86/652.26	c ................................ | *****.....           13583000 | 3721044      5712417          2592.1      5240.2    115634882
2687.78/676.59	c ................................ | *****.....           13926000 | 3817744      5862428          2688.1      5180.6    117993588
2784.05/700.74	c ................................ | *****.....           14289000 | 3919615      6017367          2784.2      5132.2    118149089
2879.89/724.99	c ................................ | ******....           14637000 | 4016370      6163987          2880.2      5081.9    121043164
2975.89/749.17	c ................................ | ******....           14985000 | 4115378      6304967          2976.2      5035.0    124182258
3071.90/773.25	c ................................ | ******....           15327000 | 4212912      6465508          3072.1      4989.1    129604623
3167.79/797.68	c ................................ | ******....           15906000 | 4377671      6667081          3168.1      5020.6    134525232
3263.82/821.98	c ................................ | ******....           16479000 | 4547728      6853072          3264.1      5048.5    137213335
3359.88/846.35	c ................................ | *******...           17070000 | 4719288      7041254          3360.1      5080.2    141728233
3456.04/870.70	c ................................ | *******...           17560000 | 4862817      7208793          3456.1      5080.9    144162120
3552.02/895.23	c ................................ | *******...           18084000 | 5013974      7385645          3552.2      5091.0    146870646
3648.14/919.62	c ................................ | *******...           18557000 | 5150257      7559317          3648.2      5086.6    147493220
3744.07/943.83	c ................................ | *******...           19052000 | 5293719      7734354          3744.2      5088.4    147948097
3839.90/968.17	c ................................ | ********..           19508000 | 5427508      7893975          3840.2      5080.0    150304663
3936.04/992.44	c ................................ | ********..           19878000 | 5536556      8034606          3936.2      5050.0    151631198
4032.02/1016.63	c ................................ | ********..           20232000 | 5640888      8161724          4032.2      5017.7    153104332
4127.82/1040.95	c ................................ | ********..           20589000 | 5746493      8283187          4128.0      4987.6    153416150
4223.99/1065.22	c ................................ | ********..           20947000 | 5851928      8408353          4224.1      4958.9    153416150
4320.06/1089.60	c ................................ | *********.           21359000 | 5972370      8541722          4320.1      4944.1    153416150
4415.81/1113.76	c ................................ | *********.           21764000 | 6089927      8683453          4416.1      4928.4    153416150
4511.76/1137.97	c ................................ | *********.           22168000 | 6208633      8817566          4512.1      4913.0    153936490
4608.20/1162.40	c ................................ | *********.           22605000 | 6336264      8950137          4608.2      4905.4    154816231
4703.87/1186.76	c ................................ | *********.           23037000 | 6463808      9084178          4704.1      4897.2    155663073
4799.92/1211.32	c CPU limit (4800) reached
4799.92/1211.32	c Time = 4800.01 seconds
4799.92/1211.32	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265038-1304101229/watcher-3265038-1304101229 -o /tmp/evaluation-result-3265038-1304101229/solver-3265038-1304101229 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265038-1304101229.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.20 7.89 7.95 2/177 18262
/proc/meminfo: memFree=26299664/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=9640 CPUtime=0 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) R 18259 18262 17388 0 -1 4202496 484 0 0 0 0 0 0 0 24 0 1 0 521722615 9871360 426 33554432000 4194304 4338100 140734390813424 18446744073709551615 208079928858 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18262/statm: 2410 426 152 36 0 315 0

[startup+0.0206179 s]
/proc/loadavg: 7.20 7.89 7.95 2/177 18262
/proc/meminfo: memFree=26299664/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=9640 CPUtime=0.01 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) R 18259 18262 17388 0 -1 4202496 497 0 0 0 1 0 0 0 24 0 1 0 521722615 9871360 439 33554432000 4194304 4338100 140734390813424 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18262/statm: 2410 439 155 36 0 315 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 9640

[startup+0.103705 s]
/proc/loadavg: 7.20 7.89 7.95 2/177 18262
/proc/meminfo: memFree=26299664/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=198400 CPUtime=0.14 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 2013 0 0 0 14 0 0 0 24 0 5 0 521722615 203161600 1922 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 49600 1925 175 36 0 47505 0
Current children cumulated CPU time (s) 0.14
Current children cumulated vsize (KiB) 198400

[startup+0.30057 s]
/proc/loadavg: 7.20 7.89 7.95 2/177 18262
/proc/meminfo: memFree=26299664/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=205180 CPUtime=0.93 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 5317 0 0 0 91 2 0 0 24 0 5 0 521722615 210104320 5226 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 51295 5226 175 36 0 49200 0
Current children cumulated CPU time (s) 0.93
Current children cumulated vsize (KiB) 205180

[startup+0.700493 s]
/proc/loadavg: 7.20 7.89 7.95 2/177 18262
/proc/meminfo: memFree=26299664/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=230588 CPUtime=2.53 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 10876 0 0 0 248 5 0 0 24 0 5 0 521722615 236122112 10785 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 57647 10786 175 36 0 55552 0
Current children cumulated CPU time (s) 2.53
Current children cumulated vsize (KiB) 230588

[startup+1.50034 s]
/proc/loadavg: 7.20 7.89 7.95 9/186 18271
/proc/meminfo: memFree=26201904/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=327788 CPUtime=5.72 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 19583 0 0 0 562 10 0 0 24 0 5 0 521722615 335654912 19492 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 81947 19492 175 36 0 79852 0
[pid=18262/tid=18268] ppid=18259 vsize=327788 CPUtime=1.4 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 3799 0 0 0 138 2 0 0 25 0 5 0 521722623 335654912 19492 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=327788 CPUtime=1.4 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 5073 0 0 0 138 2 0 0 20 0 5 0 521722623 335654912 19493 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=327788 CPUtime=1.41 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 4799 0 0 0 139 2 0 0 25 0 5 0 521722623 335654912 19493 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=327788 CPUtime=1.4 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 4415 0 0 0 137 3 0 0 25 0 5 0 521722623 335654912 19493 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 5.72
Current children cumulated vsize (KiB) 327788

[startup+3.10104 s]
/proc/loadavg: 7.20 7.89 7.95 9/186 18271
/proc/meminfo: memFree=26122180/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=387732 CPUtime=12.11 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 35508 0 0 0 1189 22 0 0 24 0 5 0 521722615 397037568 35417 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 96933 35417 175 36 0 94838 0
[pid=18262/tid=18268] ppid=18259 vsize=387732 CPUtime=2.99 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 7263 0 0 0 294 5 0 0 25 0 5 0 521722623 397037568 35417 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=387732 CPUtime=2.99 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 9434 0 0 0 295 4 0 0 19 0 5 0 521722623 397037568 35417 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=387732 CPUtime=3.01 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 8894 0 0 0 296 5 0 0 25 0 5 0 521722623 397037568 35417 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=387732 CPUtime=3.01 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 8419 0 0 0 295 6 0 0 25 0 5 0 521722623 397037568 35417 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 12.11
Current children cumulated vsize (KiB) 387732

[startup+6.30043 s]
/proc/loadavg: 7.26 7.89 7.95 9/186 18271
/proc/meminfo: memFree=25940084/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=479448 CPUtime=24.89 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 58578 0 0 0 2448 41 0 0 24 0 5 0 521722615 490954752 58487 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 119862 58488 176 36 0 117767 0
[pid=18262/tid=18268] ppid=18259 vsize=479448 CPUtime=6.18 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 12319 0 0 0 608 10 0 0 18 0 5 0 521722623 490954752 58488 33554432000 4194304 4338100 140734390813424 18446744073709551615 208079627494 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=479448 CPUtime=6.19 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 15907 0 0 0 610 9 0 0 25 0 5 0 521722623 490954752 58488 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=479448 CPUtime=6.2 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 14513 0 0 0 611 9 0 0 25 0 5 0 521722623 490954752 58488 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=479448 CPUtime=6.2 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 14342 0 0 0 610 10 0 0 23 0 5 0 521722623 490954752 58488 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.89
Current children cumulated vsize (KiB) 479448

[startup+12.7012 s]
/proc/loadavg: 7.32 7.90 7.95 9/186 18271
/proc/meminfo: memFree=25670408/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=635028 CPUtime=50.42 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 97371 0 0 0 4958 84 0 0 24 0 5 0 521722615 650268672 97280 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 158757 97282 176 36 0 156662 0
[pid=18262/tid=18268] ppid=18259 vsize=635028 CPUtime=12.53 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 21414 0 0 0 1232 21 0 0 25 0 5 0 521722623 650268672 97282 33554432000 4194304 4338100 140734390813424 18446744073709551615 208079928817 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=635028 CPUtime=12.59 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 25470 0 0 0 1239 20 0 0 25 0 5 0 521722623 650268672 97282 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=635028 CPUtime=12.59 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 24277 0 0 0 1240 19 0 0 25 0 5 0 521722623 650268672 97283 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=635028 CPUtime=12.59 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 24715 0 0 0 1237 22 0 0 20 0 5 0 521722623 650268672 97283 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 50.42
Current children cumulated vsize (KiB) 635028

[startup+25.5008 s]
/proc/loadavg: 7.47 7.90 7.95 9/186 18271
/proc/meminfo: memFree=25147632/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=909760 CPUtime=101.5 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 167943 0 0 0 9995 155 0 0 24 0 5 0 521722615 931594240 167852 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 227440 167852 176 36 0 225345 0
[pid=18262/tid=18268] ppid=18259 vsize=909760 CPUtime=25.26 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 38066 0 0 0 2489 37 0 0 19 0 5 0 521722623 931594240 167854 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=909760 CPUtime=25.38 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 43920 0 0 0 2498 40 0 0 25 0 5 0 521722623 931594240 167854 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=909760 CPUtime=25.37 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 41108 0 0 0 2502 35 0 0 25 0 5 0 521722623 931594240 167854 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=909760 CPUtime=25.38 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 43353 0 0 0 2497 41 0 0 25 0 5 0 521722623 931594240 167854 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 101.5
Current children cumulated vsize (KiB) 909760

[startup+51.1009 s]
/proc/loadavg: 7.65 7.91 7.95 10/186 18272
/proc/meminfo: memFree=24471940/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=1315080 CPUtime=203.67 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 271600 0 0 0 20099 268 0 0 24 0 5 0 521722615 1346641920 271509 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 328770 271509 176 36 0 326675 0
[pid=18262/tid=18268] ppid=18259 vsize=1315080 CPUtime=50.83 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 62119 0 0 0 5019 64 0 0 25 0 5 0 521722623 1346641920 271509 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=1315080 CPUtime=50.95 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 73070 0 0 0 5025 70 0 0 18 0 5 0 521722623 1346641920 271509 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=1315080 CPUtime=50.83 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 64542 0 0 0 5023 60 0 0 25 0 5 0 521722623 1346641920 271509 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=1315080 CPUtime=50.95 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 70371 0 0 0 5023 72 0 0 25 0 5 0 521722623 1346641920 271509 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 203.67
Current children cumulated vsize (KiB) 1315080

[startup+102.306 s]
/proc/loadavg: 7.89 7.94 7.96 9/186 18273
/proc/meminfo: memFree=23465144/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=1925016 CPUtime=407.68 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 425799 0 0 0 40299 469 0 0 24 0 5 0 521722615 1971216384 425708 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 481254 425708 176 36 0 479159 0
[pid=18262/tid=18268] ppid=18259 vsize=1925016 CPUtime=101.66 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 102490 0 0 0 10049 117 0 0 25 0 5 0 521722623 1971216384 425708 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=1925016 CPUtime=102 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 109564 0 0 0 10081 119 0 0 25 0 5 0 521722623 1971216384 425708 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=1925016 CPUtime=101.91 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 104959 0 0 0 10080 111 0 0 25 0 5 0 521722623 1971216384 425708 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=1925016 CPUtime=102 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 107288 0 0 0 10079 121 0 0 25 0 5 0 521722623 1971216384 425708 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.68
Current children cumulated vsize (KiB) 1925016

[startup+162.301 s]
/proc/loadavg: 8.00 7.96 7.96 9/186 18275
/proc/meminfo: memFree=22500268/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=2423808 CPUtime=646.69 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 553259 0 0 0 64026 643 0 0 24 0 5 0 521722615 2481979392 553168 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 605952 553168 176 36 0 603857 0
[pid=18262/tid=18268] ppid=18259 vsize=2423808 CPUtime=161.21 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 144771 0 0 0 15953 168 0 0 25 0 5 0 521722623 2481979392 553168 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=2423808 CPUtime=161.9 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 139986 0 0 0 16032 158 0 0 25 0 5 0 521722623 2481979392 553168 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=2423808 CPUtime=161.65 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 129488 0 0 0 16013 152 0 0 25 0 5 0 521722623 2481979392 553168 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=2423808 CPUtime=161.81 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 137516 0 0 0 16018 163 0 0 25 0 5 0 521722623 2481979392 553168 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.69
Current children cumulated vsize (KiB) 2423808


################
# More data... #
################

/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 204200 0 0 0 69294 498 0 0 25 0 5 0 521722623 4235452416 1001033 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2790.43
Current children cumulated vsize (KiB) 4136184

[startup+762.306 s]
/proc/loadavg: 8.03 8.01 7.99 9/186 18291
/proc/meminfo: memFree=18764248/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=4426904 CPUtime=3028.55 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1075858 0 0 0 300746 2109 0 0 24 0 5 0 521722615 4533149696 1075767 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1106726 1075767 176 36 0 1104631 0
[pid=18262/tid=18268] ppid=18259 vsize=4426904 CPUtime=755.15 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 336195 0 0 0 74958 557 0 0 25 0 5 0 521722623 4533149696 1075767 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=4426904 CPUtime=757.99 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 233493 0 0 0 75280 519 0 0 25 0 5 0 521722623 4533149696 1075767 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=4426904 CPUtime=757.86 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 279038 0 0 0 75284 502 0 0 25 0 5 0 521722623 4533149696 1075767 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=4426904 CPUtime=757.44 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 225634 0 0 0 75215 529 0 0 25 0 5 0 521722623 4533149696 1075767 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3028.55
Current children cumulated vsize (KiB) 4426904

[startup+822.301 s]
/proc/loadavg: 7.94 7.98 7.98 9/186 18292
/proc/meminfo: memFree=18362828/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=4737672 CPUtime=3265.42 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1156912 0 0 0 324279 2263 0 0 24 0 5 0 521722615 4851376128 1156821 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1184418 1156821 176 36 0 1182323 0
[pid=18262/tid=18268] ppid=18259 vsize=4737672 CPUtime=814.3 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 363882 0 0 0 80822 608 0 0 25 0 5 0 521722623 4851376128 1156821 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=4737672 CPUtime=817.25 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 242760 0 0 0 81180 545 0 0 25 0 5 0 521722623 4851376128 1156821 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=4737672 CPUtime=817.21 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 309826 0 0 0 81170 551 0 0 25 0 5 0 521722623 4851376128 1156821 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=4737672 CPUtime=816.56 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 238946 0 0 0 81098 558 0 0 25 0 5 0 521722623 4851376128 1156821 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3265.42
Current children cumulated vsize (KiB) 4737672

[startup+882.301 s]
/proc/loadavg: 7.98 7.98 7.98 9/186 18294
/proc/meminfo: memFree=18052176/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=4976368 CPUtime=3500.85 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1219947 0 0 0 347678 2407 0 0 24 0 5 0 521722615 5095800832 1219856 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1244092 1219856 176 36 0 1241997 0
[pid=18262/tid=18268] ppid=18259 vsize=4976368 CPUtime=872.86 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 392630 0 0 0 86630 656 0 0 25 0 5 0 521722623 5095800832 1219857 33554432000 4194304 4338100 140734390813424 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=4976368 CPUtime=876.27 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 245128 0 0 0 87060 567 0 0 25 0 5 0 521722623 5095800832 1219857 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=4976368 CPUtime=876.19 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 332398 0 0 0 87023 596 0 0 25 0 5 0 521722623 5095800832 1219857 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=4976368 CPUtime=875.41 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 248294 0 0 0 86955 586 0 0 25 0 5 0 521722623 5095800832 1219858 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3500.85
Current children cumulated vsize (KiB) 4976368

[startup+942.301 s]
/proc/loadavg: 7.99 7.98 7.98 9/186 18296
/proc/meminfo: memFree=17710156/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=5074048 CPUtime=3738.07 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1245858 0 0 0 371268 2539 0 0 24 0 5 0 521722615 5195825152 1245767 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1268512 1245767 176 36 0 1266417 0
[pid=18262/tid=18268] ppid=18259 vsize=5074048 CPUtime=931.99 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 405005 0 0 0 92504 695 0 0 25 0 5 0 521722623 5195825152 1245767 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=5074048 CPUtime=935.82 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 246843 0 0 0 92987 595 0 0 25 0 5 0 521722623 5195825152 1245767 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=5074048 CPUtime=935.58 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 341717 0 0 0 92924 634 0 0 25 0 5 0 521722623 5195825152 1245767 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=5074048 CPUtime=934.57 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 250795 0 0 0 92844 613 0 0 25 0 5 0 521722623 5195825152 1245767 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092046491 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3738.07
Current children cumulated vsize (KiB) 5074048

[startup+1002.3 s]
/proc/loadavg: 8.01 7.98 7.98 9/186 18297
/proc/meminfo: memFree=17379052/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=5277288 CPUtime=3975.36 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1282793 0 0 0 394874 2662 0 0 24 0 5 0 521722615 5403942912 1282702 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1319322 1282702 176 36 0 1317227 0
[pid=18262/tid=18268] ppid=18259 vsize=5277288 CPUtime=991.2 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 417972 0 0 0 98388 732 0 0 25 0 5 0 521722623 5403942912 1282702 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=5277288 CPUtime=994.83 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 248470 0 0 0 98866 617 0 0 25 0 5 0 521722623 5403942912 1282702 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=5277288 CPUtime=995.35 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 356753 0 0 0 98865 670 0 0 25 0 5 0 521722623 5403942912 1282702 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=5277288 CPUtime=993.85 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 258100 0 0 0 98745 640 0 0 25 0 5 0 521722623 5403942912 1282702 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3975.36
Current children cumulated vsize (KiB) 5277288

[startup+1062.3 s]
/proc/loadavg: 8.04 8.00 7.98 9/186 18299
/proc/meminfo: memFree=17234692/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=5409912 CPUtime=4212.5 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1298835 0 0 0 418479 2771 0 0 24 0 5 0 521722615 5539749888 1298744 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1352478 1298744 176 36 0 1350383 0
[pid=18262/tid=18268] ppid=18259 vsize=5409912 CPUtime=1050.51 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 418665 0 0 0 104285 766 0 0 25 0 5 0 521722623 5539749888 1298744 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=5409912 CPUtime=1053.86 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 254662 0 0 0 104749 637 0 0 25 0 5 0 521722623 5539749888 1298744 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=5409912 CPUtime=1054.67 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 360660 0 0 0 104764 703 0 0 25 0 5 0 521722623 5539749888 1298744 33554432000 4194304 4338100 140734390813424 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=5409912 CPUtime=1053.35 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 263350 0 0 0 104672 663 0 0 25 0 5 0 521722623 5539749888 1298744 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092058293 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4212.5
Current children cumulated vsize (KiB) 5409912

[startup+1122.3 s]
/proc/loadavg: 7.97 7.99 7.98 9/186 18300
/proc/meminfo: memFree=17162892/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=5499128 CPUtime=4449.83 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1305115 0 0 0 442102 2881 0 0 24 0 5 0 521722615 5631107072 1305024 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1374782 1305024 176 36 0 1372687 0
[pid=18262/tid=18268] ppid=18259 vsize=5499128 CPUtime=1109.54 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) S 18259 18262 17388 0 -1 4202560 421678 0 0 0 110153 801 0 0 25 0 5 0 521722623 5631107072 1305024 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092045062 0 0 4096 0 18446604471145001408 0 0 -1 5 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=5499128 CPUtime=1113.32 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 255253 0 0 0 110675 657 0 0 25 0 5 0 521722623 5631107072 1305024 33554432000 4194304 4338100 140734390813424 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=5499128 CPUtime=1114.01 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) S 18259 18262 17388 0 -1 4202560 362633 0 0 0 110664 737 0 0 25 0 5 0 521722623 5631107072 1305024 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092045062 0 0 4096 0 18446604471145001408 0 0 -1 7 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=5499128 CPUtime=1112.84 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) S 18259 18262 17388 0 -1 4202560 264053 0 0 0 110600 684 0 0 21 0 5 0 521722623 5631107072 1305024 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092045062 0 0 4096 0 18446604471145001408 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4449.83
Current children cumulated vsize (KiB) 5499128

[startup+1182.3 s]
/proc/loadavg: 8.06 8.02 7.99 9/186 18302
/proc/meminfo: memFree=17022508/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=5561888 CPUtime=4686.5 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) S 18259 18262 17388 0 -1 4202496 1322796 0 0 0 465655 2995 0 0 24 0 5 0 521722615 5695373312 1322705 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092035045 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 1390472 1322705 176 36 0 1388377 0
[pid=18262/tid=18268] ppid=18259 vsize=5561888 CPUtime=1168.45 cores=1,3,5,7
/proc/18262/task/18268/stat : 18268 (ttsth-5-4) S 18259 18262 17388 0 -1 4202560 432079 0 0 0 116003 842 0 0 25 0 5 0 521722623 5695373312 1322705 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092045062 0 0 4096 0 18446604471145001408 0 0 -1 7 0 0 0
[pid=18262/tid=18269] ppid=18259 vsize=5561888 CPUtime=1172.52 cores=1,3,5,7
/proc/18262/task/18269/stat : 18269 (ttsth-5-4) S 18259 18262 17388 0 -1 4202560 255803 0 0 0 116575 677 0 0 25 0 5 0 521722623 5695373312 1322705 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092045062 0 0 4096 0 18446604471145001408 0 0 -1 1 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=5561888 CPUtime=1173.04 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) S 18259 18262 17388 0 -1 4202560 363460 0 0 0 116535 769 0 0 25 0 5 0 521722623 5695373312 1322705 33554432000 4194304 4338100 140734390813424 18446744073709551615 208092045062 0 0 4096 0 18446604471145001408 0 0 -1 5 0 0 0
[pid=18262/tid=18271] ppid=18259 vsize=5561888 CPUtime=1172.37 cores=1,3,5,7
/proc/18262/task/18271/stat : 18271 (ttsth-5-4) R 18259 18262 17388 0 -1 4202560 269956 0 0 0 116533 704 0 0 20 0 5 0 521722623 5695373312 1322705 33554432000 4194304 4338100 140734390813424 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4686.5
Current children cumulated vsize (KiB) 5561888



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1211.4 s]
/proc/loadavg: 7.88 7.97 7.98 6/183 18303
/proc/meminfo: memFree=18977756/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) Z 18259 18262 17388 0 -1 4203548 1331830 0 0 0 476952 3058 0 0 18 0 2 0 521722615 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 0 0 0 0 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=0 CPUtime=1201.62 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4203588 364970 0 0 0 119369 793 0 0 25 0 2 0 521722623 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1211.51 s]
/proc/loadavg: 7.88 7.97 7.98 6/183 18303
/proc/meminfo: memFree=18977756/32950928 swapFree=67111364/67111528
[pid=18262] ppid=18259 vsize=0 CPUtime=4800.2 cores=1,3,5,7
/proc/18262/stat : 18262 (ttsth-5-4) Z 18259 18262 17388 0 -1 4203548 1331830 0 0 0 476952 3068 0 0 18 0 2 0 521722615 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18262/statm: 0 0 0 0 0 0 0
[pid=18262/tid=18270] ppid=18259 vsize=0 CPUtime=1201.72 cores=1,3,5,7
/proc/18262/task/18270/stat : 18270 (ttsth-5-4) R 18259 18262 17388 0 -1 4203588 364970 0 0 0 119369 803 0 0 25 0 2 0 521722623 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1211.52
CPU time (s): 4800.22
CPU user time (s): 4769.53
CPU system time (s): 30.6923
CPU usage (%): 396.215
Max. virtual memory (cumulated for all children) (KiB): 5588448

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4769.53
system time used= 30.6923
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1331830
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 95921
involuntary context switches= 30501

runsolver used 2.09468 second user time and 4.42333 second system time

The end

Launcher Data 

Begin job on node127 at 2011-04-29 20:20:30
IDJOB=3265038
IDBENCH=85368
IDSOLVER=1654
FILE ID=node127/3265038-1304101229
RUNJOBID= node127-1304101229-18243
PBS_JOBID= 13170587
Free space on /tmp= 71820 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/jarvisalo05/mod2c-rand3bip-sat/mod2c-rand3bip-sat-250-1.shuffled-as.sat05-2533.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265038-1304101229/watcher-3265038-1304101229 -o /tmp/evaluation-result-3265038-1304101229/solver-3265038-1304101229 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265038-1304101229.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= db05e30798a5a29f39775d55b01c4246
RANDOM SEED=1669465435

node127.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.60
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.823
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      26300200 kB
Buffers:       1402536 kB
Cached:        4707780 kB
SwapCached:          0 kB
Active:        1789544 kB
Inactive:      4385652 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      26300200 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:             108 kB
Writeback:           0 kB
AnonPages:       64652 kB
Mapped:          14876 kB
Slab:           410948 kB
PageTables:       4500 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   196144 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71820 MiB
End job on node127 at 2011-04-29 20:40:43