Trace number 3260198

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.43 1205.44

General information on the benchmark

Namecrafted/mosoi/
289/289-sat-11x10.cnf
MD5SUM5d217b4e2e013436c5df3f8194310a1a
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.009998
Satisfiable
(Un)Satisfiability was proved
Number of variables440
Number of clauses10670
Sum of the clauses size41360
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2660
Number of clauses of size 30
Number of clauses of size 410010
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260198-1304063811.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 440
0.00/0.01	c Clauses = 10670
0.00/0.01	c Literals = 41360
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 440
0.00/0.01	c Clauses = 10670
0.00/0.01	c Literals = 41360
0.00/0.01	c a2
0.02/0.03	c a1
0.40/0.49	c Independent sub-problems = 1
0.40/0.49	c Sub-problem with 440 variables and 10670 clauses
0.40/0.49	c using permutation method 4
0.40/0.49	c count = 2
0.40/0.49	c Ordering finished 0.490 seconds
0.40/0.49	c mapped 0.490 seconds
0.40/0.49	c permuted 0.490 seconds
0.40/0.49	c about to sort
0.40/0.49	c sorted
0.49/0.51	c clauses now 10670
0.49/0.51	c init finished 0.490 seconds
0.49/0.51	c tree built 0.510 seconds
0.49/0.51	c 
0.49/0.51	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.51	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.51	c ................................ | ..........                    |                                  0.5         0.0             
0.49/0.51	c ................................ | ..........                  1 |                                  0.5         2.0             
0.49/0.51	c ................................ | ..........                  2 |                                  0.5         3.9             
0.49/0.51	c ................................ | ..........                  3 |                                  0.5         5.9             
0.49/0.51	c ................................ | ..........                  4 |                                  0.5         7.8             
0.49/0.51	c ................................ | ..........                  5 |                                  0.5         9.8             
0.49/0.52	c ................................ | ..........               1000 | 16           2222                0.5      1886.8         2781
95.81/24.46	c ................................ | ..........             736000 | 148056       1390979            96.0      7662.7     13117200
191.89/48.57	c ................................ | ..........            1176000 | 259284       2178666           192.2      6118.6     18154180
287.95/72.65	c ................................ | ..........            1520000 | 356798       2684066           288.1      5275.4     24192354
383.88/96.79	c ................................ | ..........            1861000 | 451613       3136213           384.2      4843.2     29140459
479.87/120.90	c ................................ | *.........            2194000 | 526864       3581483           480.3      4568.1     32259669
575.94/144.93	c ................................ | *.........            2501000 | 596045       3920752           576.1      4341.5     35034337
672.24/169.10	c ................................ | *.........            2747000 | 664488       4172004           672.2      4086.3     36989529
768.22/193.22	c ................................ | *.........            2990000 | 731361       4423925           768.3      3891.7     38644820
863.92/217.29	c ................................ | *.........            3256000 | 820503       4643949           864.3      3767.2     39693473
960.31/241.42	c ................................ | **........            3541000 | 911051       4873309           960.4      3687.0     41923300
1055.92/265.44	c ................................ | **........            3758000 | 970541       5099618          1056.1      3558.3     44946846
1152.00/289.51	c ................................ | **........            3980000 | 1027424      5322393          1152.0      3454.7     48441581
1248.25/313.71	c ................................ | **........            4178000 | 1085035      5531060          1248.3      3346.9     51113293
1344.27/337.81	c ................................ | **........            4379000 | 1142174      5752237          1344.3      3257.4     52573667
1439.89/361.85	c ................................ | ***.......            4595000 | 1212665      5938323          1440.1      3190.8     54951095
1536.03/386.09	c ................................ | ***.......            4821000 | 1283955      6111397          1536.4      3137.9     56077377
1632.46/410.22	c ................................ | ***.......            5006000 | 1339349      6302107          1632.5      3066.4     57599362
1728.06/434.29	c ................................ | ***.......            5169000 | 1385519      6475157          1728.4      2990.6     59000785
1824.39/458.41	c ................................ | ***.......            5332000 | 1429454      6661233          1824.4      2922.6     60074484
1920.51/482.51	c ................................ | ****......            5487000 | 1475741      6837050          1920.6      2857.0     62323356
2016.23/506.60	c ................................ | ****......            5637000 | 1522456      6964747          2016.2      2795.8     64200206
2111.72/530.69	c ................................ | ****......            5788000 | 1572250      7088374          2112.1      2740.4     64654098
2207.83/554.74	c ................................ | ****......            5959000 | 1629906      7220263          2208.0      2698.8     64978963
2303.95/578.85	c ................................ | ****......            6133000 | 1685360      7348468          2304.2      2661.7     65547093
2400.04/603.04	c ................................ | *****.....            6275000 | 1728585      7467176          2400.2      2614.3     65769781
2496.27/627.15	c ................................ | *****.....            6420000 | 1774465      7595786          2496.5      2571.6     65988110
2591.65/651.10	c ................................ | *****.....            6565000 | 1818549      7719450          2592.1      2532.7     66449679
2688.45/675.33	c ................................ | *****.....            6711000 | 1867609      7840206          2688.6      2496.1     67039893
2784.48/699.42	c ................................ | *****.....            6848000 | 1911782      7950475          2784.6      2459.2     67039893
2879.95/723.49	c ................................ | ******....            6997000 | 1961241      8053616          2880.3      2429.2     67306387
2976.35/747.62	c ................................ | ******....            7131000 | 2002314      8165773          2976.5      2395.8     67955321
3072.58/771.70	c ................................ | ******....            7259000 | 2042022      8261960          3072.6      2362.5     69415046
3168.23/795.77	c ................................ | ******....            7389000 | 2079452      8367858          3168.5      2332.0     70818401
3263.90/819.77	c ................................ | ******....            7518000 | 2120678      8452264          3264.2      2303.2     72306837
3360.39/843.96	c ................................ | *******...            7650000 | 2161419      8531198          3360.7      2276.3     73990016
3456.24/868.01	c ................................ | *******...            7784000 | 2206110      8615451          3456.3      2252.1     74374850
3551.98/892.01	c ................................ | *******...            7913000 | 2245415      8694074          3552.0      2227.8     74765115
3648.05/916.10	c ................................ | *******...            8035000 | 2281899      8770789          3648.1      2202.5     76010626
3744.23/940.36	c ................................ | *******...            8155000 | 2319843      8859286          3744.5      2177.9     76924509
3840.21/964.44	c ................................ | ********..            8289000 | 2356448      8972017          3840.4      2158.4     77738027
3936.05/988.44	c ................................ | ********..            8421000 | 2394511      9068209          3936.2      2139.4     79122346
4031.72/1012.49	c ................................ | ********..            8539000 | 2431138      9161979          4032.1      2117.8     80227738
4128.10/1036.69	c ................................ | ********..            8669000 | 2472578      9256840          4128.5      2099.8     81548340
4224.72/1060.91	c ................................ | ********..            8888000 | 2554342      9336743          4224.8      2103.8     82817318
4320.06/1084.83	c ................................ | *********.            9009000 | 2598022      9418522          4320.2      2085.3     83577983
4416.53/1109.06	c ................................ | *********.            9144000 | 2645878      9517791          4416.8      2070.3     83941075
4512.19/1133.01	c ................................ | *********.            9316000 | 2708767      9586758          4512.2      2064.6     84608878
4608.13/1157.13	c ................................ | *********.            9523000 | 2782941      9659654          4608.3      2066.5     85290343
4704.21/1181.30	c ................................ | *********.            9652000 | 2828456      9743345          4704.6      2051.6     86083879

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3260198-1304063811/watcher-3260198-1304063811 -o /tmp/evaluation-result-3260198-1304063811/solver-3260198-1304063811 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260198-1304063811.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.73 6.89 7.53 4/176 29572
/proc/meminfo: memFree=22194980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=9244 CPUtime=0 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) R 29570 29572 28847 0 -1 4202496 389 0 0 0 0 0 0 0 23 0 1 0 517979893 9465856 331 33554432000 4194304 4338100 140734874892112 18446744073709551615 268272349893 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29572/statm: 2311 332 147 36 0 216 0

[startup+0.0248079 s]
/proc/loadavg: 4.73 6.89 7.53 4/176 29572
/proc/meminfo: memFree=22194980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=11736 CPUtime=0.02 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) R 29570 29572 28847 0 -1 4202496 1027 0 0 0 2 0 0 0 23 0 1 0 517979893 12017664 968 33554432000 4194304 4338100 140734874892112 18446744073709551615 268272084184 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29572/statm: 2934 968 152 36 0 839 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 11736

[startup+0.100794 s]
/proc/loadavg: 4.73 6.89 7.53 4/176 29572
/proc/meminfo: memFree=22194980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=11948 CPUtime=0.1 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) R 29570 29572 28847 0 -1 4202496 1074 0 0 0 10 0 0 0 23 0 1 0 517979893 12234752 1015 33554432000 4194304 4338100 140734874892112 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29572/statm: 2987 1015 155 36 0 892 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 11948

[startup+0.300758 s]
/proc/loadavg: 4.73 6.89 7.53 4/176 29572
/proc/meminfo: memFree=22194980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=11948 CPUtime=0.3 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) R 29570 29572 28847 0 -1 4202496 1074 0 0 0 30 0 0 0 25 0 1 0 517979893 12234752 1015 33554432000 4194304 4338100 140734874892112 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/29572/statm: 2987 1015 155 36 0 892 0
Current children cumulated CPU time (s) 0.3
Current children cumulated vsize (KiB) 11948

[startup+0.700691 s]
/proc/loadavg: 4.73 6.89 7.53 4/176 29572
/proc/meminfo: memFree=22194980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=202060 CPUtime=1.24 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 4469 0 0 0 124 0 0 0 25 0 5 0 517979893 206909440 4321 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 50515 4321 175 36 0 48420 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 202060

[startup+1.50055 s]
/proc/loadavg: 4.73 6.89 7.53 10/186 29582
/proc/meminfo: memFree=22124600/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=240500 CPUtime=4.44 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 13472 0 0 0 443 1 0 0 25 0 5 0 517979893 246272000 13324 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 60125 13325 175 36 0 58030 0
[pid=29572/tid=29579] ppid=29570 vsize=240500 CPUtime=0.98 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 2662 0 0 0 98 0 0 0 25 0 5 0 517979944 246272000 13325 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=240500 CPUtime=0.97 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 2803 0 0 0 97 0 0 0 25 0 5 0 517979944 246272000 13325 33554432000 4194304 4338100 140734874892112 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=240500 CPUtime=0.98 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 2744 0 0 0 98 0 0 0 25 0 5 0 517979944 246272000 13325 33554432000 4194304 4338100 140734874892112 18446744073709551615 268272385677 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=240500 CPUtime=0.97 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 2864 0 0 0 97 0 0 0 25 0 5 0 517979944 246272000 13325 33554432000 4194304 4338100 140734874892112 18446744073709551615 268272084193 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4.44
Current children cumulated vsize (KiB) 240500

[startup+3.10027 s]
/proc/loadavg: 5.07 6.93 7.54 10/186 29582
/proc/meminfo: memFree=22059908/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=282196 CPUtime=10.83 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 25334 0 0 0 1078 5 0 0 25 0 5 0 517979893 288968704 25186 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 70549 25186 175 36 0 68454 0
[pid=29572/tid=29579] ppid=29570 vsize=282196 CPUtime=2.58 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 5293 0 0 0 257 1 0 0 25 0 5 0 517979944 288968704 25186 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=282196 CPUtime=2.57 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 5802 0 0 0 256 1 0 0 25 0 5 0 517979944 288968704 25186 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=282196 CPUtime=2.57 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 5714 0 0 0 256 1 0 0 25 0 5 0 517979944 288968704 25186 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=282196 CPUtime=2.56 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 6125 0 0 0 256 0 0 0 25 0 5 0 517979944 288968704 25186 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 10.83
Current children cumulated vsize (KiB) 282196

[startup+6.3007 s]
/proc/loadavg: 5.07 6.93 7.54 10/186 29582
/proc/meminfo: memFree=21920412/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=355428 CPUtime=23.61 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 43640 0 0 0 2352 9 0 0 25 0 5 0 517979893 363958272 43492 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 88857 43492 175 36 0 86762 0
[pid=29572/tid=29579] ppid=29570 vsize=355428 CPUtime=5.77 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 9556 0 0 0 576 1 0 0 25 0 5 0 517979944 363958272 43492 33554432000 4194304 4338100 140734874892112 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=355428 CPUtime=5.77 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 10553 0 0 0 575 2 0 0 25 0 5 0 517979944 363958272 43492 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=355428 CPUtime=5.77 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 10171 0 0 0 575 2 0 0 25 0 5 0 517979944 363958272 43492 33554432000 4194304 4338100 140734874892112 18446744073709551615 4287924 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=355428 CPUtime=5.75 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 10960 0 0 0 574 1 0 0 25 0 5 0 517979944 363958272 43492 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.61
Current children cumulated vsize (KiB) 355428

[startup+12.7006 s]
/proc/loadavg: 5.52 6.96 7.55 10/186 29582
/proc/meminfo: memFree=21716944/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=470404 CPUtime=49.14 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 74084 0 0 0 4898 16 0 0 25 0 5 0 517979893 481693696 73928 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 117601 73929 176 36 0 115506 0
[pid=29572/tid=29579] ppid=29570 vsize=470404 CPUtime=12.17 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 16407 0 0 0 1214 3 0 0 25 0 5 0 517979944 481693696 73929 33554432000 4194304 4338100 140734874892112 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=470404 CPUtime=12.17 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 18573 0 0 0 1213 4 0 0 25 0 5 0 517979944 481693696 73929 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=470404 CPUtime=12.16 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 17137 0 0 0 1212 4 0 0 25 0 5 0 517979944 481693696 73929 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=470404 CPUtime=12.1 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 19568 0 0 0 1207 3 0 0 25 0 5 0 517979944 481693696 73929 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.14
Current children cumulated vsize (KiB) 470404

[startup+25.5003 s]
/proc/loadavg: 5.90 7.00 7.55 10/186 29582
/proc/meminfo: memFree=21468320/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=625532 CPUtime=100.2 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 113396 0 0 0 9992 28 0 0 25 0 5 0 517979893 640544768 113209 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 156383 113209 176 36 0 154288 0
[pid=29572/tid=29579] ppid=29570 vsize=625532 CPUtime=24.96 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 23813 0 0 0 2491 5 0 0 25 0 5 0 517979944 640544768 113209 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=625532 CPUtime=24.95 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 29012 0 0 0 2488 7 0 0 25 0 5 0 517979944 640544768 113209 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=625532 CPUtime=24.92 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 27503 0 0 0 2485 7 0 0 25 0 5 0 517979944 640544768 113209 33554432000 4194304 4338100 140734874892112 18446744073709551615 4286285 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=625532 CPUtime=24.83 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 30668 0 0 0 2476 7 0 0 25 0 5 0 517979944 640544768 113209 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.2
Current children cumulated vsize (KiB) 625532

[startup+51.1007 s]
/proc/loadavg: 6.62 7.08 7.56 9/186 29614
/proc/meminfo: memFree=21453524/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=820368 CPUtime=202.26 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 164368 0 0 0 20179 47 0 0 25 0 5 0 517979893 840056832 164181 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 205092 164181 176 36 0 202997 0
[pid=29572/tid=29579] ppid=29570 vsize=820368 CPUtime=50.44 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 34296 0 0 0 5035 9 0 0 25 0 5 0 517979944 840056832 164181 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=820368 CPUtime=50.51 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 39464 0 0 0 5040 11 0 0 25 0 5 0 517979944 840056832 164182 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=820368 CPUtime=50.46 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 38370 0 0 0 5035 11 0 0 25 0 5 0 517979944 840056832 164182 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=820368 CPUtime=50.3 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 49839 0 0 0 5016 14 0 0 25 0 5 0 517979944 840056832 164182 33554432000 4194304 4338100 140734874892112 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 202.26
Current children cumulated vsize (KiB) 820368

[startup+102.307 s]
/proc/loadavg: 7.37 7.20 7.57 9/186 29646
/proc/meminfo: memFree=20777220/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=1167108 CPUtime=406.24 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 253738 0 0 0 40549 75 0 0 25 0 5 0 517979893 1195118592 253551 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 291777 253551 176 36 0 289682 0
[pid=29572/tid=29579] ppid=29570 vsize=1167108 CPUtime=101.2 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 56563 0 0 0 10105 15 0 0 25 0 5 0 517979944 1195118592 253551 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254413 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=1167108 CPUtime=101.59 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 63815 0 0 0 10141 18 0 0 25 0 5 0 517979944 1195118592 253551 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=1167108 CPUtime=101.53 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 58053 0 0 0 10134 19 0 0 25 0 5 0 517979944 1195118592 253551 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=1167108 CPUtime=101.39 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 72907 0 0 0 10117 22 0 0 25 0 5 0 517979944 1195118592 253551 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 406.24
Current children cumulated vsize (KiB) 1167108

[startup+162.301 s]
/proc/loadavg: 7.82 7.35 7.60 9/186 29648
/proc/meminfo: memFree=20138312/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=1401756 CPUtime=645.17 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 313585 0 0 0 64417 100 0 0 25 0 5 0 517979893 1435398144 313398 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 350439 313398 176 36 0 348344 0
[pid=29572/tid=29579] ppid=29570 vsize=1401756 CPUtime=160.94 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 75662 0 0 0 16071 23 0 0 25 0 5 0 517979944 1435398144 313398 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=1401756 CPUtime=161.12 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 80927 0 0 0 16087 25 0 0 25 0 5 0 517979944 1435398144 313398 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=1401756 CPUtime=161.33 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 70173 0 0 0 16110 23 0 0 25 0 5 0 517979944 1435398144 313398 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=1401756 CPUtime=161.23 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 84423 0 0 0 16096 27 0 0 25 0 5 0 517979944 1435398144 313398 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 645.17
Current children cumulated vsize (KiB) 1401756


################
# More data... #
################

/proc/meminfo: memFree=17034980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=2456848 CPUtime=3035.07 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 584354 0 0 0 303287 220 0 0 25 0 5 0 517979893 2515812352 584167 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 614212 584167 176 36 0 612117 0
[pid=29572/tid=29579] ppid=29570 vsize=2456848 CPUtime=757.64 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 180441 0 0 0 75692 72 0 0 25 0 5 0 517979944 2515812352 584167 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=2456848 CPUtime=758.32 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 141482 0 0 0 75783 49 0 0 25 0 5 0 517979944 2515812352 584167 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=2456848 CPUtime=759.57 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 122479 0 0 0 75909 48 0 0 25 0 5 0 517979944 2515812352 584167 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=2456848 CPUtime=759 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 137552 0 0 0 75850 50 0 0 25 0 5 0 517979944 2515812352 584167 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3035.07
Current children cumulated vsize (KiB) 2456848

[startup+822.301 s]
/proc/loadavg: 8.11 8.09 7.89 9/186 29678
/proc/meminfo: memFree=16606048/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=2570480 CPUtime=3274.28 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 612953 0 0 0 327194 234 0 0 25 0 5 0 517979893 2632171520 612766 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 642620 612766 176 36 0 640525 0
[pid=29572/tid=29579] ppid=29570 vsize=2570480 CPUtime=817.47 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 191827 0 0 0 81670 77 0 0 25 0 5 0 517979944 2632171520 612766 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=2570480 CPUtime=818.02 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 150913 0 0 0 81748 54 0 0 25 0 5 0 517979944 2632171520 612766 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=2570480 CPUtime=819.45 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 122672 0 0 0 81896 49 0 0 25 0 5 0 517979944 2632171520 612766 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=2570480 CPUtime=818.8 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 145141 0 0 0 81827 53 0 0 25 0 5 0 517979944 2632171520 612766 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3274.28
Current children cumulated vsize (KiB) 2570480

[startup+882.301 s]
/proc/loadavg: 8.07 8.08 7.90 9/186 29679
/proc/meminfo: memFree=16287484/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=2637064 CPUtime=3513.21 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 629879 0 0 0 351080 241 0 0 25 0 5 0 517979893 2700353536 629692 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 659266 629692 176 36 0 657171 0
[pid=29572/tid=29579] ppid=29570 vsize=2637064 CPUtime=877.24 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 197756 0 0 0 87644 80 0 0 25 0 5 0 517979944 2700353536 629692 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=2637064 CPUtime=877.82 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 155815 0 0 0 87726 56 0 0 25 0 5 0 517979944 2700353536 629692 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=2637064 CPUtime=878.98 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 126185 0 0 0 87848 50 0 0 25 0 5 0 517979944 2700353536 629692 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=2637064 CPUtime=878.63 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 147723 0 0 0 87810 53 0 0 25 0 5 0 517979944 2700353536 629692 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3513.21
Current children cumulated vsize (KiB) 2637064

[startup+942.301 s]
/proc/loadavg: 8.08 8.08 7.91 9/186 29681
/proc/meminfo: memFree=16016896/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=2716160 CPUtime=3752.18 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 650184 0 0 0 374968 250 0 0 25 0 5 0 517979893 2781347840 649997 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 679040 649997 176 36 0 676945 0
[pid=29572/tid=29579] ppid=29570 vsize=2716160 CPUtime=936.91 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 207811 0 0 0 93607 84 0 0 25 0 5 0 517979944 2781347840 649997 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=2716160 CPUtime=937.69 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 160778 0 0 0 93711 58 0 0 25 0 5 0 517979944 2781347840 649997 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=2716160 CPUtime=938.56 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 127100 0 0 0 93805 51 0 0 25 0 5 0 517979944 2781347840 649997 33554432000 4194304 4338100 140734874892112 18446744073709551615 4242344 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=2716160 CPUtime=938.48 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 152095 0 0 0 93793 55 0 0 25 0 5 0 517979944 2781347840 649997 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3752.18
Current children cumulated vsize (KiB) 2716160

[startup+1002.3 s]
/proc/loadavg: 8.06 8.08 7.91 9/186 29682
/proc/meminfo: memFree=15694980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=2805024 CPUtime=3991.49 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 672247 0 0 0 398889 260 0 0 25 0 5 0 517979893 2872344576 672060 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 701256 672060 176 36 0 699161 0
[pid=29572/tid=29579] ppid=29570 vsize=2805024 CPUtime=996.47 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 214991 0 0 0 99560 87 0 0 25 0 5 0 517979944 2872344576 672060 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=2805024 CPUtime=997.55 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 168572 0 0 0 99693 62 0 0 25 0 5 0 517979944 2872344576 672060 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=2805024 CPUtime=998.54 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 130598 0 0 0 99802 52 0 0 25 0 5 0 517979944 2872344576 672060 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=2805024 CPUtime=998.38 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 155686 0 0 0 99781 57 0 0 25 0 5 0 517979944 2872344576 672060 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3991.49
Current children cumulated vsize (KiB) 2805024

[startup+1062.3 s]
/proc/loadavg: 8.10 8.08 7.92 9/186 29684
/proc/meminfo: memFree=15453272/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=2903144 CPUtime=4230.32 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 697475 0 0 0 422761 271 0 0 25 0 5 0 517979893 2972819456 697288 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 725786 697288 176 36 0 723691 0
[pid=29572/tid=29579] ppid=29570 vsize=2903144 CPUtime=1055.91 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 226730 0 0 0 105498 93 0 0 25 0 5 0 517979944 2972819456 697288 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=2903144 CPUtime=1057.36 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 173695 0 0 0 105673 63 0 0 24 0 5 0 517979944 2972819456 697288 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=2903144 CPUtime=1058.33 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 135392 0 0 0 105779 54 0 0 25 0 5 0 517979944 2972819456 697288 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=2903144 CPUtime=1058.18 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 159258 0 0 0 105759 59 0 0 25 0 5 0 517979944 2972819456 697288 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4230.32
Current children cumulated vsize (KiB) 2903144

[startup+1122.3 s]
/proc/loadavg: 8.03 8.06 7.92 9/186 29686
/proc/meminfo: memFree=15314504/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=2955072 CPUtime=4469.58 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 709883 0 0 0 446680 278 0 0 25 0 5 0 517979893 3025993728 709696 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 738768 709696 176 36 0 736673 0
[pid=29572/tid=29579] ppid=29570 vsize=2955072 CPUtime=1115.43 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 236004 0 0 0 111445 98 0 0 25 0 5 0 517979944 3025993728 709696 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=2955072 CPUtime=1117.27 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 174880 0 0 0 111663 64 0 0 25 0 5 0 517979944 3025993728 709696 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=2955072 CPUtime=1118.24 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 137088 0 0 0 111769 55 0 0 25 0 5 0 517979944 3025993728 709696 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=2955072 CPUtime=1118.1 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 159511 0 0 0 111751 59 0 0 25 0 5 0 517979944 3025993728 709696 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4469.58
Current children cumulated vsize (KiB) 2955072

[startup+1182.3 s]
/proc/loadavg: 8.11 8.08 7.93 9/186 29687
/proc/meminfo: memFree=15020980/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=3015400 CPUtime=4708.6 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 725804 0 0 0 470574 286 0 0 25 0 5 0 517979893 3087769600 725617 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 753850 725618 176 36 0 751755 0
[pid=29572/tid=29579] ppid=29570 vsize=3015400 CPUtime=1174.98 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 250197 0 0 0 117394 104 0 0 25 0 5 0 517979944 3087769600 725618 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=3015400 CPUtime=1177.14 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 176263 0 0 0 117649 65 0 0 25 0 5 0 517979944 3087769600 725619 33554432000 4194304 4338100 140734874892112 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=3015400 CPUtime=1178.13 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 137256 0 0 0 117757 56 0 0 25 0 5 0 517979944 3087769600 725619 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=3015400 CPUtime=1177.82 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 159689 0 0 0 117722 60 0 0 25 0 5 0 517979944 3087769600 725619 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4708.6
Current children cumulated vsize (KiB) 3015400



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.3 s]
/proc/loadavg: 8.07 8.07 7.93 9/186 29688
/proc/meminfo: memFree=14868452/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=3053320 CPUtime=4800.27 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) S 29570 29572 28847 0 -1 4202496 735631 0 0 0 479736 291 0 0 25 0 5 0 517979893 3126599680 735444 33554432000 4194304 4338100 140734874892112 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 763330 735444 176 36 0 761235 0
[pid=29572/tid=29579] ppid=29570 vsize=3053320 CPUtime=1197.91 cores=0,2,4,6
/proc/29572/task/29579/stat : 29579 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 254433 0 0 0 119684 107 0 0 25 0 5 0 517979944 3126599680 735444 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=29572/tid=29580] ppid=29570 vsize=3053320 CPUtime=1200.08 cores=0,2,4,6
/proc/29572/task/29580/stat : 29580 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 181748 0 0 0 119941 67 0 0 25 0 5 0 517979944 3126599680 735444 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=29572/tid=29581] ppid=29570 vsize=3053320 CPUtime=1201.05 cores=0,2,4,6
/proc/29572/task/29581/stat : 29581 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 137314 0 0 0 120049 56 0 0 25 0 5 0 517979944 3126599680 735444 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=3053320 CPUtime=1200.69 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4202560 159736 0 0 0 120009 60 0 0 25 0 5 0 517979944 3126599680 735444 33554432000 4194304 4338100 140734874892112 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4800.27
Current children cumulated vsize (KiB) 3053320

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.41 s]
/proc/loadavg: 8.07 8.07 7.93 9/186 29688
/proc/meminfo: memFree=14868452/32950928 swapFree=67111524/67111528
[pid=29572] ppid=29570 vsize=0 CPUtime=4800.39 cores=0,2,4,6
/proc/29572/stat : 29572 (ttsth-5-4) Z 29570 29572 28847 0 -1 4203548 735631 0 0 0 479738 301 0 0 18 0 2 0 517979893 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/29572/statm: 0 0 0 0 0 0 0
[pid=29572/tid=29582] ppid=29570 vsize=0 CPUtime=1200.8 cores=0,2,4,6
/proc/29572/task/29582/stat : 29582 (ttsth-5-4) R 29570 29572 28847 0 -1 4203588 159736 0 0 0 120010 70 0 0 25 0 2 0 517979944 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.39
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1205.44
CPU time (s): 4800.43
CPU user time (s): 4797.39
CPU system time (s): 3.04554
CPU usage (%): 398.23
Max. virtual memory (cumulated for all children) (KiB): 3053320

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.39
system time used= 3.04554
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 735631
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7464
involuntary context switches= 19343

runsolver used 1.9797 second user time and 4.34334 second system time

The end

Launcher Data 

Begin job on node122 at 2011-04-29 09:56:51
IDJOB=3260198
IDBENCH=83174
IDSOLVER=1654
FILE ID=node122/3260198-1304063811
RUNJOBID= node122-1304063811-29553
PBS_JOBID= 13170447
Free space on /tmp= 71664 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-11x10.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3260198-1304063811/watcher-3260198-1304063811 -o /tmp/evaluation-result-3260198-1304063811/solver-3260198-1304063811 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260198-1304063811.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 5d217b4e2e013436c5df3f8194310a1a
RANDOM SEED=1977731140

node122.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      22195516 kB
Buffers:       1625240 kB
Cached:        8514932 kB
SwapCached:          4 kB
Active:         978920 kB
Inactive:      9226680 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      22195516 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             420 kB
Writeback:           0 kB
AnonPages:       64128 kB
Mapped:          15072 kB
Slab:           485920 kB
PageTables:       4732 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   202088 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71664 MiB
End job on node122 at 2011-04-29 10:16:58