Trace number 3273498

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.13 1213.91

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n180-s33098896-unsat.cnf
MD5SUM1221fb51c407dee7559369d7c6eedb4e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables181
Number of clauses372
Sum of the clauses size1116
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3372
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273498-1304057438.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 181
0.00/0.00	c Clauses = 372
0.00/0.00	c Literals = 1116
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 181
0.00/0.00	c Clauses = 372
0.00/0.00	c Literals = 1116
0.00/0.00	c a2
0.00/0.00	c a1
1.90/1.92	c Independent sub-problems = 1
1.90/1.92	c Sub-problem with 181 variables and 372 clauses
1.90/1.92	c using permutation method 3
1.90/1.92	c Ordering finished 1.920 seconds
1.90/1.92	c mapped 1.920 seconds
1.90/1.92	c permuted 1.920 seconds
1.90/1.92	c about to sort
1.90/1.92	c sorted
1.90/1.92	c clauses now 372
1.90/1.92	c init finished 1.920 seconds
1.90/1.92	c tree built 1.920 seconds
1.90/1.92	c 
1.90/1.92	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.90/1.92	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.90/1.92	c ................................ | ..........                    |                                  1.9         0.0             
1.90/1.92	c ................................ | ..........                  1 |                                  1.9         0.5             
1.90/1.92	c ................................ | ..........                  2 |                                  1.9         1.0             
1.90/1.92	c ................................ | ..........                  3 |                                  1.9         1.6             
1.90/1.92	c ................................ | ..........                  4 |                                  1.9         2.1             
1.90/1.92	c ................................ | ..........                  5 |                                  1.9         2.6             
1.90/1.92	c ................................ | ..........               1000 | 34           121                 1.9       520.8         2428
95.78/25.56	c ................................ | ..........            2921000 | 471907       231561             96.0     30414.4      4645203
191.98/49.70	c ................................ | ..........            5116000 | 855659       366527            192.0     26644.4      8085659
287.65/73.89	c ................................ | ..........            6885000 | 1179415      472875            288.0     23905.4      9985366
383.95/98.11	c ................................ | ..........            8760000 | 1489626      615043            384.0     22812.5     12511814
479.90/122.43	c ................................ | *.........           10387000 | 1776036      714835            480.0     21637.8     13952295
575.95/146.61	c ................................ | *.........           11622000 | 2004434      776442            576.0     20176.4     15967901
671.85/170.74	c ................................ | *.........           12611000 | 2175821      795723            672.0     18765.8     16543511
767.74/194.86	c ................................ | *.........           13682000 | 2383746      812405            768.0     17815.1     18128006
863.68/219.08	c ................................ | *.........           14882000 | 2600812      848073            864.0     17224.1     19467991
959.68/243.29	c ................................ | **........           16098000 | 2836485      879930            960.0     16768.1     20473290
1055.91/267.53	c ................................ | **........           17286000 | 3066276      904295           1056.0     16368.7     21239341
1151.71/291.78	c ................................ | **........           18792000 | 3342052      933215           1152.0     16311.8     22527076
1247.81/316.16	c ................................ | **........           20396000 | 3645028      962790           1248.0     16342.3     23354778
1344.04/340.40	c ................................ | **........           21544000 | 3871572      982669           1344.1     16029.0     23996915
1439.98/364.71	c ................................ | ***.......           23494000 | 4193322      1186520          1440.0     16314.9     24949400
1535.91/389.12	c ................................ | ***.......           25373000 | 4510669      1282471          1536.0     16518.9     25696117
1631.79/413.55	c ................................ | ***.......           27220000 | 4834177      1419701          1632.0     16678.9     27070233
1727.71/437.98	c ................................ | ***.......           28910000 | 5140510      1524293          1728.0     16729.9     28497107
1823.72/462.47	c ................................ | ***.......           30779000 | 5461407      1714118          1824.0     16874.4     30065015
1919.66/486.89	c ................................ | ****......           32270000 | 5712108      1802496          1920.0     16806.9     30561888
2015.92/511.42	c ................................ | ****......           33856000 | 5989802      1937383          2016.0     16793.7     31751036
2111.82/535.75	c ................................ | ****......           34862000 | 6170169      1992378          2112.0     16506.6     32306704
2208.03/560.00	c ................................ | ****......           35762000 | 6330524      2013647          2208.0     16196.3     32666350
2303.83/584.16	c ................................ | ****......           36625000 | 6493863      2031406          2304.1     15895.9     32666350
2399.90/608.43	c ................................ | *****.....           37662000 | 6702608      2048585          2400.1     15692.2     32915059
2495.79/632.67	c ................................ | *****.....           38854000 | 6919234      2091215          2496.1     15565.9     33372621
2591.91/656.93	c ................................ | *****.....           40044000 | 7156672      2124548          2592.0     15449.0     34020977
2687.69/681.07	c ................................ | *****.....           41054000 | 7361130      2146583          2688.0     15273.0     34326884
2783.94/705.21	c ................................ | *****.....           41771000 | 7500574      2160567          2784.0     15004.0     35465708
2879.78/729.56	c ................................ | ******....           42915000 | 7715875      2188813          2880.0     14900.9     36119439
2975.85/753.94	c ................................ | ******....           44207000 | 7961335      2229275          2976.0     14854.3     36834167
3071.71/778.27	c ................................ | ******....           45613000 | 8228580      2276639          3072.0     14847.9     37247278
3167.90/802.53	c ................................ | ******....           46624000 | 8428048      2293219          3168.1     14716.9     37905524
3263.77/826.77	c ................................ | ******....           47473000 | 8606376      2313624          3264.1     14544.2     38567780
3359.76/850.87	c ................................ | *******...           48195000 | 8746234      2320853          3360.1     14343.5     39532103
3455.94/875.22	c ................................ | *******...           48920000 | 8876149      2344465          3456.0     14154.9     40031815
3551.67/899.48	c ................................ | *******...           49728000 | 9033426      2360224          3552.0     13999.9     41065971
3648.34/923.72	c ................................ | *******...           50310000 | 9143256      2371567          3648.4     13789.5     41168674
3743.87/947.77	c ................................ | *******...           50869000 | 9244990      2383757          3744.2     13586.3     41987359
3839.77/971.88	c ................................ | ********..           51404000 | 9348595      2394379          3840.1     13386.0     42113724
3935.80/996.05	c ................................ | ********..           52234000 | 9513295      2403225          3936.0     13270.7     43072495
4031.77/1020.25	c ................................ | ********..           53035000 | 9659725      2461945          4032.0     13153.5     43624799
4127.67/1044.48	c ................................ | ********..           53770000 | 9765334      2643635          4128.0     13025.6     44475644
4224.19/1068.71	c ................................ | ********..           54370000 | 9857718      2725858          4224.2     12871.0     44909059
4319.94/1092.83	c ................................ | *********.           55003000 | 9955597      2816886          4320.1     12732.0     45412711
4415.83/1116.94	c ................................ | *********.           55624000 | 10057395     2896844          4416.0     12596.0     45836546
4511.85/1141.15	c ................................ | *********.           56316000 | 10171541     3012193          4512.0     12481.3     46336978
4607.98/1165.44	c ................................ | *********.           57047000 | 10296715     3128124          4608.1     12379.6     47346517
4703.84/1189.67	c ................................ | *********.           57743000 | 10418687     3216711          4704.2     12274.9     47932353
4799.89/1213.83	c CPU limit (4800) reached
4799.89/1213.83	c Time = 4800.04 seconds
4799.89/1213.83	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3273498-1304057438/watcher-3273498-1304057438 -o /tmp/evaluation-result-3273498-1304057438/solver-3273498-1304057438 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273498-1304057438.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.87 7.85 7.96 2/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=8840 CPUtime=0 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) R 9395 9399 9362 0 -1 4202496 281 0 0 0 0 0 0 0 22 0 1 0 498159939 9052160 223 33554432000 4194304 4338100 140736180472960 18446744073709551615 4245888 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9399/statm: 2210 223 163 36 0 115 0

[startup+0.0561769 s]
/proc/loadavg: 6.87 7.85 7.96 2/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=8840 CPUtime=0.05 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) R 9395 9399 9362 0 -1 4202496 291 0 0 0 5 0 0 0 22 0 1 0 498159939 9052160 233 33554432000 4194304 4338100 140736180472960 18446744073709551615 4290779 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9399/statm: 2210 233 173 36 0 115 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 8840

[startup+0.10018 s]
/proc/loadavg: 6.87 7.85 7.96 2/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=8840 CPUtime=0.09 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) R 9395 9399 9362 0 -1 4202496 291 0 0 0 9 0 0 0 22 0 1 0 498159939 9052160 233 33554432000 4194304 4338100 140736180472960 18446744073709551615 4304638 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9399/statm: 2210 233 173 36 0 115 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8840

[startup+0.300134 s]
/proc/loadavg: 6.87 7.85 7.96 2/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=8840 CPUtime=0.29 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) R 9395 9399 9362 0 -1 4202496 291 0 0 0 29 0 0 0 23 0 1 0 498159939 9052160 233 33554432000 4194304 4338100 140736180472960 18446744073709551615 4245888 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9399/statm: 2210 233 173 36 0 115 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 8840

[startup+0.701092 s]
/proc/loadavg: 6.87 7.85 7.96 2/176 9399
/proc/meminfo: memFree=19559984/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=8840 CPUtime=0.7 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) R 9395 9399 9362 0 -1 4202496 291 0 0 0 70 0 0 0 25 0 1 0 498159939 9052160 233 33554432000 4194304 4338100 140736180472960 18446744073709551615 4290788 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9399/statm: 2210 233 173 36 0 115 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 8840

[startup+1.50096 s]
/proc/loadavg: 6.87 7.85 7.96 4/178 9401
/proc/meminfo: memFree=19559364/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=8840 CPUtime=1.5 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) R 9395 9399 9362 0 -1 4202496 293 0 0 0 150 0 0 0 25 0 1 0 498159939 9052160 235 33554432000 4194304 4338100 140736180472960 18446744073709551615 255106169328 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9399/statm: 2210 235 175 36 0 115 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 8840

[startup+3.10058 s]
/proc/loadavg: 6.48 7.75 7.92 7/182 9405
/proc/meminfo: memFree=19551764/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=280104 CPUtime=6.59 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 6792 0 0 0 657 2 0 0 25 0 5 0 498159939 286826496 6734 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 70026 6734 197 36 0 67931 0
[pid=9399/tid=9402] ppid=9395 vsize=280104 CPUtime=1.16 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 961 0 0 0 116 0 0 0 19 0 5 0 498160132 286826496 6734 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=280104 CPUtime=1.16 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 888 0 0 0 116 0 0 0 25 0 5 0 498160132 286826496 6734 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=280104 CPUtime=1.16 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 1160 0 0 0 116 0 0 0 18 0 5 0 498160132 286826496 6734 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=280104 CPUtime=1.15 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 2962 0 0 0 115 0 0 0 25 0 5 0 498160132 286826496 6734 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 6.59
Current children cumulated vsize (KiB) 280104

[startup+6.30101 s]
/proc/loadavg: 6.48 7.75 7.92 10/186 9410
/proc/meminfo: memFree=19428728/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=309120 CPUtime=19.34 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 13776 0 0 0 1923 11 0 0 25 0 5 0 498159939 316538880 13718 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 77280 13718 198 36 0 75185 0
[pid=9399/tid=9402] ppid=9395 vsize=309120 CPUtime=4.35 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 1731 0 0 0 432 3 0 0 18 0 5 0 498160132 316538880 13718 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=309120 CPUtime=4.36 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 1903 0 0 0 434 2 0 0 25 0 5 0 498160132 316538880 13718 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=309120 CPUtime=4.34 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 2032 0 0 0 431 3 0 0 18 0 5 0 498160132 316538880 13718 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=309120 CPUtime=4.33 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 7289 0 0 0 432 1 0 0 25 0 5 0 498160132 316538880 13718 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254218 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 19.34
Current children cumulated vsize (KiB) 309120

[startup+12.7009 s]
/proc/loadavg: 6.71 7.76 7.92 10/186 9410
/proc/meminfo: memFree=19318652/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=352912 CPUtime=44.83 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 25277 0 0 0 4460 23 0 0 25 0 5 0 498159939 361381888 25219 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 88228 25219 198 36 0 86133 0
[pid=9399/tid=9402] ppid=9395 vsize=352912 CPUtime=10.73 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 2802 0 0 0 1066 7 0 0 19 0 5 0 498160132 361381888 25219 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=352912 CPUtime=10.75 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 3640 0 0 0 1070 5 0 0 25 0 5 0 498160132 361381888 25219 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=352912 CPUtime=10.72 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 4057 0 0 0 1065 7 0 0 18 0 5 0 498160132 361381888 25219 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=352912 CPUtime=10.69 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 13957 0 0 0 1066 3 0 0 25 0 5 0 498160132 361381888 25219 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 44.83
Current children cumulated vsize (KiB) 352912

[startup+25.5006 s]
/proc/loadavg: 6.91 7.77 7.92 10/186 9410
/proc/meminfo: memFree=19161824/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=408208 CPUtime=95.78 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 39400 0 0 0 9527 51 0 0 25 0 5 0 498159939 418004992 39342 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 102052 39342 198 36 0 99957 0
[pid=9399/tid=9402] ppid=9395 vsize=408208 CPUtime=23.46 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 5243 0 0 0 2330 16 0 0 18 0 5 0 498160132 418004992 39342 33554432000 4194304 4338100 140736180472960 18446744073709551615 4206899 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=408208 CPUtime=23.52 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 5660 0 0 0 2339 13 0 0 25 0 5 0 498160132 418004992 39342 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=408208 CPUtime=23.46 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 5982 0 0 0 2330 16 0 0 20 0 5 0 498160132 418004992 39342 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=408208 CPUtime=23.39 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 21694 0 0 0 2334 5 0 0 25 0 5 0 498160132 418004992 39342 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 95.78
Current children cumulated vsize (KiB) 408208

[startup+51.1011 s]
/proc/loadavg: 7.28 7.78 7.92 10/186 9411
/proc/meminfo: memFree=18922120/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=522480 CPUtime=197.57 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 67756 0 0 0 19628 129 0 0 25 0 5 0 498159939 535019520 67698 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 130620 67698 198 36 0 128525 0
[pid=9399/tid=9402] ppid=9395 vsize=522480 CPUtime=48.89 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 7656 0 0 0 4848 41 0 0 25 0 5 0 498160132 535019520 67698 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=522480 CPUtime=48.88 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 7740 0 0 0 4853 35 0 0 25 0 5 0 498160132 535019520 67698 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=522480 CPUtime=48.92 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 8108 0 0 0 4853 39 0 0 25 0 5 0 498160132 535019520 67698 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=522480 CPUtime=48.91 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 43431 0 0 0 4879 12 0 0 25 0 5 0 498160132 535019520 67698 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 197.57
Current children cumulated vsize (KiB) 522480

[startup+102.307 s]
/proc/loadavg: 7.76 7.83 7.93 10/186 9412
/proc/meminfo: memFree=18526304/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=680120 CPUtime=400.55 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 106040 0 0 0 39804 251 0 0 25 0 5 0 498159939 696442880 105982 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 170030 105982 198 36 0 167935 0
[pid=9399/tid=9402] ppid=9395 vsize=680120 CPUtime=99.54 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 12857 0 0 0 9872 82 0 0 25 0 5 0 498160132 696442880 105982 33554432000 4194304 4338100 140736180472960 18446744073709551615 4240533 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=680120 CPUtime=99.55 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 10273 0 0 0 9886 69 0 0 24 0 5 0 498160132 696442880 105982 33554432000 4194304 4338100 140736180472960 18446744073709551615 255106164455 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=680120 CPUtime=99.67 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 12206 0 0 0 9889 78 0 0 18 0 5 0 498160132 696442880 105982 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=680120 CPUtime=99.83 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 69883 0 0 0 9962 21 0 0 25 0 5 0 498160132 696442880 105982 33554432000 4194304 4338100 140736180472960 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 400.55
Current children cumulated vsize (KiB) 680120

[startup+162.301 s]
/proc/loadavg: 8.08 7.91 7.95 10/186 9414
/proc/meminfo: memFree=18243952/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=799968 CPUtime=638.43 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 136060 0 0 0 63420 423 0 0 25 0 5 0 498159939 819167232 136002 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 199992 136002 198 36 0 197897 0
[pid=9399/tid=9402] ppid=9395 vsize=799968 CPUtime=158.72 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 16165 0 0 0 15735 137 0 0 25 0 5 0 498160132 819167232 136002 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=799968 CPUtime=158.9 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 13736 0 0 0 15765 125 0 0 25 0 5 0 498160132 819167232 136002 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=799968 CPUtime=159.14 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 13657 0 0 0 15782 132 0 0 25 0 5 0 498160132 819167232 136002 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=799968 CPUtime=159.71 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 91681 0 0 0 15944 27 0 0 25 0 5 0 498160132 819167232 136002 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 638.43
Current children cumulated vsize (KiB) 799968

[startup+222.301 s]
/proc/loadavg: 8.11 7.95 7.96 7/186 9415
/proc/meminfo: memFree=18050996/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=915552 CPUtime=876.77 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 163937 0 0 0 87113 564 0 0 25 0 5 0 498159939 937525248 163879 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 228888 163879 198 36 0 226793 0
[pid=9399/tid=9402] ppid=9395 vsize=915552 CPUtime=218.12 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 17242 0 0 0 21631 181 0 0 25 0 5 0 498160132 937525248 163879 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 240581 0 0 0 69477 137 0 0 25 0 5 0 498160132 1547886592 304866 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2772.51
Current children cumulated vsize (KiB) 1511608

[startup+762.307 s]
/proc/loadavg: 8.16 8.07 8.01 10/186 9430
/proc/meminfo: memFree=16664324/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1565040 CPUtime=3009.09 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 321537 0 0 0 298900 2009 0 0 25 0 5 0 498159939 1602600960 321479 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 391260 321479 198 36 0 389165 0
[pid=9399/tid=9402] ppid=9395 vsize=1565040 CPUtime=748.73 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 19389 0 0 0 74223 650 0 0 25 0 5 0 498160132 1602600960 321479 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1565040 CPUtime=751.6 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 21460 0 0 0 74567 593 0 0 25 0 5 0 498160132 1602600960 321479 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1565040 CPUtime=750.99 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 26112 0 0 0 74480 619 0 0 19 0 5 0 498160132 1602600960 321479 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1565040 CPUtime=755.81 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 253755 0 0 0 75436 145 0 0 25 0 5 0 498160132 1602600960 321479 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3009.09
Current children cumulated vsize (KiB) 1565040

[startup+822.3 s]
/proc/loadavg: 8.16 8.09 8.01 10/186 9431
/proc/meminfo: memFree=16531880/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1649792 CPUtime=3246.32 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 336063 0 0 0 322439 2193 0 0 25 0 5 0 498159939 1689387008 336005 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 412448 336005 198 36 0 410353 0
[pid=9399/tid=9402] ppid=9395 vsize=1649792 CPUtime=807.83 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 20229 0 0 0 80078 705 0 0 18 0 5 0 498160132 1689387008 336005 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1649792 CPUtime=811.08 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 23441 0 0 0 80456 652 0 0 19 0 5 0 498160132 1689387008 336005 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1649792 CPUtime=810.06 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 27046 0 0 0 80331 675 0 0 25 0 5 0 498160132 1689387008 336005 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1649792 CPUtime=815.38 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 264526 0 0 0 81379 159 0 0 25 0 5 0 498160132 1689387008 336005 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3246.32
Current children cumulated vsize (KiB) 1649792

[startup+882.301 s]
/proc/loadavg: 8.34 8.14 8.03 10/186 9436
/proc/meminfo: memFree=16411848/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1712480 CPUtime=3484.14 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 352074 0 0 0 346086 2328 0 0 25 0 5 0 498159939 1753579520 352016 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 428120 352016 198 36 0 426025 0
[pid=9399/tid=9402] ppid=9395 vsize=1712480 CPUtime=867.04 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 20240 0 0 0 85956 748 0 0 25 0 5 0 498160132 1753579520 352016 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1712480 CPUtime=870.41 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 23476 0 0 0 86347 694 0 0 25 0 5 0 498160132 1753579520 352016 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1712480 CPUtime=869.58 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 27222 0 0 0 86241 717 0 0 25 0 5 0 498160132 1753579520 352016 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1712480 CPUtime=875.15 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 280315 0 0 0 87347 168 0 0 25 0 5 0 498160132 1753579520 352016 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3484.14
Current children cumulated vsize (KiB) 1712480

[startup+942.3 s]
/proc/loadavg: 8.12 8.12 8.03 10/186 9437
/proc/meminfo: memFree=16305516/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1779880 CPUtime=3722.47 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 366732 0 0 0 369791 2456 0 0 25 0 5 0 498159939 1822597120 366674 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 444970 366674 198 36 0 442875 0
[pid=9399/tid=9402] ppid=9395 vsize=1779880 CPUtime=926.57 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 21327 0 0 0 91868 789 0 0 25 0 5 0 498160132 1822597120 366674 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1779880 CPUtime=929.85 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 24221 0 0 0 92249 736 0 0 25 0 5 0 498160132 1822597120 366674 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1779880 CPUtime=929.18 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 27875 0 0 0 92161 757 0 0 25 0 5 0 498160132 1822597120 366674 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1779880 CPUtime=934.92 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 292488 0 0 0 93319 173 0 0 25 0 5 0 498160132 1822597120 366674 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3722.47
Current children cumulated vsize (KiB) 1779880

[startup+1002.3 s]
/proc/loadavg: 8.04 8.09 8.02 10/186 9439
/proc/meminfo: memFree=16209164/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1826800 CPUtime=3960.76 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 381234 0 0 0 393511 2565 0 0 25 0 5 0 498159939 1870643200 381176 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 456700 381176 198 36 0 454605 0
[pid=9399/tid=9402] ppid=9395 vsize=1826800 CPUtime=986.06 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 22094 0 0 0 97782 824 0 0 25 0 5 0 498160132 1870643200 381176 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1826800 CPUtime=989.49 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 25682 0 0 0 98178 771 0 0 20 0 5 0 498160132 1870643200 381176 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1826800 CPUtime=988.77 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 28537 0 0 0 98087 790 0 0 18 0 5 0 498160132 1870643200 381176 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1826800 CPUtime=994.47 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 304100 0 0 0 99270 177 0 0 25 0 5 0 498160132 1870643200 381176 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3960.76
Current children cumulated vsize (KiB) 1826800

[startup+1062.3 s]
/proc/loadavg: 8.01 8.07 8.02 10/186 9441
/proc/meminfo: memFree=16149748/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1871232 CPUtime=4198.75 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 393978 0 0 0 417207 2668 0 0 25 0 5 0 498159939 1916141568 393920 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 467808 393920 198 36 0 465713 0
[pid=9399/tid=9402] ppid=9395 vsize=1871232 CPUtime=1045.29 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 22343 0 0 0 103672 857 0 0 25 0 5 0 498160132 1916141568 393920 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1871232 CPUtime=1049.11 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 26072 0 0 0 104108 803 0 0 19 0 5 0 498160132 1916141568 393920 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1871232 CPUtime=1048.28 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 29288 0 0 0 104005 823 0 0 25 0 5 0 498160132 1916141568 393920 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1871232 CPUtime=1054.12 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 315454 0 0 0 105229 183 0 0 25 0 5 0 498160132 1916141568 393920 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4198.75
Current children cumulated vsize (KiB) 1871232

[startup+1122.3 s]
/proc/loadavg: 8.09 8.09 8.02 10/186 9442
/proc/meminfo: memFree=16109072/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1905896 CPUtime=4437.4 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 403954 0 0 0 440961 2779 0 0 25 0 5 0 498159939 1951637504 403896 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 476474 403896 198 36 0 474379 0
[pid=9399/tid=9402] ppid=9395 vsize=1905896 CPUtime=1104.66 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 22594 0 0 0 109570 896 0 0 25 0 5 0 498160132 1951637504 403896 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1905896 CPUtime=1108.82 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 26395 0 0 0 110044 838 0 0 25 0 5 0 498160132 1951637504 403896 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1905896 CPUtime=1107.93 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 29947 0 0 0 109937 856 0 0 25 0 5 0 498160132 1951637504 403896 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1905896 CPUtime=1114.03 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 324197 0 0 0 111216 187 0 0 25 0 5 0 498160132 1951637504 403896 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4437.4
Current children cumulated vsize (KiB) 1905896

[startup+1182.3 s]
/proc/loadavg: 7.95 8.04 8.00 10/186 9444
/proc/meminfo: memFree=15984552/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=1968840 CPUtime=4674.9 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) S 9395 9399 9362 0 -1 4202496 422093 0 0 0 464574 2916 0 0 25 0 5 0 498159939 2016092160 422035 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 492210 422035 198 36 0 490115 0
[pid=9399/tid=9402] ppid=9395 vsize=1968840 CPUtime=1163.58 cores=0,2,4,6
/proc/9399/task/9402/stat : 9402 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 23251 0 0 0 115418 940 0 0 18 0 5 0 498160132 2016092160 422035 33554432000 4194304 4338100 140736180472960 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=1968840 CPUtime=1168.21 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 27648 0 0 0 115939 882 0 0 20 0 5 0 498160132 2016092160 422035 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9399/tid=9404] ppid=9395 vsize=1968840 CPUtime=1167.34 cores=0,2,4,6
/proc/9399/task/9404/stat : 9404 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 30685 0 0 0 115836 898 0 0 25 0 5 0 498160132 2016092160 422035 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9399/tid=9405] ppid=9395 vsize=1968840 CPUtime=1173.82 cores=0,2,4,6
/proc/9399/task/9405/stat : 9405 (ttsth-5-4) R 9395 9399 9362 0 -1 4202560 339688 0 0 0 117188 194 0 0 25 0 5 0 498160132 2016092160 422035 33554432000 4194304 4338100 140736180472960 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4674.9
Current children cumulated vsize (KiB) 1968840



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1213.9 s]
/proc/loadavg: 7.86 8.02 8.00 6/183 9477
/proc/meminfo: memFree=19124332/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) Z 9395 9399 9362 0 -1 4203548 429961 0 0 0 477025 2986 0 0 18 0 2 0 498159939 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 0 0 0 0 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=0 CPUtime=1199.62 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4203588 28161 0 0 0 119054 908 0 0 25 0 2 0 498160132 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1213.9 s]
/proc/loadavg: 7.86 8.02 8.00 6/183 9477
/proc/meminfo: memFree=19124332/32951124 swapFree=67111524/67111528
[pid=9399] ppid=9395 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/9399/stat : 9399 (ttsth-5-4) Z 9395 9399 9362 0 -1 4203548 429961 0 0 0 477025 2986 0 0 18 0 2 0 498159939 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9399/statm: 0 0 0 0 0 0 0
[pid=9399/tid=9403] ppid=9395 vsize=0 CPUtime=1199.62 cores=0,2,4,6
/proc/9399/task/9403/stat : 9403 (ttsth-5-4) R 9395 9399 9362 0 -1 4203588 28161 0 0 0 119054 908 0 0 25 0 2 0 498160132 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1213.91
CPU time (s): 4800.13
CPU user time (s): 4770.25
CPU system time (s): 29.8725
CPU usage (%): 395.426
Max. virtual memory (cumulated for all children) (KiB): 1997120

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4770.25
system time used= 29.8725
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 429961
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 108512
involuntary context switches= 46491

runsolver used 1.90171 second user time and 4.71228 second system time

The end

Launcher Data 

Begin job on node144 at 2011-04-29 08:10:38
IDJOB=3273498
IDBENCH=83140
IDSOLVER=1654
FILE ID=node144/3273498-1304057438
RUNJOBID= node144-1304057438-9380
PBS_JOBID= 13170439
Free space on /tmp= 72412 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n180-s33098896-unsat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3273498-1304057438/watcher-3273498-1304057438 -o /tmp/evaluation-result-3273498-1304057438/solver-3273498-1304057438 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273498-1304057438.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1221fb51c407dee7559369d7c6eedb4e
RANDOM SEED=1010862323

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19560512 kB
Buffers:       2206876 kB
Cached:       10554308 kB
SwapCached:          4 kB
Active:        4678656 kB
Inactive:      8146248 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19560512 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             148 kB
Writeback:           0 kB
AnonPages:       62500 kB
Mapped:          14320 kB
Slab:           499660 kB
PageTables:       5028 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   182712 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72408 MiB
End job on node144 at 2011-04-29 08:30:54