Trace number 3258746

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.27 1203.35

General information on the benchmark

NameCRAFTED/rbsat/random/
forced/rbsat-v2640c305320gyes3.cnf
MD5SUMd6877629cc818d7e31dc245fb95f0588
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2640
Number of clauses305320
Sum of the clauses size613120
Maximum clause length33
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2305240
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 580

Solver Data 

0.19/0.21	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258746-1304052534.cnf 
0.19/0.21	c Using SATTIMEOUT of 4800
0.19/0.21	c Using SATRAM of 15500
0.19/0.21	c Size of subset node = 32
0.19/0.21	c Variables = 2640
0.19/0.21	c Clauses = 305320
0.19/0.21	c Literals = 613120
0.19/0.21	c Reduced size:
0.19/0.21	c Variables = 2640
0.19/0.21	c Clauses = 305320
0.19/0.21	c Literals = 613120
0.19/0.21	c a2
0.29/0.38	c a1
0.79/0.81	c Independent sub-problems = 1
0.79/0.81	c Sub-problem with 2640 variables and 305320 clauses
0.79/0.81	c using permutation method 4
0.79/0.81	c count = 1
0.79/0.81	c Ordering finished 0.770 seconds
0.79/0.81	c mapped 0.800 seconds
0.79/0.81	c permuted 0.800 seconds
0.79/0.81	c about to sort
0.79/0.83	c sorted
1.49/1.59	c clauses now 305320
1.49/1.59	c init finished 0.890 seconds
1.49/1.59	c tree built 1.590 seconds
1.49/1.59	c 
1.49/1.59	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.49/1.59	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.49/1.59	c ................................ | ..........                    |                                  1.6         0.0             
1.49/1.60	c ................................ | ..........                  1 |                                  1.6         0.6             
1.60/1.60	c ................................ | ..........                  2 |                                  1.6         1.2             
1.60/1.60	c ................................ | ..........                  3 |                                  1.6         1.9             
1.60/1.60	c ................................ | ..........                  4 |                                  1.6         2.5             
1.60/1.60	c ................................ | ..........                  5 |                                  1.6         3.1             
1.60/1.60	c ................................ | ..........               1000 |              51                  1.6       625.0           63
95.86/25.24	c ................................ | ..........            3252000 | 58317        226591             96.0     33875.0     14735119
191.72/49.27	c ................................ | ..........            5879000 | 107949       403016            192.0     30615.0     24681534
287.99/73.31	c ................................ | ..........            7968000 | 150631       544577            288.0     27663.8     32790974
383.86/97.33	c ................................ | ..........            9836000 | 189352       668982            384.0     25613.2     40328905
479.72/121.37	c ................................ | *.........           11544000 | 227591       763362            480.0     24049.5     41861975
575.99/145.40	c ................................ | *.........           13258000 | 262406       864272            576.0     23016.6     45114649
671.88/169.44	c ................................ | *.........           14752000 | 298221       943080            672.0     21952.1     48473249
767.73/193.48	c ................................ | *.........           16161000 | 329168       1034013           768.0     21041.6     50254354
863.99/217.50	c ................................ | *.........           17883000 | 363879       1128966           864.0     20697.9     53665764
959.85/241.54	c ................................ | **........           19561000 | 399772       1237562           960.0     20375.4     57879127
1055.72/265.57	c ................................ | **........           21044000 | 431084       1325143          1056.0     19928.0     60816173
1151.99/289.62	c ................................ | **........           22420000 | 462221       1403969          1152.1     19460.8     61538568
1247.85/313.65	c ................................ | **........           23642000 | 491027       1481002          1248.1     18943.0     64051537
1343.74/337.67	c ................................ | **........           24939000 | 519565       1566911          1344.0     18555.8     67128311
1439.97/361.73	c ................................ | ***.......           26175000 | 548067       1639479          1440.1     18175.9     69282615
1535.83/385.75	c ................................ | ***.......           27304000 | 573949       1708534          1536.0     17775.6     72446579
1631.71/409.78	c ................................ | ***.......           28488000 | 598523       1789916          1632.0     17455.6     75737267
1727.96/433.81	c ................................ | ***.......           29790000 | 625999       1863607          1728.0     17239.6     77549350
1823.83/457.84	c ................................ | ***.......           31039000 | 653453       1936290          1824.0     17016.9     78558750
1919.69/481.89	c ................................ | ****......           32333000 | 682130       2002491          1920.0     16839.8     80880663
2015.92/505.92	c ................................ | ****......           33564000 | 709357       2064252          2016.0     16648.8     80894852
2111.76/529.97	c ................................ | ****......           35142000 | 738861       2141886          2112.0     16639.0     82630769
2207.63/553.99	c ................................ | ****......           36576000 | 766733       2214832          2208.0     16565.2     83177197
2303.89/578.03	c ................................ | ****......           37841000 | 797230       2280651          2304.1     16423.7     85162895
2399.76/602.06	c ................................ | *****.....           39161000 | 822831       2347230          2400.0     16317.1     87304503
2496.03/626.10	c ................................ | *****.....           40467000 | 851916       2414833          2496.0     16212.5     90703427
2591.91/650.14	c ................................ | *****.....           41706000 | 877603       2486581          2592.0     16090.0     93685850
2687.73/674.18	c ................................ | *****.....           43033000 | 902137       2553022          2688.1     16008.9     95245865
2784.01/698.21	c ................................ | *****.....           44323000 | 931398       2622762          2784.1     15920.3     95245865
2879.86/722.24	c ................................ | ******....           45463000 | 957443       2692886          2880.0     15785.5     97538035
2975.71/746.28	c ................................ | ******....           46702000 | 982727       2767747          2976.0     15692.7     98485565
3071.97/770.31	c ................................ | ******....           47893000 | 1008868      2834299          3072.0     15590.0    100903406
3167.85/794.34	c ................................ | ******....           49030000 | 1034707      2895663          3168.0     15476.5    102673211
3263.74/818.37	c ................................ | ******....           50091000 | 1059647      2989191          3264.0     15346.5    105903314
3359.98/842.40	c ................................ | *******...           51286000 | 1084116      3082870          3360.0     15263.6    109240838
3455.83/866.44	c ................................ | *******...           52356000 | 1109322      3159037          3456.0     15149.3    112723483
3551.70/890.47	c ................................ | *******...           53419000 | 1132369      3229102          3552.0     15039.1    113833433
3647.99/914.61	c ................................ | *******...           54429000 | 1156174      3306721          3648.0     14920.1    115505346
3743.81/938.77	c ................................ | *******...           55528000 | 1180936      3382896          3744.1     14830.8    118653787
3840.03/962.80	c ................................ | ********..           56600000 | 1200577      3452221          3840.1     14739.4    118781627
3935.86/986.84	c ................................ | ********..           57681000 | 1222986      3522832          3936.0     14654.7    118854693
4031.74/1010.88	c ................................ | ********..           58725000 | 1246637      3585301          4032.1     14564.6    119232236
4128.00/1034.90	c ................................ | ********..           59768000 | 1269319      3649699          4128.0     14478.7    119855344
4223.77/1058.96	c ................................ | ********..           60822000 | 1293444      3709012          4224.0     14399.0    120601511
4319.64/1082.99	c ................................ | *********.           61780000 | 1314676      3765638          4320.0     14300.9    120691365
4415.92/1107.03	c ................................ | *********.           62711000 | 1336481      3829481          4416.0     14200.8    120691365
4511.78/1131.06	c ................................ | *********.           63704000 | 1358636      3894614          4512.1     14118.6    122429464
4608.04/1155.11	c ................................ | *********.           64655000 | 1381023      3948817          4608.1     14030.8    124148849
4703.91/1179.15	c ................................ | *********.           65451000 | 1400782      4003948          4704.1     13913.7    124554823
4799.77/1203.18	c CPU limit (4800) reached
4799.77/1203.18	c Time = 4800.09 seconds
4799.77/1203.18	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258746-1304052534/watcher-3258746-1304052534 -o /tmp/evaluation-result-3258746-1304052534/solver-3258746-1304052534 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258746-1304052534.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.84 7.82 7.91 2/177 2346
/proc/meminfo: memFree=18289220/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) R 2344 2346 1994 0 -1 4202496 403 0 0 0 0 0 0 0 22 0 1 0 516853846 9633792 345 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808862839 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2346/statm: 2352 345 147 36 0 257 0

[startup+0.0265939 s]
/proc/loadavg: 6.84 7.82 7.91 2/177 2346
/proc/meminfo: memFree=18289220/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=13940 CPUtime=0.02 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) R 2344 2346 1994 0 -1 4202496 1512 0 0 0 2 0 0 0 22 0 1 0 516853846 14274560 1454 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2346/statm: 3485 1456 148 36 0 1390 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 13940

[startup+0.100573 s]
/proc/loadavg: 6.84 7.82 7.91 2/177 2346
/proc/meminfo: memFree=18289220/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=28676 CPUtime=0.09 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) R 2344 2346 1994 0 -1 4202496 5178 0 0 0 9 0 0 0 22 0 1 0 516853846 29364224 5120 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2346/statm: 7169 5121 148 36 0 5074 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28676

[startup+0.300532 s]
/proc/loadavg: 6.84 7.82 7.91 2/177 2346
/proc/meminfo: memFree=18289220/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=67996 CPUtime=0.29 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) R 2344 2346 1994 0 -1 4202496 15018 0 0 0 27 2 0 0 23 0 1 0 516853846 69627904 14959 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808560834 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2346/statm: 16999 14959 153 36 0 14904 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 67996

[startup+0.700465 s]
/proc/loadavg: 6.84 7.82 7.91 2/177 2346
/proc/meminfo: memFree=18289220/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=76928 CPUtime=0.69 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) R 2344 2346 1994 0 -1 4202496 18493 0 0 0 67 2 0 0 25 0 1 0 516853846 78774272 17240 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808823971 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2346/statm: 19232 17240 156 36 0 17137 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 76928

[startup+1.50032 s]
/proc/loadavg: 6.84 7.82 7.91 7/182 2351
/proc/meminfo: memFree=18085456/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=182020 CPUtime=1.49 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) R 2344 2346 1994 0 -1 4202496 45359 0 0 0 143 6 0 0 25 0 1 0 516853846 186388480 43509 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808831512 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2346/statm: 45505 43509 164 36 0 43410 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 182020

[startup+3.10099 s]
/proc/loadavg: 6.84 7.82 7.91 9/186 2355
/proc/meminfo: memFree=17925344/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=426396 CPUtime=7.59 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 62686 0 0 0 740 19 0 0 25 0 5 0 516853846 436629504 60238 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 106599 60238 176 36 0 104504 0
[pid=2346/tid=2352] ppid=2344 vsize=426396 CPUtime=1.49 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 4684 0 0 0 148 1 0 0 20 0 5 0 516854006 436629504 60238 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561378 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=426396 CPUtime=1.48 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 2421 0 0 0 144 4 0 0 25 0 5 0 516854006 436629504 60238 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808560656 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=426396 CPUtime=1.49 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 4221 0 0 0 149 0 0 0 20 0 5 0 516854006 436629504 60238 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=426396 CPUtime=1.49 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 2107 0 0 0 144 5 0 0 18 0 5 0 516854006 436629504 60238 33554432000 4194304 4338100 140736189578416 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 7.59
Current children cumulated vsize (KiB) 426396

[startup+6.3004 s]
/proc/loadavg: 6.94 7.82 7.91 10/186 2355
/proc/meminfo: memFree=17732144/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=505124 CPUtime=20.36 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 82805 0 0 0 1994 42 0 0 25 0 5 0 516853846 517246976 80357 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 126281 80357 176 36 0 124186 0
[pid=2346/tid=2352] ppid=2344 vsize=505124 CPUtime=4.69 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 10826 0 0 0 467 2 0 0 25 0 5 0 516854006 517246976 80357 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=505124 CPUtime=4.67 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 6395 0 0 0 452 15 0 0 25 0 5 0 516854006 517246976 80357 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254869 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=505124 CPUtime=4.69 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 10758 0 0 0 467 2 0 0 25 0 5 0 516854006 517246976 80357 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561121 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=505124 CPUtime=4.69 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 5573 0 0 0 454 15 0 0 25 0 5 0 516854006 517246976 80357 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561016 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 20.36
Current children cumulated vsize (KiB) 505124

[startup+12.7002 s]
/proc/loadavg: 7.02 7.82 7.91 9/186 2355
/proc/meminfo: memFree=17428868/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=637748 CPUtime=45.92 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 115539 0 0 0 4495 97 0 0 25 0 5 0 516853846 653053952 113091 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 159437 113091 176 36 0 157342 0
[pid=2346/tid=2352] ppid=2344 vsize=637748 CPUtime=11.09 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 22653 0 0 0 1104 5 0 0 25 0 5 0 516854006 653053952 113091 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=637748 CPUtime=11.03 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 12231 0 0 0 1066 37 0 0 25 0 5 0 516854006 653053952 113091 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561390 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=637748 CPUtime=11.09 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 20967 0 0 0 1103 6 0 0 25 0 5 0 516854006 653053952 113091 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820978470 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=637748 CPUtime=11.09 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 10435 0 0 0 1067 42 0 0 25 0 5 0 516854006 653053952 113091 33554432000 4194304 4338100 140736189578416 18446744073709551615 4285817 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 45.92
Current children cumulated vsize (KiB) 637748

[startup+25.5009 s]
/proc/loadavg: 7.24 7.83 7.91 9/186 2356
/proc/meminfo: memFree=16945648/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=846660 CPUtime=97.05 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 168353 0 0 0 9505 200 0 0 25 0 5 0 516853846 866979840 165905 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 211665 165905 176 36 0 209570 0
[pid=2346/tid=2352] ppid=2344 vsize=846660 CPUtime=23.89 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 43022 0 0 0 2379 10 0 0 25 0 5 0 516854006 866979840 165905 33554432000 4194304 4338100 140736189578416 18446744073709551615 4286311 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=846660 CPUtime=23.78 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 21515 0 0 0 2294 84 0 0 25 0 5 0 516854006 866979840 165905 33554432000 4194304 4338100 140736189578416 18446744073709551615 4256655 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=846660 CPUtime=23.89 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 37036 0 0 0 2380 9 0 0 25 0 5 0 516854006 866979840 165905 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808862227 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=846660 CPUtime=23.86 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 17527 0 0 0 2298 88 0 0 25 0 5 0 516854006 866979840 165905 33554432000 4194304 4338100 140736189578416 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 97.05
Current children cumulated vsize (KiB) 846660

[startup+51.1012 s]
/proc/loadavg: 7.55 7.86 7.92 9/186 2356
/proc/meminfo: memFree=16153144/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=1180012 CPUtime=199.31 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 250703 0 0 0 19557 374 0 0 25 0 5 0 516853846 1208332288 248255 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 295003 248255 176 36 0 292908 0
[pid=2346/tid=2352] ppid=2344 vsize=1180012 CPUtime=49.49 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 74219 0 0 0 4929 20 0 0 25 0 5 0 516854006 1208332288 248255 33554432000 4194304 4338100 140736189578416 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=1180012 CPUtime=49.25 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 37025 0 0 0 4761 164 0 0 25 0 5 0 516854006 1208332288 248255 33554432000 4194304 4338100 140736189578416 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=1180012 CPUtime=49.49 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 58639 0 0 0 4935 14 0 0 25 0 5 0 516854006 1208332288 248255 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808560661 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=1180012 CPUtime=49.46 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 31567 0 0 0 4778 168 0 0 25 0 5 0 516854006 1208332288 248255 33554432000 4194304 4338100 140736189578416 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 199.31
Current children cumulated vsize (KiB) 1180012

[startup+102.307 s]
/proc/loadavg: 7.87 7.89 7.92 10/186 2358
/proc/meminfo: memFree=14939788/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=1692532 CPUtime=403.86 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 378790 0 0 0 39702 684 0 0 25 0 5 0 516853846 1733152768 373494 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 423133 373494 176 36 0 421038 0
[pid=2346/tid=2352] ppid=2344 vsize=1692532 CPUtime=100.7 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 124179 0 0 0 10034 36 0 0 25 0 5 0 516854006 1733152768 373494 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808862261 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=1692532 CPUtime=100.17 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 55829 0 0 0 9715 302 0 0 25 0 5 0 516854006 1733152768 373494 33554432000 4194304 4338100 140736189578416 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=1692532 CPUtime=100.7 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 100536 0 0 0 10046 24 0 0 25 0 5 0 516854006 1733152768 373494 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=1692532 CPUtime=100.66 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 48993 0 0 0 9752 314 0 0 25 0 5 0 516854006 1733152768 373494 33554432000 4194304 4338100 140736189578416 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 403.86
Current children cumulated vsize (KiB) 1692532

[startup+162.301 s]
/proc/loadavg: 8.06 7.94 7.94 9/186 2359
/proc/meminfo: memFree=14228760/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=1924580 CPUtime=643.49 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 436854 0 0 0 63333 1016 0 0 25 0 5 0 516853846 1970769920 431558 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 481145 431558 176 36 0 479050 0
[pid=2346/tid=2352] ppid=2344 vsize=1924580 CPUtime=160.69 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 155238 0 0 0 16016 53 0 0 25 0 5 0 516854006 1970769920 431558 33554432000 4194304 4338100 140736189578416 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=1924580 CPUtime=159.84 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 58746 0 0 0 15532 452 0 0 25 0 5 0 516854006 1970769920 431558 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=1924580 CPUtime=160.69 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 122014 0 0 0 16040 29 0 0 25 0 5 0 516854006 1970769920 431558 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=1924580 CPUtime=160.65 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 51603 0 0 0 15591 474 0 0 25 0 5 0 516854006 1970769920 431558 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 643.49
Current children cumulated vsize (KiB) 1924580

[startup+222.301 s]
/proc/loadavg: 8.15 7.98 7.95 9/186 2361
/proc/meminfo: memFree=13768824/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=2149796 CPUtime=883.16 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 493361 0 0 0 86996 1320 0 0 25 0 5 0 516853846 2201391104 488065 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 537449 488065 176 36 0 535354 0
[pid=2346/tid=2352] ppid=2344 vsize=2149796 CPUtime=220.69 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 189118 0 0 0 22002 67 0 0 25 0 5 0 516854006 2201391104 488065 33554432000 4194304 4338100 140736189578416 18446744073709551615 4285738 0 0 4096 0 0 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 57489 0 0 0 68402 1618 0 0 25 0 5 0 516854006 3656036352 822426 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2800.38
Current children cumulated vsize (KiB) 3570348

[startup+762.307 s]
/proc/loadavg: 8.15 8.09 8.00 9/186 2384
/proc/meminfo: memFree=10453512/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=3720492 CPUtime=3040.04 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 865545 0 0 0 300366 3638 0 0 25 0 5 0 516853846 3809783808 860249 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 930123 860249 176 36 0 928028 0
[pid=2346/tid=2352] ppid=2344 vsize=3720492 CPUtime=759.2 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 425557 0 0 0 75740 180 0 0 25 0 5 0 516854006 3809783808 860249 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561041 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=3720492 CPUtime=758.97 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 62879 0 0 0 74236 1661 0 0 25 0 5 0 516854006 3809783808 860249 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=3720492 CPUtime=760.04 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 270367 0 0 0 75933 71 0 0 25 0 5 0 516854006 3809783808 860249 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=3720492 CPUtime=760.2 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 57489 0 0 0 74303 1717 0 0 25 0 5 0 516854006 3809783808 860249 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3040.04
Current children cumulated vsize (KiB) 3720492

[startup+822.301 s]
/proc/loadavg: 8.12 8.08 8.00 9/186 2386
/proc/meminfo: memFree=9969516/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=3920684 CPUtime=3279.7 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 915332 0 0 0 324078 3892 0 0 25 0 5 0 516853846 4014780416 910036 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 980171 910036 176 36 0 978076 0
[pid=2346/tid=2352] ppid=2344 vsize=3920684 CPUtime=818.87 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 453921 0 0 0 81696 191 0 0 25 0 5 0 516854006 4014780416 910036 33554432000 4194304 4338100 140736189578416 18446744073709551615 4243536 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=3920684 CPUtime=818.97 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 62879 0 0 0 80118 1779 0 0 25 0 5 0 516854006 4014780416 910036 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808560687 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=3920684 CPUtime=820.03 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 291790 0 0 0 81927 76 0 0 25 0 5 0 516854006 4014780416 910036 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808560656 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=3920684 CPUtime=820.19 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 57489 0 0 0 80182 1837 0 0 25 0 5 0 516854006 4014780416 910036 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561041 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3279.7
Current children cumulated vsize (KiB) 3920684

[startup+882.301 s]
/proc/loadavg: 8.10 8.08 8.00 9/186 2387
/proc/meminfo: memFree=9491220/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=4133388 CPUtime=3519.35 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 968156 0 0 0 347744 4191 0 0 25 0 5 0 516853846 4232589312 962860 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 1033347 962860 176 36 0 1031252 0
[pid=2346/tid=2352] ppid=2344 vsize=4133388 CPUtime=878.55 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 478682 0 0 0 87653 202 0 0 25 0 5 0 516854006 4232589312 962860 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=4133388 CPUtime=878.97 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 76195 0 0 0 85975 1922 0 0 25 0 5 0 516854006 4232589312 962860 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561390 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=4133388 CPUtime=880.02 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 306537 0 0 0 87922 80 0 0 25 0 5 0 516854006 4232589312 962860 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808561082 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=4133388 CPUtime=880.19 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 57489 0 0 0 86040 1979 0 0 25 0 5 0 516854006 4232589312 962860 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3519.35
Current children cumulated vsize (KiB) 4133388

[startup+942.301 s]
/proc/loadavg: 8.33 8.14 8.02 9/186 2392
/proc/meminfo: memFree=8964948/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=4308556 CPUtime=3758.19 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 1012538 0 0 0 371361 4458 0 0 25 0 5 0 516853846 4411961344 1007242 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 1077139 1007242 176 36 0 1075044 0
[pid=2346/tid=2352] ppid=2344 vsize=4308556 CPUtime=938.22 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 501096 0 0 0 93609 213 0 0 25 0 5 0 516854006 4411961344 1007242 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=4308556 CPUtime=938.14 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 80668 0 0 0 91766 2048 0 0 25 0 5 0 516854006 4411961344 1007242 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=4308556 CPUtime=940.02 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 320406 0 0 0 93918 84 0 0 25 0 5 0 516854006 4411961344 1007242 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=4308556 CPUtime=940.18 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 61115 0 0 0 91913 2105 0 0 25 0 5 0 516854006 4411961344 1007242 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3758.19
Current children cumulated vsize (KiB) 4308556

[startup+1002.3 s]
/proc/loadavg: 8.23 8.15 8.03 10/186 2395
/proc/meminfo: memFree=8613936/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=4335292 CPUtime=3997.78 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 1015803 0 0 0 395078 4700 0 0 25 0 5 0 516853846 4439339008 1010507 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 1083823 1010507 176 36 0 1081728 0
[pid=2346/tid=2352] ppid=2344 vsize=4335292 CPUtime=998.11 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 503128 0 0 0 99593 218 0 0 25 0 5 0 516854006 4439339008 1010507 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=4335292 CPUtime=997.87 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 80668 0 0 0 97621 2166 0 0 25 0 5 0 516854006 4439339008 1010507 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=4335292 CPUtime=1000.02 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 321639 0 0 0 99917 85 0 0 25 0 5 0 516854006 4439339008 1010507 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=4335292 CPUtime=1000.15 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 61115 0 0 0 97792 2223 0 0 25 0 5 0 516854006 4439339008 1010507 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3997.78
Current children cumulated vsize (KiB) 4335292

[startup+1062.3 s]
/proc/loadavg: 8.19 8.15 8.03 10/186 2396
/proc/meminfo: memFree=8356360/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=4388468 CPUtime=4237.36 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 1029523 0 0 0 418814 4922 0 0 25 0 5 0 516853846 4493791232 1024227 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 1097117 1024227 176 36 0 1095022 0
[pid=2346/tid=2352] ppid=2344 vsize=4388468 CPUtime=1058.09 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 516848 0 0 0 105583 226 0 0 25 0 5 0 516854006 4493791232 1024227 33554432000 4194304 4338100 140736189578416 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=4388468 CPUtime=1057.51 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 80668 0 0 0 103477 2274 0 0 25 0 5 0 516854006 4493791232 1024227 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=4388468 CPUtime=1060.02 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 321639 0 0 0 105917 85 0 0 25 0 5 0 516854006 4493791232 1024227 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=4388468 CPUtime=1060.11 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 61115 0 0 0 103682 2329 0 0 25 0 5 0 516854006 4493791232 1024227 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4237.36
Current children cumulated vsize (KiB) 4388468

[startup+1122.3 s]
/proc/loadavg: 8.12 8.13 8.03 10/186 2398
/proc/meminfo: memFree=8225128/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=4419748 CPUtime=4477.03 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 1037197 0 0 0 442540 5163 0 0 25 0 5 0 516853846 4525821952 1031901 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 1104937 1031901 176 36 0 1102842 0
[pid=2346/tid=2352] ppid=2344 vsize=4419748 CPUtime=1118.09 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 521292 0 0 0 111578 231 0 0 25 0 5 0 516854006 4525821952 1031901 33554432000 4194304 4338100 140736189578416 18446744073709551615 230808560674 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=4419748 CPUtime=1117.18 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 80668 0 0 0 109330 2388 0 0 25 0 5 0 516854006 4525821952 1031901 33554432000 4194304 4338100 140736189578416 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=4419748 CPUtime=1120.03 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 324869 0 0 0 111916 87 0 0 25 0 5 0 516854006 4525821952 1031901 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=4419748 CPUtime=1120.1 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 61115 0 0 0 109561 2449 0 0 25 0 5 0 516854006 4525821952 1031901 33554432000 4194304 4338100 140736189578416 18446744073709551615 4279536 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4477.03
Current children cumulated vsize (KiB) 4419748

[startup+1182.3 s]
/proc/loadavg: 8.09 8.12 8.03 9/186 2399
/proc/meminfo: memFree=8065520/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=4513588 CPUtime=4716.68 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) S 2344 2346 1994 0 -1 4202496 1060351 0 0 0 466280 5388 0 0 25 0 5 0 516853846 4621914112 1055055 33554432000 4194304 4338100 140736189578416 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 1128397 1055055 176 36 0 1126302 0
[pid=2346/tid=2352] ppid=2344 vsize=4513588 CPUtime=1178.09 cores=1,3,5,7
/proc/2346/task/2352/stat : 2352 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 537723 0 0 0 117570 239 0 0 25 0 5 0 516854006 4621914112 1055055 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2346/tid=2353] ppid=2344 vsize=4513588 CPUtime=1176.84 cores=1,3,5,7
/proc/2346/task/2353/stat : 2353 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 80668 0 0 0 115188 2496 0 0 25 0 5 0 516854006 4621914112 1055055 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=4513588 CPUtime=1180.02 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 331592 0 0 0 117914 88 0 0 25 0 5 0 516854006 4621914112 1055055 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2346/tid=2355] ppid=2344 vsize=4513588 CPUtime=1180.1 cores=1,3,5,7
/proc/2346/task/2355/stat : 2355 (ttsth-5-4) R 2344 2346 1994 0 -1 4202560 61115 0 0 0 115454 2556 0 0 25 0 5 0 516854006 4621914112 1055055 33554432000 4194304 4338100 140736189578416 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4716.68
Current children cumulated vsize (KiB) 4513588



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.2 s]
/proc/loadavg: 8.06 8.11 8.03 2/177 2401
/proc/meminfo: memFree=14438560/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=0 CPUtime=4800.12 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) Z 2344 2346 1994 0 -1 4203548 1069886 0 0 0 474538 5474 0 0 18 0 2 0 516853846 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 0 0 0 0 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=0 CPUtime=1200.92 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4203588 333196 0 0 0 120001 91 0 0 25 0 2 0 516854006 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.31 s]
/proc/loadavg: 7.50 7.99 7.99 2/178 2402
/proc/meminfo: memFree=17193096/32951124 swapFree=67111528/67111528
[pid=2346] ppid=2344 vsize=0 CPUtime=4800.22 cores=1,3,5,7
/proc/2346/stat : 2346 (ttsth-5-4) Z 2344 2346 1994 0 -1 4203548 1069886 0 0 0 474538 5484 0 0 18 0 2 0 516853846 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2346/statm: 0 0 0 0 0 0 0
[pid=2346/tid=2354] ppid=2344 vsize=0 CPUtime=1201.03 cores=1,3,5,7
/proc/2346/task/2354/stat : 2354 (ttsth-5-4) R 2344 2346 1994 0 -1 4203588 333196 0 0 0 120001 102 0 0 25 0 2 0 516854006 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.35
CPU time (s): 4800.27
CPU user time (s): 4745.39
CPU system time (s): 54.8867
CPU usage (%): 398.909
Max. virtual memory (cumulated for all children) (KiB): 4551124

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4745.39
system time used= 54.8867
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1069886
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 4084
involuntary context switches= 33173

runsolver used 1.9717 second user time and 4.6083 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 06:48:54
IDJOB=3258746
IDBENCH=71423
IDSOLVER=1654
FILE ID=node104/3258746-1304052534
RUNJOBID= node104-1304052534-2327
PBS_JOBID= 13170409
Free space on /tmp= 71544 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v2640c305320gyes3.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258746-1304052534/watcher-3258746-1304052534 -o /tmp/evaluation-result-3258746-1304052534/solver-3258746-1304052534 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258746-1304052534.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d6877629cc818d7e31dc245fb95f0588
RANDOM SEED=2085286933

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18289872 kB
Buffers:       2195952 kB
Cached:       11802612 kB
SwapCached:          0 kB
Active:        2356036 kB
Inactive:     11719848 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18289872 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            6348 kB
Writeback:           0 kB
AnonPages:       77004 kB
Mapped:          14872 kB
Slab:           523000 kB
PageTables:       4308 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   220656 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71544 MiB
End job on node104 at 2011-04-29 07:08:59