Trace number 3263454

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.36 1203.33

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_22_70_7-dir.shuffled-as.sat03-154.cnf
MD5SUMaa9ee2fa26214e80fac6d004d5d0b003
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables660
Number of clauses56654
Sum of the clauses size113924
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 256632
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 522

Solver Data 

0.00/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263454-1304096555.cnf 
0.00/0.03	c Using SATTIMEOUT of 4800
0.00/0.03	c Using SATRAM of 15500
0.00/0.03	c Size of subset node = 32
0.00/0.03	c Variables = 660
0.00/0.03	c Clauses = 56654
0.00/0.03	c Literals = 113924
0.00/0.03	c Reduced size:
0.00/0.03	c Variables = 660
0.00/0.03	c Clauses = 56654
0.00/0.03	c Literals = 113924
0.00/0.03	c a2
0.00/0.07	c a1
0.09/0.15	c Independent sub-problems = 1
0.09/0.15	c Sub-problem with 660 variables and 56654 clauses
0.09/0.15	c using permutation method 4
0.09/0.15	c count = 1
0.09/0.15	c Ordering finished 0.140 seconds
0.09/0.15	c mapped 0.140 seconds
0.09/0.15	c permuted 0.140 seconds
0.09/0.15	c about to sort
0.09/0.15	c sorted
0.19/0.29	c clauses now 56654
0.19/0.29	c init finished 0.160 seconds
0.19/0.29	c tree built 0.280 seconds
0.19/0.29	c 
0.19/0.29	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.29	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.29	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.29	c ................................ | ..........                  1 |                                  0.3         3.6             
0.19/0.29	c ................................ | ..........                  2 |                                  0.3         7.1             
0.19/0.29	c ................................ | ..........                  3 |                                  0.3        10.7             
0.19/0.29	c ................................ | ..........                  4 |                                  0.3        14.3             
0.19/0.29	c ................................ | ..........                  5 |                                  0.3        17.9             
0.19/0.29	c ................................ | ..........               1000 | 1            41                  0.3      3448.3          888
95.92/24.32	c ................................ | ..........            4420000 | 109340       566967             96.0     46041.7     11201103
191.61/48.40	c ................................ | ..........            7418000 | 181678       898433            192.0     38633.4     19040017
288.05/72.51	c ................................ | ..........           10042000 | 240545       1312150           288.0     34865.6     24216811
383.68/96.60	c ................................ | ..........           12316000 | 287257       1596166           384.1     32068.7     29877342
480.19/120.74	c ................................ | *.........           14348000 | 340715       1865298           480.1     29888.6     34287891
575.92/144.75	c ................................ | *.........           15987000 | 384316       2125938           576.0     27753.8     37632225
672.03/168.85	c ................................ | *.........           17404000 | 427823       2413348           672.0     25898.8     41966376
768.19/192.97	c ................................ | *.........           18637000 | 478764       2612991           768.1     24263.1     45801017
863.97/216.96	c ................................ | *.........           19692000 | 517568       2772745           864.1     22789.8     48579008
960.09/241.07	c ................................ | **........           20980000 | 562893       2962249           960.0     21853.3     50627398
1056.28/265.14	c ................................ | **........           22050000 | 604351       3094496          1056.3     20874.6     51404921
1151.99/289.15	c ................................ | **........           22978000 | 648607       3221523          1152.1     19944.4     53293239
1247.74/313.19	c ................................ | **........           24201000 | 691029       3379990          1248.1     19390.9     55324397
1343.91/337.28	c ................................ | **........           25367000 | 733187       3544192          1344.0     18873.7     58006835
1440.05/361.32	c ................................ | ***.......           26688000 | 770384       3684346          1440.0     18533.3     59372696
1535.82/385.35	c ................................ | ***.......           27730000 | 800026       3850684          1536.0     18052.8     59741147
1632.39/409.50	c ................................ | ***.......           28774000 | 836431       3984378          1632.2     17628.9     61433722
1727.79/433.46	c ................................ | ***.......           29726000 | 870093       4101399          1728.0     17202.5     63929979
1823.93/457.57	c ................................ | ***.......           31011000 | 901766       4234698          1824.0     17001.5     65223680
1919.64/481.59	c ................................ | ****......           32607000 | 941184       4416427          1920.0     16982.6     68118746
2016.15/505.71	c ................................ | ****......           34342000 | 981303       4579548          2016.0     17034.3     70994575
2111.92/529.72	c ................................ | ****......           35576000 | 1010797      4718362          2112.0     16844.7     72749472
2208.07/553.81	c ................................ | ****......           36476000 | 1029956      4834460          2208.1     16519.3     74336205
2303.80/577.88	c ................................ | ****......           37673000 | 1066598      4967319          2304.1     16350.1     76363096
2399.98/601.98	c ................................ | *****.....           38632000 | 1094747      5101389          2400.0     16096.5     76732511
2495.77/625.99	c ................................ | *****.....           39540000 | 1120305      5226400          2496.1     15840.7     77959417
2591.97/650.07	c ................................ | *****.....           40369000 | 1150742      5384816          2592.1     15573.6     80224120
2687.73/674.09	c ................................ | *****.....           41244000 | 1181807      5498685          2688.1     15343.4     82518515
2783.93/698.14	c ................................ | *****.....           41981000 | 1209684      5567347          2784.0     15079.2     83769695
2880.10/722.23	c ................................ | ******....           42842000 | 1242842      5657068          2880.1     14875.4     84528307
2976.24/746.32	c ................................ | ******....           43861000 | 1280497      5742114          2976.0     14738.2     85984178
3072.04/770.38	c ................................ | ******....           44805000 | 1312613      5848066          3072.0     14584.8     86828221
3167.82/794.36	c ................................ | ******....           45765000 | 1339488      5922743          3168.1     14445.7     87765277
3264.40/818.52	c ................................ | ******....           46593000 | 1366973      6024547          3264.1     14274.3     89907778
3360.12/842.51	c ................................ | *******...           48190000 | 1406831      6143430          3360.0     14342.1     91519907
3456.26/866.60	c ................................ | *******...           49503000 | 1444056      6270619          3456.1     14323.2     93409725
3552.03/890.60	c ................................ | *******...           50645000 | 1482855      6367800          3552.0     14258.1     95445709
3647.75/914.68	c ................................ | *******...           51939000 | 1519995      6466960          3648.1     14237.4     96530664
3743.97/938.79	c ................................ | *******...           52697000 | 1536290      6534334          3744.0     14074.9     96835067
3840.09/962.83	c ................................ | ********..           53552000 | 1558609      6620133          3840.2     13945.0     97195239
3935.96/986.81	c ................................ | ********..           54196000 | 1577455      6691536          3936.0     13769.3     97320770
4031.81/1010.87	c ................................ | ********..           55117000 | 1602977      6763397          4032.1     13669.7     98275903
4127.64/1034.89	c ................................ | ********..           56231000 | 1627116      6860872          4128.0     13621.8     99167364
4223.89/1058.94	c ................................ | ********..           57231000 | 1658266      6947755          4224.1     13548.8    100380361
4319.78/1082.96	c ................................ | *********.           58080000 | 1687960      7050882          4320.0     13444.4    102237664
4416.04/1107.06	c ................................ | *********.           59060000 | 1723096      7140375          4416.3     13373.2    103283809
4511.84/1131.03	c ................................ | *********.           59953000 | 1752316      7203543          4512.0     13287.5    103792651
4607.75/1155.08	c ................................ | *********.           61182000 | 1786239      7328817          4608.0     13277.2    103792651
4704.02/1179.14	c ................................ | *********.           62187000 | 1816180      7430719          4704.2     13219.6    104289349
4799.90/1203.17	c CPU limit (4800) reached
4799.90/1203.17	c Time = 4800.19 seconds
4799.90/1203.17	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263454-1304096555/watcher-3263454-1304096555 -o /tmp/evaluation-result-3263454-1304096555/solver-3263454-1304096555 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263454-1304096555.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.05 7.83 7.83 5/181 16562
/proc/meminfo: memFree=28735984/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=9756 CPUtime=0 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) R 16560 16562 16318 0 -1 4202496 500 0 0 0 0 0 0 0 25 0 1 0 521254235 9990144 443 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/16562/statm: 2439 444 147 36 0 344 0

[startup+0.096761 s]
/proc/loadavg: 8.05 7.83 7.83 5/181 16562
/proc/meminfo: memFree=28735984/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=18568 CPUtime=0.09 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) R 16560 16562 16318 0 -1 4202496 3746 0 0 0 9 0 0 0 25 0 1 0 521254235 19013632 2619 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/16562/statm: 4642 2620 155 36 0 2547 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18568

[startup+0.100757 s]
/proc/loadavg: 8.05 7.83 7.83 5/181 16562
/proc/meminfo: memFree=28735984/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=19096 CPUtime=0.09 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) R 16560 16562 16318 0 -1 4202496 3870 0 0 0 9 0 0 0 25 0 1 0 521254235 19554304 2742 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/16562/statm: 4774 2742 155 36 0 2679 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19096

[startup+0.300727 s]
/proc/loadavg: 8.05 7.83 7.83 5/181 16562
/proc/meminfo: memFree=28735984/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=230636 CPUtime=0.29 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 11092 0 0 0 29 0 0 0 25 0 5 0 521254235 236171264 9740 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 57659 9741 176 36 0 55564 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 230636

[startup+0.70066 s]
/proc/loadavg: 8.05 7.83 7.83 5/181 16562
/proc/meminfo: memFree=28735984/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=234556 CPUtime=1.9 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 13901 0 0 0 188 2 0 0 25 0 5 0 521254235 240185344 12549 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 58639 12549 176 36 0 56544 0
Current children cumulated CPU time (s) 1.9
Current children cumulated vsize (KiB) 234556

[startup+1.50053 s]
/proc/loadavg: 7.80 7.78 7.81 9/186 16567
/proc/meminfo: memFree=28672960/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=253852 CPUtime=5.08 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 18672 0 0 0 504 4 0 0 25 0 5 0 521254235 259944448 17320 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 63463 17320 176 36 0 61368 0
[pid=16562/tid=16564] ppid=16560 vsize=253852 CPUtime=1.19 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 1041 0 0 0 118 1 0 0 19 0 5 0 521254265 259944448 17320 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813636611 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=253852 CPUtime=1.19 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 2343 0 0 0 119 0 0 0 25 0 5 0 521254265 259944448 17320 33554432000 4194304 4338100 140734689946256 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=253852 CPUtime=1.18 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 1986 0 0 0 118 0 0 0 18 0 5 0 521254265 259944448 17320 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813635137 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=253852 CPUtime=1.18 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 2467 0 0 0 118 0 0 0 18 0 5 0 521254265 259944448 17321 33554432000 4194304 4338100 140734689946256 18446744073709551615 4239191 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 5.08
Current children cumulated vsize (KiB) 253852

[startup+3.10026 s]
/proc/loadavg: 7.80 7.78 7.81 9/186 16567
/proc/meminfo: memFree=28642208/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=289052 CPUtime=11.47 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 27827 0 0 0 1139 8 0 0 25 0 5 0 521254235 295989248 26475 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 72263 26475 176 36 0 70168 0
[pid=16562/tid=16564] ppid=16560 vsize=289052 CPUtime=2.8 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 2384 0 0 0 277 3 0 0 24 0 5 0 521254265 295989248 26475 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=289052 CPUtime=2.79 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 5437 0 0 0 278 1 0 0 25 0 5 0 521254265 295989248 26475 33554432000 4194304 4338100 140734689946256 18446744073709551615 4256989 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=289052 CPUtime=2.77 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 4282 0 0 0 276 1 0 0 18 0 5 0 521254265 295989248 26475 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=289052 CPUtime=2.78 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 4889 0 0 0 277 1 0 0 21 0 5 0 521254265 295989248 26475 33554432000 4194304 4338100 140734689946256 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.47
Current children cumulated vsize (KiB) 289052

[startup+6.30072 s]
/proc/loadavg: 7.82 7.79 7.81 9/186 16567
/proc/meminfo: memFree=28557392/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=355796 CPUtime=24.2 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 43613 0 0 0 2403 17 0 0 25 0 5 0 521254235 364335104 42261 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 88949 42261 176 36 0 86854 0
[pid=16562/tid=16564] ppid=16560 vsize=355796 CPUtime=5.99 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 4753 0 0 0 592 7 0 0 25 0 5 0 521254265 364335104 42261 33554432000 4194304 4338100 140734689946256 18446744073709551615 4256352 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=355796 CPUtime=5.99 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 10058 0 0 0 597 2 0 0 25 0 5 0 521254265 364335104 42261 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=355796 CPUtime=5.92 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 7504 0 0 0 590 2 0 0 18 0 5 0 521254265 364335104 42261 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=355796 CPUtime=5.96 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 10463 0 0 0 594 2 0 0 25 0 5 0 521254265 364335104 42261 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.2
Current children cumulated vsize (KiB) 355796

[startup+12.7006 s]
/proc/loadavg: 7.83 7.79 7.82 9/186 16567
/proc/meminfo: memFree=28433512/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=445176 CPUtime=49.7 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 66700 0 0 0 4939 31 0 0 25 0 5 0 521254235 455860224 65348 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 111294 65348 176 36 0 109199 0
[pid=16562/tid=16564] ppid=16560 vsize=445176 CPUtime=12.39 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 8855 0 0 0 1223 16 0 0 25 0 5 0 521254265 455860224 65348 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=445176 CPUtime=12.4 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 16947 0 0 0 1236 4 0 0 25 0 5 0 521254265 455860224 65348 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=445176 CPUtime=12.27 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 12493 0 0 0 1222 5 0 0 18 0 5 0 521254265 455860224 65348 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=445176 CPUtime=12.33 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 17570 0 0 0 1229 4 0 0 25 0 5 0 521254265 455860224 65348 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813936625 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.7
Current children cumulated vsize (KiB) 445176

[startup+25.5005 s]
/proc/loadavg: 7.95 7.82 7.82 9/186 16567
/proc/meminfo: memFree=28207436/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=594504 CPUtime=100.7 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 105669 0 0 0 10009 61 0 0 25 0 5 0 521254235 608772096 104317 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 148626 104317 176 36 0 146531 0
[pid=16562/tid=16564] ppid=16560 vsize=594504 CPUtime=25.18 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 17093 0 0 0 2483 35 0 0 25 0 5 0 521254265 608772096 104317 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=594504 CPUtime=25.19 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 27973 0 0 0 2513 6 0 0 25 0 5 0 521254265 608772096 104317 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813635473 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=594504 CPUtime=24.94 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 22140 0 0 0 2483 11 0 0 18 0 5 0 521254265 608772096 104317 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=594504 CPUtime=25.06 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 27628 0 0 0 2500 6 0 0 25 0 5 0 521254265 608772096 104317 33554432000 4194304 4338100 140734689946256 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.7
Current children cumulated vsize (KiB) 594504

[startup+51.1012 s]
/proc/loadavg: 8.03 7.85 7.83 9/186 16568
/proc/meminfo: memFree=27793736/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=861848 CPUtime=202.75 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 172863 0 0 0 20167 108 0 0 25 0 5 0 521254235 882532352 171511 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 215462 171511 176 36 0 213367 0
[pid=16562/tid=16564] ppid=16560 vsize=861848 CPUtime=50.77 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 28166 0 0 0 5014 63 0 0 25 0 5 0 521254265 882532352 171511 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813635952 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=861848 CPUtime=50.79 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 49360 0 0 0 5066 13 0 0 25 0 5 0 521254265 882532352 171511 33554432000 4194304 4338100 140734689946256 18446744073709551615 4256745 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=861848 CPUtime=50.35 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 34194 0 0 0 5017 18 0 0 18 0 5 0 521254265 882532352 171511 33554432000 4194304 4338100 140734689946256 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=861848 CPUtime=50.51 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 50308 0 0 0 5039 12 0 0 25 0 5 0 521254265 882532352 171511 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 202.75
Current children cumulated vsize (KiB) 861848

[startup+102.307 s]
/proc/loadavg: 8.07 7.88 7.84 9/186 16571
/proc/meminfo: memFree=27221840/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=1206188 CPUtime=406.83 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 260653 0 0 0 40501 182 0 0 25 0 5 0 521254235 1235136512 259301 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 301547 259301 176 36 0 299452 0
[pid=16562/tid=16564] ppid=16560 vsize=1206188 CPUtime=101.96 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 42138 0 0 0 10084 112 0 0 25 0 5 0 521254265 1235136512 259301 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=1206188 CPUtime=102 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 78816 0 0 0 10180 20 0 0 25 0 5 0 521254265 1235136512 259301 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813635098 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=1206188 CPUtime=101.15 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 49427 0 0 0 10085 30 0 0 18 0 5 0 521254265 1235136512 259301 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=1206188 CPUtime=101.41 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 79437 0 0 0 10123 18 0 0 25 0 5 0 521254265 1235136512 259301 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813635497 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 406.83
Current children cumulated vsize (KiB) 1206188

[startup+162.301 s]
/proc/loadavg: 8.22 7.97 7.87 9/186 16574
/proc/meminfo: memFree=26786612/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=1545900 CPUtime=646.1 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 348605 0 0 0 64354 256 0 0 25 0 5 0 521254235 1583001600 347253 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 386475 347253 176 36 0 384380 0
[pid=16562/tid=16564] ppid=16560 vsize=1545900 CPUtime=161.93 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 53806 0 0 0 16034 159 0 0 25 0 5 0 521254265 1583001600 347253 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=1545900 CPUtime=161.96 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 110652 0 0 0 16169 27 0 0 25 0 5 0 521254265 1583001600 347253 33554432000 4194304 4338100 140734689946256 18446744073709551615 4284552 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=1545900 CPUtime=160.82 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 61837 0 0 0 16041 41 0 0 18 0 5 0 521254265 1583001600 347253 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=1545900 CPUtime=161.07 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 111475 0 0 0 16080 27 0 0 25 0 5 0 521254265 1583001600 347253 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 646.1
Current children cumulated vsize (KiB) 1545900


################
# More data... #
################

/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 222016 0 0 0 69982 61 0 0 25 0 5 0 521254265 3003813888 704976 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2800.7
Current children cumulated vsize (KiB) 2933412

[startup+762.307 s]
/proc/loadavg: 8.10 8.05 7.94 9/186 16590
/proc/meminfo: memFree=24214956/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3025972 CPUtime=3040.13 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 730564 0 0 0 303277 736 0 0 25 0 5 0 521254235 3098595328 729212 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 756493 729213 176 36 0 754398 0
[pid=16562/tid=16564] ppid=16560 vsize=3025972 CPUtime=761.73 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 122171 0 0 0 75673 500 0 0 25 0 5 0 521254265 3098595328 729213 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3025972 CPUtime=759.29 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 241739 0 0 0 75860 69 0 0 25 0 5 0 521254265 3098595328 729213 33554432000 4194304 4338100 140734689946256 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3025972 CPUtime=758.52 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 128495 0 0 0 75750 102 0 0 19 0 5 0 521254265 3098595328 729213 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3025972 CPUtime=760.27 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 227325 0 0 0 75964 63 0 0 25 0 5 0 521254265 3098595328 729213 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3040.13
Current children cumulated vsize (KiB) 3025972

[startup+822.301 s]
/proc/loadavg: 8.10 8.05 7.94 9/186 16591
/proc/meminfo: memFree=23970656/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3137300 CPUtime=3279.58 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 759746 0 0 0 327184 774 0 0 25 0 5 0 521254235 3212595200 758394 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 784325 758394 176 36 0 782230 0
[pid=16562/tid=16564] ppid=16560 vsize=3137300 CPUtime=821.72 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 126996 0 0 0 81644 528 0 0 25 0 5 0 521254265 3212595200 758394 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3137300 CPUtime=819.28 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 252924 0 0 0 81856 72 0 0 25 0 5 0 521254265 3212595200 758394 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3137300 CPUtime=818.34 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 133047 0 0 0 81727 107 0 0 18 0 5 0 521254265 3212595200 758394 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3137300 CPUtime=819.93 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 235944 0 0 0 81928 65 0 0 25 0 5 0 521254265 3212595200 758394 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3279.58
Current children cumulated vsize (KiB) 3137300

[startup+882.301 s]
/proc/loadavg: 6.09 7.54 7.78 5/176 16611
/proc/meminfo: memFree=27741236/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3280568 CPUtime=3518.9 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 796966 0 0 0 351054 836 0 0 25 0 5 0 521254235 3359301632 795614 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 820142 795614 176 36 0 818047 0
[pid=16562/tid=16564] ppid=16560 vsize=3280568 CPUtime=881.68 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 137129 0 0 0 87593 575 0 0 25 0 5 0 521254265 3359301632 795614 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3280568 CPUtime=879.28 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 262429 0 0 0 87852 76 0 0 25 0 5 0 521254265 3359301632 795614 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3280568 CPUtime=877.99 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 144167 0 0 0 87685 114 0 0 18 0 5 0 521254265 3359301632 795614 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3280568 CPUtime=879.63 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 242406 0 0 0 87895 68 0 0 25 0 5 0 521254265 3359301632 795614 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3518.9
Current children cumulated vsize (KiB) 3280568

[startup+942.301 s]
/proc/loadavg: 4.84 6.91 7.54 5/176 16612
/proc/meminfo: memFree=27666904/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3357488 CPUtime=3758.34 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 815870 0 0 0 374956 878 0 0 25 0 5 0 521254235 3438067712 814518 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 839372 814518 176 36 0 837277 0
[pid=16562/tid=16564] ppid=16560 vsize=3357488 CPUtime=941.67 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 141222 0 0 0 93560 607 0 0 25 0 5 0 521254265 3438067712 814518 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3357488 CPUtime=939.26 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 267276 0 0 0 93848 78 0 0 25 0 5 0 521254265 3438067712 814518 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3357488 CPUtime=937.74 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 150450 0 0 0 93654 120 0 0 18 0 5 0 521254265 3438067712 814518 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3357488 CPUtime=939.35 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 246087 0 0 0 93864 71 0 0 25 0 5 0 521254265 3438067712 814518 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3758.34
Current children cumulated vsize (KiB) 3357488

[startup+1002.3 s]
/proc/loadavg: 4.30 6.38 7.32 5/176 16614
/proc/meminfo: memFree=27630236/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3389216 CPUtime=3997.87 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 824783 0 0 0 398846 941 0 0 25 0 5 0 521254235 3470557184 823431 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 847304 823431 176 36 0 845209 0
[pid=16562/tid=16564] ppid=16560 vsize=3389216 CPUtime=1001.45 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 141654 0 0 0 99503 642 0 0 25 0 5 0 521254265 3470557184 823432 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813635518 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3389216 CPUtime=999.23 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 272255 0 0 0 99842 81 0 0 25 0 5 0 521254265 3470557184 823432 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3389216 CPUtime=997.64 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 151172 0 0 0 99620 144 0 0 25 0 5 0 521254265 3470557184 823432 33554432000 4194304 4338100 140734689946256 18446744073709551615 251813635214 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3389216 CPUtime=999.23 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 248868 0 0 0 99851 72 0 0 25 0 5 0 521254265 3470557184 823432 33554432000 4194304 4338100 140734689946256 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3997.87
Current children cumulated vsize (KiB) 3389216

[startup+1062.3 s]
/proc/loadavg: 4.11 5.94 7.11 5/176 16616
/proc/meminfo: memFree=27542736/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3478516 CPUtime=4237.47 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 847045 0 0 0 422691 1056 0 0 25 0 5 0 521254235 3562000384 845693 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 869629 845693 176 36 0 867534 0
[pid=16562/tid=16564] ppid=16560 vsize=3478516 CPUtime=1061.11 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 144342 0 0 0 105415 696 0 0 25 0 5 0 521254265 3562000384 845693 33554432000 4194304 4338100 140734689946256 18446744073709551615 4285817 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3478516 CPUtime=1059.21 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 280101 0 0 0 105838 83 0 0 25 0 5 0 521254265 3562000384 845693 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3478516 CPUtime=1057.58 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 157743 0 0 0 105559 199 0 0 25 0 5 0 521254265 3562000384 845693 33554432000 4194304 4338100 140734689946256 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3478516 CPUtime=1059.23 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 254024 0 0 0 105849 74 0 0 25 0 5 0 521254265 3562000384 845693 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4237.47
Current children cumulated vsize (KiB) 3478516

[startup+1122.3 s]
/proc/loadavg: 4.04 5.59 6.91 5/176 16618
/proc/meminfo: memFree=27433784/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3577200 CPUtime=4477.11 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 873696 0 0 0 446559 1152 0 0 25 0 5 0 521254235 3663052800 872344 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 894300 872344 176 36 0 892205 0
[pid=16562/tid=16564] ppid=16560 vsize=3577200 CPUtime=1121.07 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 149888 0 0 0 111364 743 0 0 25 0 5 0 521254265 3663052800 872344 33554432000 4194304 4338100 140734689946256 18446744073709551615 4289126 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3577200 CPUtime=1119.2 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 291192 0 0 0 111833 87 0 0 25 0 5 0 521254265 3663052800 872344 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3577200 CPUtime=1117.29 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 162008 0 0 0 111485 244 0 0 25 0 5 0 521254265 3663052800 872344 33554432000 4194304 4338100 140734689946256 18446744073709551615 4289508 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3577200 CPUtime=1119.23 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 259773 0 0 0 111847 76 0 0 25 0 5 0 521254265 3663052800 872344 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4477.11
Current children cumulated vsize (KiB) 3577200

[startup+1182.3 s]
/proc/loadavg: 4.07 5.31 6.73 5/176 16619
/proc/meminfo: memFree=27406448/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=3605192 CPUtime=4716.78 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) S 16560 16562 16318 0 -1 4202496 880504 0 0 0 470418 1260 0 0 25 0 5 0 521254235 3691716608 879152 33554432000 4194304 4338100 140734689946256 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 901298 879152 176 36 0 899203 0
[pid=16562/tid=16564] ppid=16560 vsize=3605192 CPUtime=1181.05 cores=0,2,4,6
/proc/16562/task/16564/stat : 16564 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 150388 0 0 0 117308 797 0 0 25 0 5 0 521254265 3691716608 879152 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=3605192 CPUtime=1179.2 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 292467 0 0 0 117832 88 0 0 25 0 5 0 521254265 3691716608 879152 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16562/tid=16566] ppid=16560 vsize=3605192 CPUtime=1176.99 cores=0,2,4,6
/proc/16562/task/16566/stat : 16566 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 162612 0 0 0 117406 293 0 0 25 0 5 0 521254265 3691716608 879152 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16562/tid=16567] ppid=16560 vsize=3605192 CPUtime=1179.21 cores=0,2,4,6
/proc/16562/task/16567/stat : 16567 (ttsth-5-4) R 16560 16562 16318 0 -1 4202560 264202 0 0 0 117842 79 0 0 25 0 5 0 521254265 3691716608 879152 33554432000 4194304 4338100 140734689946256 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4716.78
Current children cumulated vsize (KiB) 3605192



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.2 s]
/proc/loadavg: 4.05 5.22 6.67 2/173 16620
/proc/meminfo: memFree=27905388/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=0 CPUtime=4800.22 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) Z 16560 16562 16318 0 -1 4203548 887485 0 0 0 478722 1300 0 0 18 0 2 0 521254235 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 0 0 0 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=0 CPUtime=1200.11 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4203588 292814 0 0 0 119919 92 0 0 25 0 2 0 521254265 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.31 s]
/proc/loadavg: 4.05 5.22 6.67 2/174 16621
/proc/meminfo: memFree=30468856/32950928 swapFree=67111448/67111528
[pid=16562] ppid=16560 vsize=0 CPUtime=4800.33 cores=0,2,4,6
/proc/16562/stat : 16562 (ttsth-5-4) Z 16560 16562 16318 0 -1 4203548 887485 0 0 0 478722 1311 0 0 18 0 2 0 521254235 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/16562/statm: 0 0 0 0 0 0 0
[pid=16562/tid=16565] ppid=16560 vsize=0 CPUtime=1200.22 cores=0,2,4,6
/proc/16562/task/16565/stat : 16565 (ttsth-5-4) R 16560 16562 16318 0 -1 4203588 292814 0 0 0 119919 103 0 0 25 0 2 0 521254265 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.33
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.33
CPU time (s): 4800.36
CPU user time (s): 4787.23
CPU system time (s): 13.131
CPU usage (%): 398.922
Max. virtual memory (cumulated for all children) (KiB): 3637132

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4787.23
system time used= 13.131
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 887485
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10010
involuntary context switches= 28677

runsolver used 1.9947 second user time and 4.28935 second system time

The end

Launcher Data 

Begin job on node126 at 2011-04-29 19:02:35
IDJOB=3263454
IDBENCH=83976
IDSOLVER=1654
FILE ID=node126/3263454-1304096555
RUNJOBID= node126-1304094101-16336
PBS_JOBID= 13170590
Free space on /tmp= 71596 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_22_70_7-dir.shuffled-as.sat03-154.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263454-1304096555/watcher-3263454-1304096555 -o /tmp/evaluation-result-3263454-1304096555/solver-3263454-1304096555 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263454-1304096555.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= aa9ee2fa26214e80fac6d004d5d0b003
RANDOM SEED=313827340

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      28736264 kB
Buffers:        247120 kB
Cached:        1398596 kB
SwapCached:         76 kB
Active:        2305432 kB
Inactive:      1594252 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      28736264 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:             848 kB
Writeback:           0 kB
AnonPages:     2253780 kB
Mapped:          15220 kB
Slab:           246624 kB
PageTables:       9384 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  2461412 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71596 MiB
End job on node126 at 2011-04-29 19:22:41