Trace number 3262970

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.24 1202.66

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_80_28_70_8-dir.shuffled-as.sat03-215.cnf
MD5SUM731331ace9dcb7404987f98927f5c914
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables840
Number of clauses72688
Sum of the clauses size146160
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 272660
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 528

Solver Data 

0.00/0.05	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262970-1304095130.cnf 
0.00/0.05	c Using SATTIMEOUT of 4800
0.00/0.05	c Using SATRAM of 15500
0.00/0.05	c Size of subset node = 32
0.00/0.05	c Variables = 840
0.00/0.05	c Clauses = 72688
0.00/0.05	c Literals = 146160
0.00/0.05	c Reduced size:
0.00/0.05	c Variables = 840
0.00/0.05	c Clauses = 72688
0.00/0.05	c Literals = 146160
0.00/0.05	c a2
0.00/0.10	c a1
0.19/0.20	c Independent sub-problems = 1
0.19/0.20	c Sub-problem with 840 variables and 72688 clauses
0.19/0.20	c using permutation method 4
0.19/0.20	c count = 1
0.19/0.20	c Ordering finished 0.180 seconds
0.19/0.20	c mapped 0.190 seconds
0.19/0.20	c permuted 0.190 seconds
0.19/0.20	c about to sort
0.19/0.21	c sorted
0.29/0.38	c clauses now 72688
0.29/0.38	c init finished 0.220 seconds
0.29/0.38	c tree built 0.370 seconds
0.29/0.38	c 
0.29/0.38	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.38	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.38	c ................................ | ..........                    |                                  0.4         0.0             
0.29/0.38	c ................................ | ..........                  1 |                                  0.4         2.7             
0.29/0.38	c ................................ | ..........                  2 |                                  0.4         5.4             
0.29/0.38	c ................................ | ..........                  3 |                                  0.4         8.1             
0.29/0.38	c ................................ | ..........                  4 |                                  0.4        10.8             
0.29/0.38	c ................................ | ..........                  5 |                                  0.4        13.5             
0.29/0.39	c ................................ | ..........               1000 |              46                  0.4      2631.6         2085
95.73/24.37	c ................................ | ..........            3701000 | 105926       331860             96.0     38548.1     12355950
191.83/48.44	c ................................ | ..........            5826000 | 187655       501650            192.0     30343.8     20227023
287.95/72.52	c ................................ | ..........            8118000 | 261889       655138            288.1     28182.6     25927663
383.70/96.58	c ................................ | ..........            9796000 | 328739       783203            384.0     25509.8     30609155
479.86/120.63	c ................................ | *.........           11654000 | 389562       904115            480.0     24278.7     35978473
575.63/144.69	c ................................ | *.........           13445000 | 451348       1017050           576.0     23342.0     39591477
671.80/168.75	c ................................ | *.........           14858000 | 518394       1116273           672.0     22109.8     43594706
767.95/192.82	c ................................ | *.........           16045000 | 566732       1213474           768.1     20890.3     48132591
863.69/216.88	c ................................ | *.........           17232000 | 606206       1292237           864.0     19944.0     51942790
959.83/240.95	c ................................ | **........           18528000 | 645338       1378327           960.0     19299.2     54739294
1056.03/265.02	c ................................ | **........           19602000 | 683565       1456503          1056.1     18560.6     57414821
1151.90/289.05	c ................................ | **........           20677000 | 725587       1523141          1152.1     17947.1     60072538
1247.79/313.05	c ................................ | **........           21703000 | 771216       1609466          1248.0     17389.9     62958101
1344.07/337.12	c ................................ | **........           22580000 | 803061       1681255          1344.2     16798.7     65723933
1439.88/361.15	c ................................ | ***.......           23436000 | 824607       1756648          1440.1     16274.0     68254807
1536.11/385.20	c ................................ | ***.......           24288000 | 856353       1820036          1536.2     15811.0     70441331
1631.98/409.24	c ................................ | ***.......           25441000 | 900180       1896810          1632.1     15587.5     71974388
1727.86/433.24	c ................................ | ***.......           26529000 | 942385       1973952          1728.0     15352.3     71981870
1823.73/457.27	c ................................ | ***.......           27261000 | 970575       2025710          1824.0     14945.5     73035253
1920.02/481.32	c ................................ | ****......           28209000 | 1026272      2104204          1920.1     14691.3     74410593
2015.88/505.33	c ................................ | ****......           28899000 | 1062202      2196759          2016.0     14334.8     76121219
2111.77/529.36	c ................................ | ****......           29727000 | 1103156      2286995          2112.0     14075.2     77505126
2207.65/553.40	c ................................ | ****......           30509000 | 1134790      2364966          2208.0     13817.2     78923895
2303.93/577.43	c ................................ | ****......           31125000 | 1160846      2412955          2304.1     13508.7     81716624
2399.78/601.47	c ................................ | *****.....           31727000 | 1183717      2463126          2400.1     13219.1     84190890
2496.07/625.54	c ................................ | *****.....           32206000 | 1203632      2516583          2496.3     12901.7     85335850
2591.95/649.52	c ................................ | *****.....           32881000 | 1230922      2582067          2592.1     12685.3     86133937
2687.84/673.55	c ................................ | *****.....           33594000 | 1260987      2652242          2688.1     12497.5     87138528
2783.74/697.57	c ................................ | *****.....           34201000 | 1291153      2715787          2784.0     12284.8     88829298
2879.60/721.60	c ................................ | ******....           34858000 | 1320595      2776032          2880.0     12103.4     89233580
2975.84/745.64	c ................................ | ******....           35566000 | 1351459      2818004          2976.0     11950.9     90034503
3071.73/769.68	c ................................ | ******....           36451000 | 1380791      2881053          3072.1     11865.4     92168283
3167.62/793.69	c ................................ | ******....           37384000 | 1418437      2947040          3168.0     11800.5     94093002
3263.84/817.78	c ................................ | ******....           38533000 | 1459449      3036988          3264.2     11804.8     95840249
3359.67/841.78	c ................................ | *******...           39107000 | 1486556      3111490          3360.0     11639.0     97373496
3455.86/865.87	c ................................ | *******...           39779000 | 1516269      3166906          3456.1     11509.7     99416879
3552.07/889.90	c ................................ | *******...           40414000 | 1541990      3207116          3552.1     11377.5    100301081
3647.88/913.94	c ................................ | *******...           40974000 | 1564839      3237718          3648.1     11231.7    101873276
3743.70/937.99	c ................................ | *******...           41608000 | 1584890      3290847          3744.1     11113.0    102570506
3839.93/962.03	c ................................ | ********..           42179000 | 1608509      3339090          3840.1     10984.0    103890755
3936.16/986.10	c ................................ | ********..           42730000 | 1629989      3392207          3936.2     10855.7    104817298
4032.00/1010.10	c ................................ | ********..           43286000 | 1659575      3430977          4032.0     10735.5    105805300
4127.83/1034.14	c ................................ | ********..           43865000 | 1682801      3476374          4128.0     10626.2    106762555
4224.06/1058.21	c ................................ | ********..           44320000 | 1704543      3512271          4224.1     10492.2    108431838
4319.88/1082.24	c ................................ | *********.           44879000 | 1727602      3548141          4320.1     10388.5    110489732
4415.69/1106.28	c ................................ | *********.           45602000 | 1750589      3583173          4416.0     10326.5    111997583
4511.86/1130.35	c ................................ | *********.           46275000 | 1777006      3610171          4512.1     10255.8    112727439
4608.10/1154.41	c ................................ | *********.           46885000 | 1801531      3645319          4608.1     10174.4    112799172
4703.93/1178.41	c ................................ | *********.           47459000 | 1820691      3688195          4704.0     10089.1    113490330
4799.74/1202.48	c CPU limit (4800) reached
4799.74/1202.48	c Time = 4800.06 seconds
4799.74/1202.48	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262970-1304095130/watcher-3262970-1304095130 -o /tmp/evaluation-result-3262970-1304095130/solver-3262970-1304095130 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262970-1304095130.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.10 7.88 7.95 3/174 1963
/proc/meminfo: memFree=31075864/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=8848 CPUtime=0 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) R 1961 1963 933 0 -1 4202496 286 0 0 0 0 0 0 0 23 0 1 0 521113805 9060352 228 33554432000 4194304 4338100 140734737881504 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/1963/statm: 2212 229 146 36 0 117 0

[startup+0.103866 s]
/proc/loadavg: 7.10 7.88 7.95 3/174 1963
/proc/meminfo: memFree=31075864/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=24192 CPUtime=0.1 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) R 1961 1963 933 0 -1 4202496 3977 0 0 0 10 0 0 0 23 0 1 0 521113805 24772608 3918 33554432000 4194304 4338100 140734737881504 18446744073709551615 4228108 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/1963/statm: 6048 3919 153 36 0 3953 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 24192

[startup+0.200846 s]
/proc/loadavg: 7.10 7.88 7.95 3/174 1963
/proc/meminfo: memFree=31075864/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=24956 CPUtime=0.19 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) R 1961 1963 933 0 -1 4202496 5578 0 0 0 19 0 0 0 24 0 1 0 521113805 25554944 4274 33554432000 4194304 4338100 140734737881504 18446744073709551615 222921432076 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/1963/statm: 6239 4274 161 36 0 4144 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 24956

[startup+0.300827 s]
/proc/loadavg: 7.10 7.88 7.95 3/174 1963
/proc/meminfo: memFree=31075864/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=41556 CPUtime=0.29 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) R 1961 1963 933 0 -1 4202496 9864 0 0 0 29 0 0 0 25 0 1 0 521113805 42553344 8418 33554432000 4194304 4338100 140734737881504 18446744073709551615 222921435293 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/1963/statm: 10422 8427 164 36 0 8327 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 41556

[startup+0.701698 s]
/proc/loadavg: 7.10 7.88 7.95 3/174 1963
/proc/meminfo: memFree=31075864/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=176260 CPUtime=1.62 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 16242 0 0 0 161 1 0 0 25 0 5 0 521113805 180490240 14796 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 44065 14796 176 36 0 41970 0
Current children cumulated CPU time (s) 1.62
Current children cumulated vsize (KiB) 176260

[startup+1.50054 s]
/proc/loadavg: 7.10 7.88 7.95 6/182 1974
/proc/meminfo: memFree=30964584/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=198420 CPUtime=4.81 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 21312 0 0 0 478 3 0 0 25 0 5 0 521113805 203182080 19866 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 49605 19866 176 36 0 47510 0
[pid=1963/tid=1971] ppid=1961 vsize=198420 CPUtime=1.09 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 2498 0 0 0 109 0 0 0 18 0 5 0 521113844 203182080 19867 33554432000 4194304 4338100 140734737881504 18446744073709551615 222921171984 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=198420 CPUtime=1.1 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 2359 0 0 0 110 0 0 0 18 0 5 0 521113844 203182080 19867 33554432000 4194304 4338100 140734737881504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=198420 CPUtime=1.1 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 366 0 0 0 110 0 0 0 25 0 5 0 521113844 203182080 19867 33554432000 4194304 4338100 140734737881504 18446744073709551615 4241581 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=198420 CPUtime=1.1 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 2541 0 0 0 110 0 0 0 19 0 5 0 521113844 203182080 19868 33554432000 4194304 4338100 140734737881504 18446744073709551615 4253933 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4.81
Current children cumulated vsize (KiB) 198420

[startup+3.10022 s]
/proc/loadavg: 6.93 7.84 7.93 6/182 1974
/proc/meminfo: memFree=30960248/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=295764 CPUtime=11.18 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 29936 0 0 0 1112 6 0 0 25 0 5 0 521113805 302862336 28490 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 73941 28490 176 36 0 71846 0
[pid=1963/tid=1971] ppid=1961 vsize=295764 CPUtime=2.67 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 5681 0 0 0 266 1 0 0 18 0 5 0 521113844 302862336 28490 33554432000 4194304 4338100 140734737881504 18446744073709551615 222921172211 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=295764 CPUtime=2.7 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 4726 0 0 0 269 1 0 0 21 0 5 0 521113844 302862336 28490 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=295764 CPUtime=2.7 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 977 0 0 0 269 1 0 0 25 0 5 0 521113844 302862336 28490 33554432000 4194304 4338100 140734737881504 18446744073709551615 222921172718 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=295764 CPUtime=2.7 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 5003 0 0 0 270 0 0 0 23 0 5 0 521113844 302862336 28490 33554432000 4194304 4338100 140734737881504 18446744073709551615 4240554 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.18
Current children cumulated vsize (KiB) 295764

[startup+6.30063 s]
/proc/loadavg: 6.93 7.84 7.93 9/186 1978
/proc/meminfo: memFree=30706920/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=368520 CPUtime=23.94 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 48022 0 0 0 2382 12 0 0 25 0 5 0 521113805 377364480 46576 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 92130 46576 176 36 0 90035 0
[pid=1963/tid=1971] ppid=1961 vsize=368520 CPUtime=5.87 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 8610 0 0 0 583 4 0 0 25 0 5 0 521113844 377364480 46576 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=368520 CPUtime=5.9 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 11953 0 0 0 588 2 0 0 25 0 5 0 521113844 377364480 46576 33554432000 4194304 4338100 140734737881504 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=368520 CPUtime=5.86 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 4328 0 0 0 584 2 0 0 18 0 5 0 521113844 377364480 46576 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=368520 CPUtime=5.9 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 9582 0 0 0 588 2 0 0 25 0 5 0 521113844 377364480 46576 33554432000 4194304 4338100 140734737881504 18446744073709551615 4287100 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 23.94
Current children cumulated vsize (KiB) 368520

[startup+12.7005 s]
/proc/loadavg: 7.17 7.86 7.94 9/186 1978
/proc/meminfo: memFree=30315084/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=475664 CPUtime=49.44 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 74979 0 0 0 4919 25 0 0 25 0 5 0 521113805 487079936 73533 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 118916 73533 176 36 0 116821 0
[pid=1963/tid=1971] ppid=1961 vsize=475664 CPUtime=12.27 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 12692 0 0 0 1217 10 0 0 25 0 5 0 521113844 487079936 73533 33554432000 4194304 4338100 140734737881504 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=475664 CPUtime=12.3 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 23137 0 0 0 1226 4 0 0 25 0 5 0 521113844 487079936 73533 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=475664 CPUtime=12.17 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 8947 0 0 0 1212 5 0 0 18 0 5 0 521113844 487079936 73533 33554432000 4194304 4338100 140734737881504 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=475664 CPUtime=12.3 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 16654 0 0 0 1226 4 0 0 25 0 5 0 521113844 487079936 73533 33554432000 4194304 4338100 140734737881504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 49.44
Current children cumulated vsize (KiB) 475664

[startup+25.5011 s]
/proc/loadavg: 7.30 7.86 7.94 9/186 1979
/proc/meminfo: memFree=29739480/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=640152 CPUtime=100.49 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 116392 0 0 0 10000 49 0 0 25 0 5 0 521113805 655515648 114946 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 160038 114947 176 36 0 157943 0
[pid=1963/tid=1971] ppid=1961 vsize=640152 CPUtime=25.06 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 19079 0 0 0 2483 23 0 0 25 0 5 0 521113844 655515648 114947 33554432000 4194304 4338100 140734737881504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=640152 CPUtime=25.1 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 40358 0 0 0 2502 8 0 0 25 0 5 0 521113844 655515648 114947 33554432000 4194304 4338100 140734737881504 18446744073709551615 4288215 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=640152 CPUtime=24.81 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 15682 0 0 0 2472 9 0 0 18 0 5 0 521113844 655515648 114947 33554432000 4194304 4338100 140734737881504 18446744073709551615 4241507 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=640152 CPUtime=25.1 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 27725 0 0 0 2504 6 0 0 25 0 5 0 521113844 655515648 114947 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 100.49
Current children cumulated vsize (KiB) 640152

[startup+51.1004 s]
/proc/loadavg: 7.54 7.87 7.94 9/186 1980
/proc/meminfo: memFree=28931900/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=904788 CPUtime=202.59 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 181907 0 0 0 20175 84 0 0 25 0 5 0 521113805 926502912 180461 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 226197 180461 176 36 0 224102 0
[pid=1963/tid=1971] ppid=1961 vsize=904788 CPUtime=50.66 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 29754 0 0 0 5023 43 0 0 25 0 5 0 521113844 926502912 180462 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=904788 CPUtime=50.7 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 63049 0 0 0 5057 13 0 0 25 0 5 0 521113844 926502912 180462 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=904788 CPUtime=50.11 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 26673 0 0 0 4996 15 0 0 18 0 5 0 521113844 926502912 180462 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=904788 CPUtime=50.7 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 48883 0 0 0 5059 11 0 0 25 0 5 0 521113844 926502912 180462 33554432000 4194304 4338100 140734737881504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 202.59
Current children cumulated vsize (KiB) 904788

[startup+102.301 s]
/proc/loadavg: 7.95 7.92 7.95 9/186 1981
/proc/meminfo: memFree=27886548/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=1246044 CPUtime=406.86 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 268741 0 0 0 40539 147 0 0 25 0 5 0 521113805 1275949056 267295 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 311511 267295 176 36 0 309416 0
[pid=1963/tid=1971] ppid=1961 vsize=1246044 CPUtime=101.74 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 46671 0 0 0 10103 71 0 0 18 0 5 0 521113844 1275949056 267298 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=1246044 CPUtime=101.91 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 91576 0 0 0 10172 19 0 0 25 0 5 0 521113844 1275949056 267298 33554432000 4194304 4338100 140734737881504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=1246044 CPUtime=101.02 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 43261 0 0 0 10063 39 0 0 25 0 5 0 521113844 1275949056 267298 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=1246044 CPUtime=101.78 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 73687 0 0 0 10162 16 0 0 25 0 5 0 521113844 1275949056 267298 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 406.86
Current children cumulated vsize (KiB) 1246044

[startup+162.301 s]
/proc/loadavg: 8.04 7.95 7.95 9/186 1982
/proc/meminfo: memFree=27270952/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=1579628 CPUtime=646.27 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 353171 0 0 0 64410 217 0 0 25 0 5 0 521113805 1617539072 351725 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 394907 351725 176 36 0 392812 0
[pid=1963/tid=1971] ppid=1961 vsize=1579628 CPUtime=161.44 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 64731 0 0 0 16057 87 0 0 18 0 5 0 521113844 1617539072 351725 33554432000 4194304 4338100 140734737881504 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=1579628 CPUtime=161.9 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 117999 0 0 0 16166 24 0 0 25 0 5 0 521113844 1617539072 351725 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=1579628 CPUtime=161.01 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 60335 0 0 0 16020 81 0 0 25 0 5 0 521113844 1617539072 351725 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=1579628 CPUtime=161.5 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 96557 0 0 0 16128 22 0 0 25 0 5 0 521113844 1617539072 351725 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 646.27
Current children cumulated vsize (KiB) 1579628


################
# More data... #
################

/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 205981 0 0 0 70036 64 0 0 25 0 5 0 521113844 3193966592 737596 33554432000 4194304 4338100 140734737881504 18446744073709551615 222921172048 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2802.9
Current children cumulated vsize (KiB) 3119108

[startup+762.301 s]
/proc/loadavg: 4.12 4.58 6.08 5/176 2015
/proc/meminfo: memFree=28030160/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3215452 CPUtime=3042.58 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 761869 0 0 0 303450 808 0 0 25 0 5 0 521113805 3292622848 760423 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 803863 760423 176 36 0 801768 0
[pid=1963/tid=1971] ppid=1961 vsize=3215452 CPUtime=760.83 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 140843 0 0 0 75755 328 0 0 25 0 5 0 521113844 3292622848 760423 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3215452 CPUtime=761.8 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 275253 0 0 0 76106 74 0 0 25 0 5 0 521113844 3292622848 760423 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3215452 CPUtime=758.56 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 115494 0 0 0 75520 336 0 0 18 0 5 0 521113844 3292622848 760423 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3215452 CPUtime=760.99 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 216730 0 0 0 76031 68 0 0 25 0 5 0 521113844 3292622848 760423 33554432000 4194304 4338100 140734737881504 18446744073709551615 4285363 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3042.58
Current children cumulated vsize (KiB) 3215452

[startup+822.301 s]
/proc/loadavg: 4.12 4.50 5.96 5/176 2017
/proc/meminfo: memFree=27892084/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3351540 CPUtime=3282.21 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 796461 0 0 0 327348 873 0 0 25 0 5 0 521113805 3431976960 795015 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 837885 795015 176 36 0 835790 0
[pid=1963/tid=1971] ppid=1961 vsize=3351540 CPUtime=820.78 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 141073 0 0 0 81723 355 0 0 18 0 5 0 521113844 3431976960 795015 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3351540 CPUtime=821.79 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 290418 0 0 0 82099 80 0 0 25 0 5 0 521113844 3431976960 795015 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3351540 CPUtime=818.3 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 115701 0 0 0 81469 361 0 0 25 0 5 0 521113844 3431976960 795015 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3351540 CPUtime=820.93 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 235720 0 0 0 82018 75 0 0 25 0 5 0 521113844 3431976960 795015 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3282.21
Current children cumulated vsize (KiB) 3351540

[startup+882.306 s]
/proc/loadavg: 4.11 4.41 5.83 5/176 2019
/proc/meminfo: memFree=27754764/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3490360 CPUtime=3521.74 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 830558 0 0 0 351265 909 0 0 25 0 5 0 521113805 3574128640 829112 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 872590 829112 176 36 0 870495 0
[pid=1963/tid=1971] ppid=1961 vsize=3490360 CPUtime=880.6 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 141197 0 0 0 87701 359 0 0 21 0 5 0 521113844 3574128640 829112 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3490360 CPUtime=881.79 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 308657 0 0 0 88094 85 0 0 25 0 5 0 521113844 3574128640 829112 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3490360 CPUtime=878.29 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 115824 0 0 0 87447 382 0 0 25 0 5 0 521113844 3574128640 829112 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3490360 CPUtime=880.64 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 251331 0 0 0 87984 80 0 0 25 0 5 0 521113844 3574128640 829112 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3521.74
Current children cumulated vsize (KiB) 3490360

[startup+942.3 s]
/proc/loadavg: 4.04 4.34 5.72 5/176 2020
/proc/meminfo: memFree=27674844/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3569352 CPUtime=3761.27 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 850723 0 0 0 375188 939 0 0 25 0 5 0 521113805 3655016448 849277 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 892338 849277 176 36 0 890243 0
[pid=1963/tid=1971] ppid=1961 vsize=3569352 CPUtime=940.43 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 143885 0 0 0 93679 364 0 0 25 0 5 0 521113844 3655016448 849277 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3569352 CPUtime=941.77 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 319283 0 0 0 94089 88 0 0 25 0 5 0 521113844 3655016448 849278 33554432000 4194304 4338100 140734737881504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3569352 CPUtime=938.29 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 115927 0 0 0 93429 400 0 0 25 0 5 0 521113844 3655016448 849278 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3569352 CPUtime=940.36 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 258080 0 0 0 93952 84 0 0 25 0 5 0 521113844 3655016448 849278 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3761.27
Current children cumulated vsize (KiB) 3569352

[startup+1002.3 s]
/proc/loadavg: 4.07 4.29 5.61 5/176 2022
/proc/meminfo: memFree=27584632/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3657860 CPUtime=4000.85 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 873036 0 0 0 399115 970 0 0 25 0 5 0 521113805 3745648640 871590 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 914465 871590 176 36 0 912370 0
[pid=1963/tid=1971] ppid=1961 vsize=3657860 CPUtime=1000.29 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 144017 0 0 0 99662 367 0 0 21 0 5 0 521113844 3745648640 871590 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3657860 CPUtime=1001.78 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 339219 0 0 0 100081 97 0 0 25 0 5 0 521113844 3745648640 871590 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3657860 CPUtime=998.29 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 116033 0 0 0 99410 419 0 0 25 0 5 0 521113844 3745648640 871590 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254244 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3657860 CPUtime=1000.09 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 260218 0 0 0 99923 86 0 0 25 0 5 0 521113844 3745648640 871590 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4000.85
Current children cumulated vsize (KiB) 3657860

[startup+1062.3 s]
/proc/loadavg: 4.03 4.24 5.51 5/176 2023
/proc/meminfo: memFree=27481288/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3761876 CPUtime=4240.43 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 899232 0 0 0 423042 1001 0 0 25 0 5 0 521113805 3852161024 897786 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 940469 897786 176 36 0 938374 0
[pid=1963/tid=1971] ppid=1961 vsize=3761876 CPUtime=1060.14 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 145125 0 0 0 105644 370 0 0 20 0 5 0 521113844 3852161024 897786 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3761876 CPUtime=1061.78 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 354779 0 0 0 106077 101 0 0 25 0 5 0 521113844 3852161024 897786 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3761876 CPUtime=1058.28 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 116126 0 0 0 105392 436 0 0 25 0 5 0 521113844 3852161024 897786 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3761876 CPUtime=1059.81 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 269653 0 0 0 105890 91 0 0 25 0 5 0 521113844 3852161024 897786 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4240.43
Current children cumulated vsize (KiB) 3761876

[startup+1122.3 s]
/proc/loadavg: 3.97 4.17 5.40 5/176 2025
/proc/meminfo: memFree=27353756/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3884396 CPUtime=4479.92 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 930566 0 0 0 446955 1037 0 0 25 0 5 0 521113805 3977621504 929120 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 971099 929120 176 36 0 969004 0
[pid=1963/tid=1971] ppid=1961 vsize=3884396 CPUtime=1119.94 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 153257 0 0 0 111617 377 0 0 18 0 5 0 521113844 3977621504 929120 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3884396 CPUtime=1121.78 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 367550 0 0 0 112073 105 0 0 25 0 5 0 521113844 3977621504 929120 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3884396 CPUtime=1118.27 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 116217 0 0 0 111371 456 0 0 25 0 5 0 521113844 3977621504 929120 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3884396 CPUtime=1119.52 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 279993 0 0 0 111856 96 0 0 25 0 5 0 521113844 3977621504 929120 33554432000 4194304 4338100 140734737881504 18446744073709551615 4256104 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4479.92
Current children cumulated vsize (KiB) 3884396

[startup+1182.3 s]
/proc/loadavg: 3.99 4.14 5.32 5/176 2027
/proc/meminfo: memFree=27322084/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=3916336 CPUtime=4719.5 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) S 1961 1963 933 0 -1 4202496 938987 0 0 0 470888 1062 0 0 25 0 5 0 521113805 4010328064 937541 33554432000 4194304 4338100 140734737881504 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 979084 937541 176 36 0 976989 0
[pid=1963/tid=1971] ppid=1961 vsize=3916336 CPUtime=1179.8 cores=0,2,4,6
/proc/1963/task/1971/stat : 1971 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 154974 0 0 0 117600 380 0 0 19 0 5 0 521113844 4010328064 937541 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=1963/tid=1972] ppid=1961 vsize=3916336 CPUtime=1181.77 cores=0,2,4,6
/proc/1963/task/1972/stat : 1972 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 369389 0 0 0 118071 106 0 0 25 0 5 0 521113844 4010328064 937541 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=3916336 CPUtime=1178.27 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 116298 0 0 0 117352 475 0 0 25 0 5 0 521113844 4010328064 937541 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=1963/tid=1974] ppid=1961 vsize=3916336 CPUtime=1179.26 cores=0,2,4,6
/proc/1963/task/1974/stat : 1974 (ttsth-5-4) R 1961 1963 933 0 -1 4202560 284777 0 0 0 117827 99 0 0 25 0 5 0 521113844 4010328064 937541 33554432000 4194304 4338100 140734737881504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4719.5
Current children cumulated vsize (KiB) 3916336



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.5 s]
/proc/loadavg: 3.99 4.13 5.28 2/173 2027
/proc/meminfo: memFree=27568812/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=0 CPUtime=4800.07 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) Z 1961 1963 933 0 -1 4203548 944006 0 0 0 478933 1074 0 0 18 0 2 0 521113805 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 0 0 0 0 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=0 CPUtime=1198.46 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4203588 116346 0 0 0 119362 484 0 0 25 0 2 0 521113844 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.07
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.61 s]
/proc/loadavg: 3.99 4.13 5.28 2/173 2027
/proc/meminfo: memFree=27568812/32951124 swapFree=67110188/67111528
[pid=1963] ppid=1961 vsize=0 CPUtime=4800.18 cores=0,2,4,6
/proc/1963/stat : 1963 (ttsth-5-4) Z 1961 1963 933 0 -1 4203548 944006 0 0 0 478933 1085 0 0 18 0 2 0 521113805 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/1963/statm: 0 0 0 0 0 0 0
[pid=1963/tid=1973] ppid=1961 vsize=0 CPUtime=1198.57 cores=0,2,4,6
/proc/1963/task/1973/stat : 1973 (ttsth-5-4) R 1961 1963 933 0 -1 4203588 116346 0 0 0 119362 495 0 0 25 0 2 0 521113844 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.66
CPU time (s): 4800.24
CPU user time (s): 4789.33
CPU system time (s): 10.9023
CPU usage (%): 399.135
Max. virtual memory (cumulated for all children) (KiB): 3935500

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4789.33
system time used= 10.9023
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 944006
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6083
involuntary context switches= 19482

runsolver used 1.64075 second user time and 3.87041 second system time

The end

Launcher Data 

Begin job on node109 at 2011-04-29 18:38:51
IDJOB=3262970
IDBENCH=83907
IDSOLVER=1654
FILE ID=node109/3262970-1304095130
RUNJOBID= node109-1304095130-1949
PBS_JOBID= 13170563
Free space on /tmp= 72384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_80_28_70_8-dir.shuffled-as.sat03-215.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262970-1304095130/watcher-3262970-1304095130 -o /tmp/evaluation-result-3262970-1304095130/solver-3262970-1304095130 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262970-1304095130.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 731331ace9dcb7404987f98927f5c914
RANDOM SEED=1109323620

node109.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31076392 kB
Buffers:        721032 kB
Cached:         700836 kB
SwapCached:        712 kB
Active:         111968 kB
Inactive:      1375664 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31076392 kB
SwapTotal:    67111528 kB
SwapFree:     67110188 kB
Dirty:             944 kB
Writeback:           0 kB
AnonPages:       64604 kB
Mapped:          14396 kB
Slab:           324648 kB
PageTables:       4724 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   189244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72384 MiB
End job on node109 at 2011-04-29 18:58:55