Trace number 3249900

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.22 1203.92

General information on the benchmark

Namecrafted/skvortsov/
automata-synchronization/crn_20_361_s.cnf
MD5SUM6e5ac2b9425e98989eedf4cf638bde54
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark170.249
Satisfiable
(Un)Satisfiability was proved
Number of variables7942
Number of clauses14991
Sum of the clauses size44382
Maximum clause length3
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2591
Number of clauses of size 314400
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249900-1304073072.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 7942
0.00/0.01	c Clauses = 14991
0.00/0.01	c Literals = 44382
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 7942
0.00/0.01	c Clauses = 14991
0.00/0.01	c Literals = 44382
0.00/0.01	c a2
0.00/0.02	c a1
0.09/0.12	c Independent sub-problems = 1
0.09/0.12	c Sub-problem with 7942 variables and 14991 clauses
0.09/0.12	c using permutation method 4
0.09/0.12	c count = 2
0.09/0.12	c Ordering finished 0.110 seconds
0.09/0.12	c mapped 0.110 seconds
0.09/0.12	c permuted 0.110 seconds
0.09/0.12	c about to sort
0.09/0.12	c sorted
0.19/0.22	c clauses now 14991
0.19/0.22	c init finished 0.180 seconds
0.19/0.22	c tree built 0.210 seconds
0.19/0.22	c 
0.19/0.22	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.22	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.22	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.22	c ................................ | ..........                  1 |                                  0.2         4.8             
0.19/0.22	c ................................ | ..........                  2 |                                  0.2         9.5             
0.19/0.22	c ................................ | ..........                  3 |                                  0.2        14.3             
0.19/0.22	c ................................ | ..........                  4 |                                  0.2        19.0             
0.19/0.22	c ................................ | ..........                  5 |                                  0.2        23.8             
0.19/0.22	c ................................ | ..........               1000 |              42                  0.2      4761.9             
95.82/24.24	c ................................ | ..........            2210000 | 69228        161789             96.0     23020.8     14330816
191.98/48.31	c ................................ | ..........            3804000 | 118911       264550            192.0     19810.4     22709215
288.09/72.40	c ................................ | ..........            5432000 | 170434       367063            288.1     18853.9     30731780
383.84/96.45	c ................................ | ..........            6977000 | 220061       462650            384.0     18168.3     37053067
479.92/120.52	c ................................ | *.........            8738000 | 275975       566095            480.0     18204.2     42667496
575.62/144.59	c ................................ | *.........           10642000 | 336183       675192            576.0     18475.7     49201474
671.75/168.67	c ................................ | *.........           12447000 | 393287       779501            672.0     18521.8     54383537
768.19/192.80	c ................................ | *.........           14362000 | 454234       887772            768.0     18700.5     57751519
863.86/216.88	c ................................ | *.........           16157000 | 511498       999694            864.0     18699.1     62335901
959.93/240.94	c ................................ | **........           17564000 | 558482       1088595           960.1     18294.1     66633955
1056.16/265.02	c ................................ | **........           18144000 | 576692       1136315          1056.0     17181.8     66742576
1151.95/289.06	c ................................ | **........           18782000 | 596319       1206040          1152.1     16302.4     69749379
1248.17/313.12	c ................................ | **........           19512000 | 618579       1286544          1248.0     15634.1     74915334
1343.94/337.12	c ................................ | **........           20342000 | 643927       1326554          1344.0     15135.0     76146651
1440.14/361.20	c ................................ | ***.......           21164000 | 669287       1385706          1440.2     14695.7     77870848
1536.29/385.31	c ................................ | ***.......           22073000 | 697278       1461782          1536.0     14370.3     81319737
1632.03/409.39	c ................................ | ***.......           22987000 | 725557       1546100          1632.2     14083.9     86028142
1728.21/433.46	c ................................ | ***.......           23913000 | 754335       1588132          1728.1     13837.8     89680405
1823.96/457.45	c ................................ | ***.......           24860000 | 783871       1643470          1824.1     13628.6     91977511
1919.77/481.56	c ................................ | ****......           25923000 | 816957       1718335          1920.0     13501.4     96549014
2015.89/505.65	c ................................ | ****......           26669000 | 840284       1787950          2016.1     13228.3    101062060
2112.02/529.72	c ................................ | ****......           27729000 | 873876       1854465          2112.1     13128.6    104681780
2208.20/553.81	c ................................ | ****......           28415000 | 895381       1894178          2208.2     12867.7    105603930
2303.91/577.83	c ................................ | ****......           29421000 | 927464       1955814          2304.0     12769.4    107106658
2400.03/601.90	c ................................ | *****.....           30246000 | 953741       2020049          2400.0     12602.3    107125086
2495.77/625.96	c ................................ | *****.....           31050000 | 979113       2087497          2496.0     12439.9    110897553
2591.93/650.06	c ................................ | *****.....           31887000 | 1005180      2142907          2592.2     12301.2    113959636
2688.03/674.11	c ................................ | *****.....           32892000 | 1035198      2199353          2688.1     12236.2    116349378
2783.77/698.18	c ................................ | *****.....           33724000 | 1060237      2261600          2784.1     12113.0    120300635
2879.88/722.23	c ................................ | ******....           34692000 | 1090183      2329883          2880.0     12045.8    123962276
2975.98/746.36	c ................................ | ******....           35833000 | 1126390      2402624          2976.2     12039.6    126643790
3071.80/770.37	c ................................ | ******....           36711000 | 1154196      2452067          3072.1     11949.8    130414128
3167.92/794.42	c ................................ | ******....           37738000 | 1186142      2510409          3168.0     11912.2    131503003
3263.96/818.52	c ................................ | ******....           38856000 | 1221099      2576264          3264.1     11904.2    133429122
3359.71/842.59	c ................................ | *******...           39806000 | 1250204      2641530          3360.1     11846.7    133429122
3455.82/866.65	c ................................ | *******...           41010000 | 1288041      2716436          3456.1     11866.1    133620550
3551.92/890.73	c ................................ | *******...           42212000 | 1325267      2772535          3552.0     11883.9    135309458
3647.68/914.80	c ................................ | *******...           43153000 | 1355327      2834554          3648.1     11829.0    137048828
3743.74/938.88	c ................................ | *******...           44363000 | 1393428      2901894          3744.1     11848.8    138095352
3839.88/962.95	c ................................ | ********..           45751000 | 1437476      2980269          3840.1     11914.1    140564878
3936.00/987.02	c ................................ | ********..           46833000 | 1472177      3040409          3936.1     11898.4    142857228
4031.73/1011.07	c ................................ | ********..           47923000 | 1506395      3099503          4032.0     11885.6    144191920
4127.83/1035.14	c ................................ | ********..           49199000 | 1547022      3170070          4128.0     11918.3    146999163
4224.26/1059.30	c ................................ | ********..           50638000 | 1592860      3248643          4224.0     11988.1    149351694
4320.00/1083.33	c ................................ | *********.           51615000 | 1624313      3303148          4320.1     11947.6    152334316
4415.71/1107.38	c ................................ | *********.           52852000 | 1663562      3368442          4416.0     11968.2    154703950
4511.83/1131.45	c ................................ | *********.           54265000 | 1708683      3441693          4512.0     12026.7    157580645
4607.81/1155.54	c ................................ | *********.           55727000 | 1755524      3519577          4608.0     12093.5    158912723
4703.85/1179.65	c ................................ | *********.           56671000 | 1786385      3575019          4704.1     12047.3    158932294
4799.91/1203.72	c CPU limit (4800) reached
4799.91/1203.72	c Time = 4800.02 seconds
4799.91/1203.72	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249900-1304073072/watcher-3249900-1304073072 -o /tmp/evaluation-result-3249900-1304073072/solver-3249900-1304073072 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249900-1304073072.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.83 7.99 7.66 2/177 22622
/proc/meminfo: memFree=18468596/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) R 22620 22622 20564 0 -1 4202496 421 0 0 0 0 0 0 0 25 0 1 0 597537857 9633792 364 33554432000 4194304 4338100 140733246387152 18446744073709551615 256415052485 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22622/statm: 2352 364 147 36 0 257 0

[startup+0.103287 s]
/proc/loadavg: 7.83 7.99 7.66 2/177 22622
/proc/meminfo: memFree=18468596/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=14832 CPUtime=0.09 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) R 22620 22622 20564 0 -1 4202496 1800 0 0 0 9 0 0 0 25 0 1 0 597537857 15187968 1713 33554432000 4194304 4338100 140733246387152 18446744073709551615 4240533 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22622/statm: 3708 1713 153 36 0 1613 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 14832

[startup+0.201275 s]
/proc/loadavg: 7.83 7.99 7.66 2/177 22622
/proc/meminfo: memFree=18468596/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=98480 CPUtime=0.19 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) R 22620 22622 20564 0 -1 4202496 22406 0 0 0 17 2 0 0 25 0 1 0 597537857 100843520 22319 33554432000 4194304 4338100 140733246387152 18446744073709551615 256415052485 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22622/statm: 24621 22329 161 36 0 22526 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 98480

[startup+0.323137 s]
/proc/loadavg: 7.83 7.99 7.66 2/177 22622
/proc/meminfo: memFree=18468596/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=360264 CPUtime=0.58 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 29290 0 0 0 55 3 0 0 25 0 5 0 597537857 368910336 29203 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 90066 29722 174 36 0 87971 0
Current children cumulated CPU time (s) 0.58
Current children cumulated vsize (KiB) 360264

[startup+1.25096 s]
/proc/loadavg: 7.83 7.99 7.66 2/177 22622
/proc/meminfo: memFree=18468596/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=398064 CPUtime=4.28 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 39159 0 0 0 420 8 0 0 25 0 5 0 597537857 407617536 39072 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 99516 39073 174 36 0 97421 0
Current children cumulated CPU time (s) 4.28
Current children cumulated vsize (KiB) 398064

[startup+1.53491 s]
/proc/loadavg: 7.83 7.99 7.66 2/177 22622
/proc/meminfo: memFree=18468596/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=410576 CPUtime=5.41 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 41404 0 0 0 531 10 0 0 25 0 5 0 597537857 420429824 41317 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 102644 41317 174 36 0 100549 0
Current children cumulated CPU time (s) 5.41
Current children cumulated vsize (KiB) 410576

[startup+3.10062 s]
/proc/loadavg: 7.84 7.99 7.66 9/186 22631
/proc/meminfo: memFree=18196708/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=454664 CPUtime=11.66 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 52352 0 0 0 1146 20 0 0 25 0 5 0 597537857 465575936 52265 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 113666 52265 174 36 0 111571 0
[pid=22622/tid=22624] ppid=22620 vsize=454664 CPUtime=2.87 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 6603 0 0 0 281 6 0 0 25 0 5 0 597537879 465575936 52265 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=454664 CPUtime=2.86 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 5819 0 0 0 282 4 0 0 24 0 5 0 597537879 465575936 52265 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=454664 CPUtime=2.86 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 8217 0 0 0 281 5 0 0 24 0 5 0 597537879 465575936 52265 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=454664 CPUtime=2.82 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 6992 0 0 0 281 1 0 0 18 0 5 0 597537879 465575936 52265 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.66
Current children cumulated vsize (KiB) 454664

[startup+6.30101 s]
/proc/loadavg: 7.84 7.99 7.66 9/186 22663
/proc/meminfo: memFree=18147452/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=527036 CPUtime=24.42 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 70083 0 0 0 2401 41 0 0 25 0 5 0 597537857 539684864 69996 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 131759 69997 174 36 0 129664 0
[pid=22622/tid=22624] ppid=22620 vsize=527036 CPUtime=6.06 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 10426 0 0 0 593 13 0 0 25 0 5 0 597537879 539684864 69997 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=527036 CPUtime=6.06 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 9803 0 0 0 599 7 0 0 25 0 5 0 597537879 539684864 69997 33554432000 4194304 4338100 140733246387152 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=527036 CPUtime=6.05 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 13422 0 0 0 593 12 0 0 25 0 5 0 597537879 539684864 69997 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=527036 CPUtime=5.99 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 11712 0 0 0 595 4 0 0 18 0 5 0 597537879 539684864 69997 33554432000 4194304 4338100 140733246387152 18446744073709551615 4286628 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.42
Current children cumulated vsize (KiB) 527036

[startup+12.7008 s]
/proc/loadavg: 7.86 7.99 7.66 9/186 22663
/proc/meminfo: memFree=17991964/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=646044 CPUtime=49.98 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 100113 0 0 0 4921 77 0 0 25 0 5 0 597537857 661549056 100026 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 161511 100026 174 36 0 159416 0
[pid=22622/tid=22624] ppid=22620 vsize=646044 CPUtime=12.46 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 17351 0 0 0 1221 25 0 0 25 0 5 0 597537879 661549056 100027 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=646044 CPUtime=12.47 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 17686 0 0 0 1232 15 0 0 25 0 5 0 597537879 661549056 100027 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254809 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=646044 CPUtime=12.46 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 21123 0 0 0 1221 25 0 0 25 0 5 0 597537879 661549056 100027 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=646044 CPUtime=12.34 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 19233 0 0 0 1227 7 0 0 18 0 5 0 597537879 661549056 100027 33554432000 4194304 4338100 140733246387152 18446744073709551615 256414786594 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.98
Current children cumulated vsize (KiB) 646044

[startup+25.5004 s]
/proc/loadavg: 7.88 7.99 7.66 9/186 22663
/proc/meminfo: memFree=17774876/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=820176 CPUtime=101.01 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 144203 0 0 0 9957 144 0 0 25 0 5 0 597537857 839860224 144116 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 205044 144117 175 36 0 202949 0
[pid=22622/tid=22624] ppid=22620 vsize=820176 CPUtime=25.25 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 26856 0 0 0 2474 51 0 0 25 0 5 0 597537879 839860224 144117 33554432000 4194304 4338100 140733246387152 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=820176 CPUtime=25.26 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 27390 0 0 0 2499 27 0 0 25 0 5 0 597537879 839860224 144117 33554432000 4194304 4338100 140733246387152 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=820176 CPUtime=25.25 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 33234 0 0 0 2478 47 0 0 25 0 5 0 597537879 839860224 144117 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=820176 CPUtime=24.99 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 32003 0 0 0 2485 14 0 0 18 0 5 0 597537879 839860224 144118 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.01
Current children cumulated vsize (KiB) 820176

[startup+51.1006 s]
/proc/loadavg: 7.92 7.99 7.67 9/186 22664
/proc/meminfo: memFree=17407820/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=1100168 CPUtime=203.14 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 213561 0 0 0 20053 261 0 0 25 0 5 0 597537857 1126572032 213474 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 275042 213474 175 36 0 272947 0
[pid=22622/tid=22624] ppid=22620 vsize=1100168 CPUtime=50.83 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 43953 0 0 0 4980 103 0 0 25 0 5 0 597537879 1126572032 213474 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=1100168 CPUtime=50.86 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 42872 0 0 0 5040 46 0 0 25 0 5 0 597537879 1126572032 213474 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=1100168 CPUtime=50.82 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 52242 0 0 0 4997 85 0 0 25 0 5 0 597537879 1126572032 213474 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=1100168 CPUtime=50.38 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 49773 0 0 0 5015 23 0 0 18 0 5 0 597537879 1126572032 213474 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.14
Current children cumulated vsize (KiB) 1100168

[startup+102.301 s]
/proc/loadavg: 8.13 8.03 7.70 9/186 22665
/proc/meminfo: memFree=16759044/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=1558412 CPUtime=407.34 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 327708 0 0 0 40248 486 0 0 25 0 5 0 597537857 1595813888 327621 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 389603 327621 175 36 0 387508 0
[pid=22622/tid=22624] ppid=22620 vsize=1558412 CPUtime=101.97 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 73663 0 0 0 10007 190 0 0 25 0 5 0 597537879 1595813888 327621 33554432000 4194304 4338100 140733246387152 18446744073709551615 256414787028 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=1558412 CPUtime=102.04 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 65921 0 0 0 10120 84 0 0 25 0 5 0 597537879 1595813888 327622 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=1558412 CPUtime=101.98 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 85905 0 0 0 10029 169 0 0 18 0 5 0 597537879 1595813888 327622 33554432000 4194304 4338100 140733246387152 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=1558412 CPUtime=101.09 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 77499 0 0 0 10070 39 0 0 18 0 5 0 597537879 1595813888 327622 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 407.34
Current children cumulated vsize (KiB) 1558412

[startup+162.301 s]
/proc/loadavg: 8.18 8.07 7.73 9/186 22698
/proc/meminfo: memFree=16635320/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=2017936 CPUtime=646.63 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 442279 0 0 0 63894 769 0 0 25 0 5 0 597537857 2066366464 442192 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 504484 442192 175 36 0 502389 0
[pid=22622/tid=22624] ppid=22620 vsize=2017936 CPUtime=161.89 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 105434 0 0 0 15885 304 0 0 25 0 5 0 597537879 2066366464 442192 33554432000 4194304 4338100 140733246387152 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=2017936 CPUtime=161.75 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 85409 0 0 0 16063 112 0 0 25 0 5 0 597537879 2066366464 442192 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=2017936 CPUtime=161.79 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 120790 0 0 0 15898 281 0 0 25 0 5 0 597537879 2066366464 442192 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=2017936 CPUtime=160.94 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 105925 0 0 0 16027 67 0 0 19 0 5 0 597537879 2066366464 442192 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 646.63
Current children cumulated vsize (KiB) 2017936

[startup+222.3 s]
/proc/loadavg: 8.14 8.08 7.76 9/186 22699
/proc/meminfo: memFree=16095336/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=2391156 CPUtime=885.78 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 535468 0 0 0 87543 1035 0 0 25 0 5 0 597537857 2448543744 535381 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 597789 535381 175 36 0 595694 0
[pid=22622/tid=22624] ppid=22620 vsize=2391156 CPUtime=221.58 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 139845 0 0 0 21818 340 0 0 18 0 5 0 597537879 2448543744 535381 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 242912 0 0 0 69475 451 0 0 25 0 5 0 597537879 4316643328 987712 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2800.51
Current children cumulated vsize (KiB) 4215472

[startup+762.301 s]
/proc/loadavg: 8.07 8.09 7.89 9/186 22717
/proc/meminfo: memFree=13115816/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=4484480 CPUtime=3039.86 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1055255 0 0 0 301534 2452 0 0 25 0 5 0 597537857 4592107520 1055168 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1121120 1055168 175 36 0 1119025 0
[pid=22622/tid=22624] ppid=22620 vsize=4484480 CPUtime=759.48 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 278584 0 0 0 75279 669 0 0 25 0 5 0 597537879 4592107520 1055168 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=4484480 CPUtime=760.32 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 222620 0 0 0 75473 559 0 0 18 0 5 0 597537879 4592107520 1055168 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=4484480 CPUtime=760.56 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 271781 0 0 0 75317 739 0 0 25 0 5 0 597537879 4592107520 1055168 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=4484480 CPUtime=759.24 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 257549 0 0 0 75443 481 0 0 25 0 5 0 597537879 4592107520 1055169 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3039.86
Current children cumulated vsize (KiB) 4484480

[startup+822.301 s]
/proc/loadavg: 8.11 8.10 7.91 9/186 22718
/proc/meminfo: memFree=12871284/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=4660932 CPUtime=3279.13 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1091844 0 0 0 325297 2616 0 0 25 0 5 0 597537857 4772794368 1091757 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1165233 1091757 175 36 0 1163138 0
[pid=22622/tid=22624] ppid=22620 vsize=4660932 CPUtime=819.04 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 291145 0 0 0 81165 739 0 0 25 0 5 0 597537879 4772794368 1091757 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=4660932 CPUtime=820.15 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 227981 0 0 0 81451 564 0 0 18 0 5 0 597537879 4772794368 1091757 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=4660932 CPUtime=820.52 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 283040 0 0 0 81248 804 0 0 25 0 5 0 597537879 4772794368 1091757 33554432000 4194304 4338100 140733246387152 18446744073709551615 4241507 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=4660932 CPUtime=819.18 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 264957 0 0 0 81412 506 0 0 25 0 5 0 597537879 4772794368 1091757 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3279.13
Current children cumulated vsize (KiB) 4660932

[startup+882.301 s]
/proc/loadavg: 8.13 8.11 7.92 9/186 22720
/proc/meminfo: memFree=12775544/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=4689084 CPUtime=3518.42 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1099065 0 0 0 349064 2778 0 0 25 0 5 0 597537857 4801622016 1098978 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1172271 1098978 175 36 0 1170176 0
[pid=22622/tid=22624] ppid=22620 vsize=4689084 CPUtime=878.84 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 296943 0 0 0 87082 802 0 0 25 0 5 0 597537879 4801622016 1098978 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=4689084 CPUtime=879.94 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 227984 0 0 0 87426 568 0 0 18 0 5 0 597537879 4801622016 1098978 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=4689084 CPUtime=880.21 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 284456 0 0 0 87151 870 0 0 25 0 5 0 597537879 4801622016 1098978 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=4689084 CPUtime=879.16 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 264961 0 0 0 87383 533 0 0 25 0 5 0 597537879 4801622016 1098978 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3518.42
Current children cumulated vsize (KiB) 4689084

[startup+942.307 s]
/proc/loadavg: 8.04 8.08 7.92 9/186 22721
/proc/meminfo: memFree=12545148/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=4814204 CPUtime=3757.72 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1130114 0 0 0 372831 2941 0 0 25 0 5 0 597537857 4929744896 1130027 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1203551 1130027 175 36 0 1201456 0
[pid=22622/tid=22624] ppid=22620 vsize=4814204 CPUtime=938.43 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 310152 0 0 0 92973 870 0 0 25 0 5 0 597537879 4929744896 1130027 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=4814204 CPUtime=939.77 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 227985 0 0 0 93405 572 0 0 18 0 5 0 597537879 4929744896 1130027 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=4814204 CPUtime=940.15 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 302295 0 0 0 93077 938 0 0 25 0 5 0 597537879 4929744896 1130027 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=4814204 CPUtime=939.13 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 264961 0 0 0 93356 557 0 0 25 0 5 0 597537879 4929744896 1130027 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3757.72
Current children cumulated vsize (KiB) 4814204

[startup+1002.3 s]
/proc/loadavg: 8.02 8.07 7.92 9/186 22723
/proc/meminfo: memFree=12299728/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=4970736 CPUtime=3997.05 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1170031 0 0 0 396597 3108 0 0 25 0 5 0 597537857 5090033664 1169944 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1242684 1169944 175 36 0 1240589 0
[pid=22622/tid=22624] ppid=22620 vsize=4970736 CPUtime=998.36 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 326811 0 0 0 98895 941 0 0 18 0 5 0 597537879 5090033664 1169944 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=4970736 CPUtime=999.58 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 232860 0 0 0 99381 577 0 0 18 0 5 0 597537879 5090033664 1169944 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=4970736 CPUtime=999.79 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 319149 0 0 0 98974 1005 0 0 25 0 5 0 597537879 5090033664 1169944 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=4970736 CPUtime=999.07 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 266490 0 0 0 99326 581 0 0 25 0 5 0 597537879 5090033664 1169944 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3997.05
Current children cumulated vsize (KiB) 4970736

[startup+1062.3 s]
/proc/loadavg: 8.11 8.08 7.93 9/186 22725
/proc/meminfo: memFree=12031508/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=5155288 CPUtime=4236.22 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1217017 0 0 0 420316 3306 0 0 25 0 5 0 597537857 5279014912 1216930 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1288822 1216930 175 36 0 1286727 0
[pid=22622/tid=22624] ppid=22620 vsize=5155288 CPUtime=1058.21 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 343405 0 0 0 104800 1021 0 0 25 0 5 0 597537879 5279014912 1216930 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=5155288 CPUtime=1059.4 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 235590 0 0 0 105350 590 0 0 22 0 5 0 597537879 5279014912 1216930 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=5155288 CPUtime=1059.41 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 338364 0 0 0 104862 1079 0 0 18 0 5 0 597537879 5279014912 1216930 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=5155288 CPUtime=1058.95 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 274937 0 0 0 105283 612 0 0 23 0 5 0 597537879 5279014912 1216930 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4236.22
Current children cumulated vsize (KiB) 5155288

[startup+1122.3 s]
/proc/loadavg: 8.20 8.11 7.95 9/186 22726
/proc/meminfo: memFree=11768736/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=5367992 CPUtime=4475.52 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1269843 0 0 0 444079 3473 0 0 25 0 5 0 597537857 5496823808 1269756 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1341998 1269756 175 36 0 1339903 0
[pid=22622/tid=22624] ppid=22620 vsize=5367992 CPUtime=1117.93 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 364444 0 0 0 110725 1068 0 0 25 0 5 0 597537879 5496823808 1269756 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=5367992 CPUtime=1119.23 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 240879 0 0 0 111295 628 0 0 25 0 5 0 597537879 5496823808 1269756 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=5367992 CPUtime=1119.24 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 355025 0 0 0 110795 1129 0 0 25 0 5 0 597537879 5496823808 1269756 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=5367992 CPUtime=1118.85 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 284774 0 0 0 111242 643 0 0 18 0 5 0 597537879 5496823808 1269756 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427217589 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4475.52
Current children cumulated vsize (KiB) 5367992

[startup+1182.3 s]
/proc/loadavg: 8.11 8.10 7.95 9/186 22728
/proc/meminfo: memFree=11593888/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=5455576 CPUtime=4714.57 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) S 22620 22622 20564 0 -1 4202496 1291435 0 0 0 467753 3704 0 0 25 0 5 0 597537857 5586509824 1291348 33554432000 4194304 4338100 140733246387152 18446744073709551615 256427194341 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 1363894 1291348 175 36 0 1361799 0
[pid=22622/tid=22624] ppid=22620 vsize=5455576 CPUtime=1177.41 cores=0,2,4,6
/proc/22622/task/22624/stat : 22624 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 374016 0 0 0 116585 1156 0 0 25 0 5 0 597537879 5586509824 1291348 33554432000 4194304 4338100 140733246387152 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22622/tid=22625] ppid=22620 vsize=5455576 CPUtime=1179.19 cores=0,2,4,6
/proc/22622/task/22625/stat : 22625 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 242525 0 0 0 117257 662 0 0 25 0 5 0 597537879 5586509824 1291348 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=5455576 CPUtime=1179.08 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 362205 0 0 0 116695 1213 0 0 25 0 5 0 597537879 5586509824 1291348 33554432000 4194304 4338100 140733246387152 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22622/tid=22627] ppid=22620 vsize=5455576 CPUtime=1178.64 cores=0,2,4,6
/proc/22622/task/22627/stat : 22627 (ttsth-5-4) R 22620 22622 20564 0 -1 4202560 287968 0 0 0 117196 668 0 0 18 0 5 0 597537879 5586509824 1291348 33554432000 4194304 4338100 140733246387152 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4714.57
Current children cumulated vsize (KiB) 5455576



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.8 s]
/proc/loadavg: 8.14 8.11 7.96 6/183 22729
/proc/meminfo: memFree=13421080/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=0 CPUtime=4800.1 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) Z 22620 22622 20564 0 -1 4203548 1295167 0 0 0 476232 3778 0 0 18 0 2 0 597537857 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 0 0 0 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=0 CPUtime=1200.57 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4203588 362681 0 0 0 118810 1247 0 0 25 0 2 0 597537879 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.91 s]
/proc/loadavg: 8.14 8.11 7.96 6/183 22729
/proc/meminfo: memFree=13421080/32950928 swapFree=67111524/67111528
[pid=22622] ppid=22620 vsize=0 CPUtime=4800.2 cores=0,2,4,6
/proc/22622/stat : 22622 (ttsth-5-4) Z 22620 22622 20564 0 -1 4203548 1295167 0 0 0 476232 3788 0 0 18 0 2 0 597537857 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22622/statm: 0 0 0 0 0 0 0
[pid=22622/tid=22626] ppid=22620 vsize=0 CPUtime=1200.68 cores=0,2,4,6
/proc/22622/task/22626/stat : 22626 (ttsth-5-4) R 22620 22622 20564 0 -1 4203588 362681 0 0 0 118810 1258 0 0 25 0 2 0 597537879 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.92
CPU time (s): 4800.22
CPU user time (s): 4762.33
CPU system time (s): 37.8952
CPU usage (%): 398.716
Max. virtual memory (cumulated for all children) (KiB): 5471216

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4762.33
system time used= 37.8952
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1295167
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 34445
involuntary context switches= 22812

runsolver used 2.02269 second user time and 4.56731 second system time

The end

Launcher Data 

Begin job on node115 at 2011-04-29 12:31:12
IDJOB=3249900
IDBENCH=83194
IDSOLVER=1654
FILE ID=node115/3249900-1304073072
RUNJOBID= node115-1304071862-22507
PBS_JOBID= 13170430
Free space on /tmp= 72732 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/skvortsov/automata-synchronization/crn_20_361_s.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249900-1304073072/watcher-3249900-1304073072 -o /tmp/evaluation-result-3249900-1304073072/solver-3249900-1304073072 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249900-1304073072.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 6e5ac2b9425e98989eedf4cf638bde54
RANDOM SEED=1587509783

node115.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.62
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      18468876 kB
Buffers:       2320244 kB
Cached:       11541472 kB
SwapCached:          4 kB
Active:        6841876 kB
Inactive:      7086088 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      18468876 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             648 kB
Writeback:           0 kB
AnonPages:       65680 kB
Mapped:          15184 kB
Slab:           490076 kB
PageTables:       4716 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   205744 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72732 MiB
End job on node115 at 2011-04-29 12:51:18