Trace number 3263322

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.41 1202.57

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_20_70_1-dir.shuffled-as.sat03-146.cnf
MD5SUM7ef2b563e57ba91e22d23e55abe69485
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses46988
Sum of the clauses size94536
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 246968
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 520

Solver Data 

0.00/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263322-1304096194.cnf 
0.00/0.03	c Using SATTIMEOUT of 4800
0.00/0.03	c Using SATRAM of 15500
0.00/0.03	c Size of subset node = 32
0.00/0.03	c Variables = 600
0.00/0.03	c Clauses = 46988
0.00/0.03	c Literals = 94536
0.00/0.03	c Reduced size:
0.00/0.03	c Variables = 600
0.00/0.03	c Clauses = 46988
0.00/0.03	c Literals = 94536
0.00/0.03	c a2
0.04/0.06	c a1
0.09/0.12	c Independent sub-problems = 1
0.09/0.12	c Sub-problem with 600 variables and 46988 clauses
0.09/0.12	c using permutation method 4
0.09/0.12	c count = 1
0.09/0.12	c Ordering finished 0.110 seconds
0.09/0.12	c mapped 0.110 seconds
0.09/0.12	c permuted 0.110 seconds
0.09/0.12	c about to sort
0.09/0.12	c sorted
0.19/0.23	c clauses now 46988
0.19/0.23	c init finished 0.130 seconds
0.19/0.23	c tree built 0.230 seconds
0.19/0.23	c 
0.19/0.23	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.23	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.23	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.23	c ................................ | ..........                  1 |                                  0.2         4.3             
0.19/0.23	c ................................ | ..........                  2 |                                  0.2         8.7             
0.19/0.23	c ................................ | ..........                  3 |                                  0.2        13.0             
0.19/0.23	c ................................ | ..........                  4 |                                  0.2        17.4             
0.19/0.23	c ................................ | ..........                  5 |                                  0.2        21.7             
0.19/0.24	c ................................ | ..........               1000 | 1            17                  0.2      4347.8          815
95.94/24.21	c ................................ | ..........            4269000 | 123524       460556             96.0     44468.8     10462335
191.77/48.26	c ................................ | ..........            7318000 | 232000       754156            192.0     38110.6     16682860
287.99/72.30	c ................................ | ..........            9287000 | 303717       1018047           288.0     32245.4     22849731
383.82/96.35	c ................................ | ..........           11774000 | 400016       1246126           384.1     30657.5     27652371
479.67/120.39	c ................................ | *.........           13121000 | 453845       1449282           480.1     27332.0     31937024
575.92/144.43	c ................................ | *.........           14779000 | 510385       1616974           576.0     25656.2     33855730
671.77/168.46	c ................................ | *.........           16101000 | 568183       1785597           672.0     23958.4     36341886
767.65/192.49	c ................................ | *.........           17013000 | 607764       1947859           768.0     22152.1     40477909
863.88/216.53	c ................................ | *.........           18117000 | 652001       2125068           864.0     20968.8     45756629
959.69/240.57	c ................................ | **........           19398000 | 693431       2301366           960.0     20206.2     49558111
1055.93/264.62	c ................................ | **........           20825000 | 758463       2450002          1056.0     19719.9     50878932
1151.76/288.68	c ................................ | **........           22163000 | 817647       2613067          1152.1     19237.2     53772234
1248.00/312.72	c ................................ | **........           23022000 | 856762       2813021          1248.1     18445.6     58401039
1343.83/336.76	c ................................ | **........           24055000 | 900555       3020832          1344.1     17897.3     62595304
1439.70/360.79	c ................................ | ***.......           25185000 | 949348       3156689          1440.0     17489.1     65770681
1535.94/384.83	c ................................ | ***.......           26147000 | 997436       3267390          1536.1     17022.1     69145445
1631.79/408.85	c ................................ | ***.......           26904000 | 1027902      3391598          1632.0     16485.1     73518181
1727.62/432.90	c ................................ | ***.......           27558000 | 1058215      3515145          1728.0     15947.8     77561263
1823.84/456.94	c ................................ | ***.......           28148000 | 1088101      3641587          1824.0     15431.8     80358980
1919.72/481.18	c ................................ | ****......           28904000 | 1119523      3749094          1920.0     15053.9     80751836
2015.94/505.21	c ................................ | ****......           30004000 | 1160992      3856382          2016.0     14882.9     81347426
2111.81/529.25	c ................................ | ****......           30965000 | 1206368      3952806          2112.0     14661.3     82982415
2208.08/553.33	c ................................ | ****......           31621000 | 1238093      4034991          2208.2     14319.7     84523093
2303.93/577.32	c ................................ | ****......           32203000 | 1267671      4132774          2304.0     13976.8     85955542
2399.76/601.37	c ................................ | *****.....           33261000 | 1305192      4255080          2400.1     13858.3     87289091
2495.99/625.42	c ................................ | *****.....           34316000 | 1350768      4375285          2496.1     13747.9     87938002
2591.86/649.44	c ................................ | *****.....           35049000 | 1386401      4490146          2592.1     13521.7     88820754
2687.62/673.49	c ................................ | *****.....           36244000 | 1443051      4592966          2688.0     13483.6     88990493
2783.84/697.55	c ................................ | *****.....           37404000 | 1509536      4697792          2784.0     13435.2     89502596
2880.11/721.60	c ................................ | ******....           38397000 | 1571782      4771339          2880.1     13331.8     90451809
2975.93/745.64	c ................................ | ******....           39099000 | 1623165      4846569          2976.1     13137.5     91431011
3071.77/769.67	c ................................ | ******....           40132000 | 1669132      4922330          3072.1     13063.6     93074729
3167.99/793.71	c ................................ | ******....           41432000 | 1715057      5033557          3168.0     13078.2     94884438
3264.20/817.84	c ................................ | ******....           42101000 | 1746657      5129802          3264.4     12897.1     97211656
3360.06/841.82	c ................................ | *******...           42854000 | 1778641      5238478          3360.1     12753.6     97415716
3455.89/865.89	c ................................ | *******...           43411000 | 1801658      5325602          3456.2     12560.1     98111865
3551.78/889.86	c ................................ | *******...           44440000 | 1842061      5420848          3552.0     12511.2     98869099
3647.63/913.90	c ................................ | *******...           45326000 | 1880408      5509208          3648.0     12424.8    100503399
3743.83/937.97	c ................................ | *******...           45840000 | 1906670      5598938          3744.1     12243.2    101427958
3840.09/962.02	c ................................ | ********..           46466000 | 1939613      5680377          3840.2     12100.0    102465068
3935.90/986.08	c ................................ | ********..           47186000 | 1973369      5779018          3936.2     11987.6    104388585
4031.77/1010.07	c ................................ | ********..           47837000 | 2007055      5855391          4032.1     11864.2    106073303
4128.04/1034.11	c ................................ | ********..           48369000 | 2035273      5944566          4128.1     11717.0    106992841
4223.89/1058.14	c ................................ | ********..           49132000 | 2070787      6033272          4224.0     11631.5    109441201
4319.74/1082.20	c ................................ | *********.           49790000 | 2102507      6126572          4320.1     11525.1    110443402
4416.00/1106.24	c ................................ | *********.           50450000 | 2137192      6219007          4416.2     11424.0    112047178
4511.86/1130.24	c ................................ | *********.           50895000 | 2161359      6301068          4512.0     11279.8    114024521
4608.08/1154.34	c ................................ | *********.           51311000 | 2183594      6384043          4608.3     11134.6    115830688
4703.94/1178.32	c ................................ | *********.           51624000 | 2202826      6457733          4704.1     10974.4    117426352

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263322-1304096194/watcher-3263322-1304096194 -o /tmp/evaluation-result-3263322-1304096194/solver-3263322-1304096194 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263322-1304096194.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.69 7.71 7.83 5/181 16511
/proc/meminfo: memFree=24876296/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=9756 CPUtime=0 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) R 16509 16511 16318 0 -1 4202496 487 0 0 0 0 0 0 0 25 0 1 0 521218140 9990144 429 33554432000 4194304 4338100 140737106852000 18446744073709551615 251813937360 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/16511/statm: 2439 429 147 36 0 344 0

[startup+0.046345 s]
/proc/loadavg: 7.69 7.71 7.83 5/181 16511
/proc/meminfo: memFree=24876296/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=18336 CPUtime=0.04 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) R 16509 16511 16318 0 -1 4202496 2629 0 0 0 4 0 0 0 25 0 1 0 521218140 18776064 2570 33554432000 4194304 4338100 140737106852000 18446744073709551615 251813635288 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/16511/statm: 4584 2570 153 36 0 2489 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 18336

[startup+0.100339 s]
/proc/loadavg: 7.69 7.71 7.83 5/181 16511
/proc/meminfo: memFree=24876296/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=19168 CPUtime=0.09 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) R 16509 16511 16318 0 -1 4202496 3846 0 0 0 9 0 0 0 25 0 1 0 521218140 19628032 2787 33554432000 4194304 4338100 140737106852000 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/16511/statm: 4792 2787 155 36 0 2697 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19168

[startup+0.300304 s]
/proc/loadavg: 7.69 7.71 7.83 5/181 16511
/proc/meminfo: memFree=24876296/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=159408 CPUtime=0.47 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 10112 0 0 0 46 1 0 0 25 0 5 0 521218140 163233792 8868 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 39852 8868 176 36 0 37757 0
Current children cumulated CPU time (s) 0.47
Current children cumulated vsize (KiB) 159408

[startup+0.700237 s]
/proc/loadavg: 7.69 7.71 7.83 5/181 16511
/proc/meminfo: memFree=24876296/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=234592 CPUtime=2.07 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 13237 0 0 0 205 2 0 0 25 0 5 0 521218140 240222208 11993 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 58648 11993 176 36 0 56553 0
Current children cumulated CPU time (s) 2.07
Current children cumulated vsize (KiB) 234592

[startup+1.5011 s]
/proc/loadavg: 7.69 7.71 7.83 9/186 16516
/proc/meminfo: memFree=24818108/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=256884 CPUtime=5.26 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 18590 0 0 0 522 4 0 0 25 0 5 0 521218140 263049216 17346 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 64221 17346 176 36 0 62126 0
[pid=16511/tid=16513] ppid=16509 vsize=256884 CPUtime=1.26 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 1012 0 0 0 125 1 0 0 25 0 5 0 521218164 263049216 17347 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=256884 CPUtime=1.25 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 2824 0 0 0 125 0 0 0 25 0 5 0 521218164 263049216 17348 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=256884 CPUtime=1.24 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 2508 0 0 0 123 1 0 0 25 0 5 0 521218164 263049216 17348 33554432000 4194304 4338100 140737106852000 18446744073709551615 251813635155 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=256884 CPUtime=1.25 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 3045 0 0 0 125 0 0 0 25 0 5 0 521218164 263049216 17348 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.26
Current children cumulated vsize (KiB) 256884

[startup+3.10083 s]
/proc/loadavg: 7.71 7.72 7.83 9/186 16516
/proc/meminfo: memFree=24788968/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=285828 CPUtime=11.67 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 27393 0 0 0 1155 12 0 0 25 0 5 0 521218140 292687872 26149 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 71457 26150 176 36 0 69362 0
[pid=16511/tid=16513] ppid=16509 vsize=285828 CPUtime=2.85 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 2435 0 0 0 281 4 0 0 25 0 5 0 521218164 292687872 26150 33554432000 4194304 4338100 140737106852000 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=285828 CPUtime=2.85 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 5540 0 0 0 284 1 0 0 25 0 5 0 521218164 292687872 26150 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=285828 CPUtime=2.84 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 4726 0 0 0 280 4 0 0 25 0 5 0 521218164 292687872 26150 33554432000 4194304 4338100 140737106852000 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=285828 CPUtime=2.85 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 5492 0 0 0 285 0 0 0 25 0 5 0 521218164 292687872 26151 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.67
Current children cumulated vsize (KiB) 285828

[startup+6.3003 s]
/proc/loadavg: 7.71 7.72 7.83 9/186 16516
/proc/meminfo: memFree=24710984/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=346184 CPUtime=24.44 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 41965 0 0 0 2415 29 0 0 25 0 5 0 521218140 354492416 40721 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 86546 40721 176 36 0 84451 0
[pid=16511/tid=16513] ppid=16509 vsize=346184 CPUtime=6.05 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 4653 0 0 0 594 11 0 0 25 0 5 0 521218164 354492416 40721 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=346184 CPUtime=6.05 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 10365 0 0 0 603 2 0 0 25 0 5 0 521218164 354492416 40722 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=346184 CPUtime=6.02 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 7643 0 0 0 591 11 0 0 25 0 5 0 521218164 354492416 40722 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=346184 CPUtime=6.05 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 10103 0 0 0 604 1 0 0 25 0 5 0 521218164 354492416 40722 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.44
Current children cumulated vsize (KiB) 346184

[startup+12.7002 s]
/proc/loadavg: 7.83 7.74 7.84 9/186 16516
/proc/meminfo: memFree=24564372/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=444888 CPUtime=50 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 66463 0 0 0 4935 65 0 0 25 0 5 0 521218140 455565312 65219 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 111222 65219 176 36 0 109127 0
[pid=16511/tid=16513] ppid=16509 vsize=444888 CPUtime=12.45 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 8463 0 0 0 1217 28 0 0 25 0 5 0 521218164 455565312 65219 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=444888 CPUtime=12.45 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 18175 0 0 0 1241 4 0 0 25 0 5 0 521218164 455565312 65219 33554432000 4194304 4338100 140737106852000 18446744073709551615 4286307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=444888 CPUtime=12.38 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 12250 0 0 0 1212 26 0 0 25 0 5 0 521218164 455565312 65219 33554432000 4194304 4338100 140737106852000 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=444888 CPUtime=12.46 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 18373 0 0 0 1242 4 0 0 25 0 5 0 521218164 455565312 65220 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50
Current children cumulated vsize (KiB) 444888

[startup+25.5011 s]
/proc/loadavg: 7.86 7.75 7.84 9/186 16516
/proc/meminfo: memFree=24359408/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=565236 CPUtime=101.11 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 96901 0 0 0 9979 132 0 0 25 0 5 0 521218140 578801664 95657 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 141309 95657 176 36 0 139214 0
[pid=16511/tid=16513] ppid=16509 vsize=565236 CPUtime=25.25 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 14158 0 0 0 2462 63 0 0 25 0 5 0 521218164 578801664 95657 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=565236 CPUtime=25.24 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 27073 0 0 0 2518 6 0 0 25 0 5 0 521218164 578801664 95657 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=565236 CPUtime=25.1 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 18019 0 0 0 2456 54 0 0 25 0 5 0 521218164 578801664 95657 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=565236 CPUtime=25.24 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 28449 0 0 0 2518 6 0 0 25 0 5 0 521218164 578801664 95657 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 101.11
Current children cumulated vsize (KiB) 565236

[startup+51.1008 s]
/proc/loadavg: 7.87 7.75 7.84 9/186 16517
/proc/meminfo: memFree=24006332/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=780712 CPUtime=203.35 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 152238 0 0 0 20062 273 0 0 25 0 5 0 521218140 799449088 150994 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 195178 150994 176 36 0 193083 0
[pid=16511/tid=16513] ppid=16509 vsize=780712 CPUtime=50.85 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 25975 0 0 0 4955 130 0 0 25 0 5 0 521218164 799449088 150994 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=780712 CPUtime=50.83 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 42508 0 0 0 5072 11 0 0 25 0 5 0 521218164 799449088 150994 33554432000 4194304 4338100 140737106852000 18446744073709551615 4289342 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=780712 CPUtime=50.57 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 30804 0 0 0 4938 119 0 0 25 0 5 0 521218164 799449088 150994 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=780712 CPUtime=50.84 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 43749 0 0 0 5074 10 0 0 25 0 5 0 521218164 799449088 150994 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.35
Current children cumulated vsize (KiB) 780712

[startup+102.306 s]
/proc/loadavg: 7.67 7.73 7.82 9/186 16518
/proc/meminfo: memFree=23369180/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=1137836 CPUtime=407.8 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 242989 0 0 0 40274 506 0 0 25 0 5 0 521218140 1165144064 241745 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 284459 241745 176 36 0 282364 0
[pid=16511/tid=16513] ppid=16509 vsize=1137836 CPUtime=102.03 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 45163 0 0 0 9962 241 0 0 25 0 5 0 521218164 1165144064 241746 33554432000 4194304 4338100 140737106852000 18446744073709551615 4284123 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=1137836 CPUtime=102.02 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 65183 0 0 0 10183 19 0 0 25 0 5 0 521218164 1165144064 241746 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=1137836 CPUtime=101.46 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 50625 0 0 0 9922 224 0 0 25 0 5 0 521218164 1165144064 241746 33554432000 4194304 4338100 140737106852000 18446744073709551615 4284181 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=1137836 CPUtime=102.03 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 72817 0 0 0 10184 19 0 0 25 0 5 0 521218164 1165144064 241746 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.8
Current children cumulated vsize (KiB) 1137836

[startup+162.301 s]
/proc/loadavg: 7.93 7.79 7.83 9/186 16520
/proc/meminfo: memFree=22782380/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=1388520 CPUtime=647.41 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 303707 0 0 0 64012 729 0 0 25 0 5 0 521218140 1421844480 302463 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 347130 302463 176 36 0 345035 0
[pid=16511/tid=16513] ppid=16509 vsize=1388520 CPUtime=162.01 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 54641 0 0 0 15851 350 0 0 25 0 5 0 521218164 1421844480 302463 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=1388520 CPUtime=162 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 83571 0 0 0 16175 25 0 0 25 0 5 0 521218164 1421844480 302463 33554432000 4194304 4338100 140737106852000 18446744073709551615 4279490 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=1388520 CPUtime=161.12 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 61062 0 0 0 15788 324 0 0 25 0 5 0 521218164 1421844480 302463 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=1388520 CPUtime=162.01 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 95231 0 0 0 16174 27 0 0 25 0 5 0 521218164 1421844480 302463 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 647.41
Current children cumulated vsize (KiB) 1388520


################
# More data... #
################

/proc/meminfo: memFree=25770460/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=3221276 CPUtime=3042.63 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 772432 0 0 0 301891 2372 0 0 25 0 5 0 521218140 3298586624 771188 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 805319 771188 176 36 0 803224 0
[pid=16511/tid=16513] ppid=16509 vsize=3221276 CPUtime=761.06 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 94797 0 0 0 74984 1122 0 0 25 0 5 0 521218164 3298586624 771189 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=3221276 CPUtime=760.72 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 351451 0 0 0 75958 114 0 0 25 0 5 0 521218164 3298586624 771189 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=3221276 CPUtime=759 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 107227 0 0 0 74835 1065 0 0 25 0 5 0 521218164 3298586624 771189 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=3221276 CPUtime=761.58 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 209756 0 0 0 76089 69 0 0 25 0 5 0 521218164 3298586624 771189 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3042.63
Current children cumulated vsize (KiB) 3221276

[startup+822.3 s]
/proc/loadavg: 8.07 8.02 7.92 9/186 16582
/proc/meminfo: memFree=25458084/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=3376924 CPUtime=3282.18 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 812050 0 0 0 325691 2527 0 0 25 0 5 0 521218140 3457970176 810806 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 844231 810806 176 36 0 842136 0
[pid=16511/tid=16513] ppid=16509 vsize=3376924 CPUtime=820.82 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 102048 0 0 0 80885 1197 0 0 25 0 5 0 521218164 3457970176 810806 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=3376924 CPUtime=820.72 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 362789 0 0 0 81953 119 0 0 25 0 5 0 521218164 3457970176 810806 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=3376924 CPUtime=818.83 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 110103 0 0 0 80750 1133 0 0 25 0 5 0 521218164 3457970176 810806 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=3376924 CPUtime=821.55 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 227908 0 0 0 82079 76 0 0 25 0 5 0 521218164 3457970176 810806 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3282.18
Current children cumulated vsize (KiB) 3376924

[startup+882.3 s]
/proc/loadavg: 8.07 8.03 7.92 9/186 16583
/proc/meminfo: memFree=25205348/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=3410316 CPUtime=3521.8 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 821295 0 0 0 349524 2656 0 0 25 0 5 0 521218140 3492163584 820051 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 852579 820051 176 36 0 850484 0
[pid=16511/tid=16513] ppid=16509 vsize=3410316 CPUtime=880.81 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 102358 0 0 0 86822 1259 0 0 25 0 5 0 521218164 3492163584 820051 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=3410316 CPUtime=880.7 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 365764 0 0 0 87950 120 0 0 25 0 5 0 521218164 3492163584 820051 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=3410316 CPUtime=878.77 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 110387 0 0 0 86682 1195 0 0 25 0 5 0 521218164 3492163584 820051 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=3410316 CPUtime=881.27 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 233584 0 0 0 88047 80 0 0 25 0 5 0 521218164 3492163584 820051 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3521.8
Current children cumulated vsize (KiB) 3410316

[startup+942.301 s]
/proc/loadavg: 8.03 8.03 7.92 9/186 16585
/proc/meminfo: memFree=24957200/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=3518648 CPUtime=3761.41 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 848492 0 0 0 373364 2777 0 0 25 0 5 0 521218140 3603095552 847248 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 879662 847248 176 36 0 877567 0
[pid=16511/tid=16513] ppid=16509 vsize=3518648 CPUtime=940.65 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 104911 0 0 0 92750 1315 0 0 25 0 5 0 521218164 3603095552 847248 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=3518648 CPUtime=940.69 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 378318 0 0 0 93944 125 0 0 25 0 5 0 521218164 3603095552 847248 33554432000 4194304 4338100 140737106852000 18446744073709551615 4253888 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=3518648 CPUtime=938.74 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 112625 0 0 0 92623 1251 0 0 25 0 5 0 521218164 3603095552 847248 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=3518648 CPUtime=941.05 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 243436 0 0 0 94022 83 0 0 25 0 5 0 521218164 3603095552 847248 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3761.41
Current children cumulated vsize (KiB) 3518648

[startup+1002.3 s]
/proc/loadavg: 8.09 8.04 7.93 9/186 16586
/proc/meminfo: memFree=24724048/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=3645616 CPUtime=4001.02 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 882244 0 0 0 397184 2918 0 0 25 0 5 0 521218140 3733110784 881000 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 911404 881000 176 36 0 909309 0
[pid=16511/tid=16513] ppid=16509 vsize=3645616 CPUtime=1000.29 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 109648 0 0 0 98648 1381 0 0 25 0 5 0 521218164 3733110784 881001 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=3645616 CPUtime=1000.68 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 392762 0 0 0 99936 132 0 0 25 0 5 0 521218164 3733110784 881001 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=3645616 CPUtime=998.72 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 116947 0 0 0 98559 1313 0 0 25 0 5 0 521218164 3733110784 881001 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=3645616 CPUtime=1001.05 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 253686 0 0 0 100016 89 0 0 25 0 5 0 521218164 3733110784 881001 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4001.02
Current children cumulated vsize (KiB) 3645616

[startup+1062.3 s]
/proc/loadavg: 8.10 8.04 7.93 9/186 16588
/proc/meminfo: memFree=24440052/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=3778576 CPUtime=4240.65 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 916001 0 0 0 421022 3043 0 0 25 0 5 0 521218140 3869261824 914757 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 944644 914757 176 36 0 942549 0
[pid=16511/tid=16513] ppid=16509 vsize=3778576 CPUtime=1059.95 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 113611 0 0 0 104556 1439 0 0 25 0 5 0 521218164 3869261824 914757 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=3778576 CPUtime=1060.67 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 406209 0 0 0 105930 137 0 0 25 0 5 0 521218164 3869261824 914757 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=3778576 CPUtime=1058.71 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 122424 0 0 0 104499 1372 0 0 25 0 5 0 521218164 3869261824 914757 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=3778576 CPUtime=1061.05 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 264555 0 0 0 106012 93 0 0 25 0 5 0 521218164 3869261824 914757 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4240.65
Current children cumulated vsize (KiB) 3778576

[startup+1122.3 s]
/proc/loadavg: 8.10 8.05 7.94 9/186 16590
/proc/meminfo: memFree=24223512/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=3899288 CPUtime=4480.31 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 946478 0 0 0 444879 3152 0 0 25 0 5 0 521218140 3992870912 945234 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 974822 945234 176 36 0 972727 0
[pid=16511/tid=16513] ppid=16509 vsize=3899288 CPUtime=1119.61 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 117108 0 0 0 110471 1490 0 0 25 0 5 0 521218164 3992870912 945234 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=3899288 CPUtime=1120.68 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 415898 0 0 0 111926 142 0 0 25 0 5 0 521218164 3992870912 945234 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=3899288 CPUtime=1118.7 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 128299 0 0 0 110451 1419 0 0 25 0 5 0 521218164 3992870912 945234 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=3899288 CPUtime=1121.04 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 275971 0 0 0 112006 98 0 0 25 0 5 0 521218164 3992870912 945234 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4480.31
Current children cumulated vsize (KiB) 3899288

[startup+1182.3 s]
/proc/loadavg: 8.10 8.05 7.94 9/186 16591
/proc/meminfo: memFree=23976236/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=4022864 CPUtime=4719.92 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 979334 0 0 0 468760 3232 0 0 25 0 5 0 521218140 4119412736 978090 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 1005716 978090 176 36 0 1003621 0
[pid=16511/tid=16513] ppid=16509 vsize=4022864 CPUtime=1179.27 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 117648 0 0 0 116403 1524 0 0 25 0 5 0 521218164 4119412736 978090 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=4022864 CPUtime=1180.67 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 430457 0 0 0 117919 148 0 0 25 0 5 0 521218164 4119412736 978090 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=4022864 CPUtime=1178.7 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 129829 0 0 0 116416 1454 0 0 25 0 5 0 521218164 4119412736 978090 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=4022864 CPUtime=1181.03 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 292198 0 0 0 117998 105 0 0 25 0 5 0 521218164 4119412736 978090 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4719.92
Current children cumulated vsize (KiB) 4022864



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.4 s]
/proc/loadavg: 8.07 8.05 7.94 9/186 16592
/proc/meminfo: memFree=23879764/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=4073572 CPUtime=4800.22 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) S 16509 16511 16318 0 -1 4202496 991382 0 0 0 476762 3260 0 0 25 0 5 0 521218140 4171337728 990138 33554432000 4194304 4338100 140737106852000 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 1018393 990138 176 36 0 1016298 0
[pid=16511/tid=16513] ppid=16509 vsize=4073572 CPUtime=1199.26 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 118327 0 0 0 118391 1535 0 0 25 0 5 0 521218164 4171337728 990139 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16511/tid=16514] ppid=16509 vsize=4073572 CPUtime=1200.77 cores=1,3,5,7
/proc/16511/task/16514/stat : 16514 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 436021 0 0 0 119927 150 0 0 25 0 5 0 521218164 4171337728 990139 33554432000 4194304 4338100 140737106852000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16511/tid=16515] ppid=16509 vsize=4073572 CPUtime=1198.79 cores=1,3,5,7
/proc/16511/task/16515/stat : 16515 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 130178 0 0 0 118414 1465 0 0 25 0 5 0 521218164 4171337728 990139 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16511/tid=16516] ppid=16509 vsize=4073572 CPUtime=1201.13 cores=1,3,5,7
/proc/16511/task/16516/stat : 16516 (ttsth-5-4) R 16509 16511 16318 0 -1 4202560 297655 0 0 0 120006 107 0 0 25 0 5 0 521218164 4171337728 990139 33554432000 4194304 4338100 140737106852000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 4073572

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.51 s]
/proc/loadavg: 8.07 8.05 7.94 9/186 16592
/proc/meminfo: memFree=23879764/32950928 swapFree=67111448/67111528
[pid=16511] ppid=16509 vsize=0 CPUtime=4800.34 cores=1,3,5,7
/proc/16511/stat : 16511 (ttsth-5-4) Z 16509 16511 16318 0 -1 4203548 991383 0 0 0 476764 3270 0 0 18 0 2 0 521218140 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/16511/statm: 0 0 0 0 0 0 0
[pid=16511/tid=16513] ppid=16509 vsize=0 CPUtime=1199.37 cores=1,3,5,7
/proc/16511/task/16513/stat : 16513 (ttsth-5-4) R 16509 16511 16318 0 -1 4203588 118327 0 0 0 118392 1545 0 0 25 0 2 0 521218164 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.57
CPU time (s): 4800.41
CPU user time (s): 4767.65
CPU system time (s): 32.763
CPU usage (%): 399.18
Max. virtual memory (cumulated for all children) (KiB): 4073572

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4767.65
system time used= 32.763
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 991383
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6599
involuntary context switches= 19792

runsolver used 2.00069 second user time and 4.5693 second system time

The end

Launcher Data 

Begin job on node126 at 2011-04-29 18:56:34
IDJOB=3263322
IDBENCH=83952
IDSOLVER=1654
FILE ID=node126/3263322-1304096194
RUNJOBID= node126-1304094101-16337
PBS_JOBID= 13170590
Free space on /tmp= 71596 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_20_70_1-dir.shuffled-as.sat03-146.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263322-1304096194/watcher-3263322-1304096194 -o /tmp/evaluation-result-3263322-1304096194/solver-3263322-1304096194 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263322-1304096194.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 7ef2b563e57ba91e22d23e55abe69485
RANDOM SEED=313827340

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      24876576 kB
Buffers:        246772 kB
Cached:        1398844 kB
SwapCached:         76 kB
Active:        6158004 kB
Inactive:      1593972 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      24876576 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:             448 kB
Writeback:           0 kB
AnonPages:     6106244 kB
Mapped:          15248 kB
Slab:           246572 kB
PageTables:      16724 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  6558608 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71596 MiB
End job on node126 at 2011-04-29 19:16:39