Trace number 3249196

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.07 1203.53

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-13x32.cnf
MD5SUM73c8ae5bbdfe41463e8e40ea211c5e14
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1664
Number of clauses157664
Sum of the clauses size625664
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22496
Number of clauses of size 30
Number of clauses of size 4155168
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.09/0.17	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249196-1304060966.cnf 
0.09/0.17	c Using SATTIMEOUT of 4800
0.09/0.17	c Using SATRAM of 15500
0.09/0.17	c Size of subset node = 32
0.09/0.17	c Variables = 1664
0.09/0.17	c Clauses = 157664
0.09/0.17	c Literals = 625664
0.09/0.17	c Reduced size:
0.09/0.17	c Variables = 1664
0.09/0.17	c Clauses = 157664
0.09/0.17	c Literals = 625664
0.09/0.17	c a2
0.39/0.48	c a1
1.09/1.18	c Independent sub-problems = 1
1.09/1.18	c Sub-problem with 1664 variables and 157664 clauses
1.09/1.18	c using permutation method 4
1.09/1.18	c count = 1
1.09/1.18	c Ordering finished 1.140 seconds
1.09/1.18	c mapped 1.170 seconds
1.09/1.18	c permuted 1.170 seconds
1.09/1.18	c about to sort
1.19/1.20	c sorted
1.49/1.50	c clauses now 157664
1.49/1.50	c init finished 1.220 seconds
1.49/1.50	c tree built 1.500 seconds
1.49/1.50	c 
1.49/1.50	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.49/1.50	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.49/1.50	c ................................ | ..........                    |                                  1.5         0.0             
1.49/1.50	c ................................ | ..........                  1 |              14                  1.5         0.7             
1.49/1.50	c ................................ | ..........                  2 |              15                  1.5         1.3             
1.49/1.50	c ................................ | ..........                  3 |              15                  1.5         2.0             
1.49/1.50	c ................................ | ..........                  4 |              15                  1.5         2.7             
1.49/1.50	c ................................ | ..........                  5 |              15                  1.5         3.3             
1.49/1.51	c ................................ | ..........               1000 | 15           245                 1.5       653.6         1416
95.68/25.19	c ................................ | ..........            2281000 | 116842       781978             96.0     23753.0      2947484
191.90/49.24	c ................................ | ..........            4107000 | 205727       1408868           192.1     21382.8      5191527
287.72/73.28	c ................................ | ..........            5804000 | 310418       1881021           288.1     20149.3      6693859
383.95/97.31	c ................................ | ..........            7362000 | 396025       2337243           384.0     19171.9      8200745
479.75/121.36	c ................................ | *.........            8713000 | 482939       2696461           480.0     18151.7      9914324
575.97/145.41	c ................................ | *.........            9925000 | 547887       3070980           576.0     17230.3     11440816
671.80/169.47	c ................................ | *.........           11110000 | 616293       3389045           672.1     16530.8     12572676
768.01/193.51	c ................................ | *.........           12229000 | 674861       3669038           768.0     15922.1     13770719
863.79/217.56	c ................................ | *.........           13298000 | 742426       3921904           864.0     15390.8     15383315
959.99/241.60	c ................................ | **........           14295000 | 794309       4178173           960.0     14890.5     16421164
1055.80/265.65	c ................................ | **........           15227000 | 843882       4407158          1056.0     14419.5     17375081
1151.99/289.71	c ................................ | **........           16277000 | 906306       4670286          1152.1     14128.6     18578038
1247.81/313.76	c ................................ | **........           17313000 | 956399       4958074          1248.1     13871.9     19498623
1344.01/337.81	c ................................ | **........           18085000 | 998133       5148187          1344.1     13455.3     19955229
1439.84/361.87	c ................................ | ***.......           19022000 | 1057663      5361071          1440.1     13208.5     20317318
1536.02/385.92	c ................................ | ***.......           19941000 | 1104419      5599414          1536.1     12981.6     20766575
1631.82/409.94	c ................................ | ***.......           20811000 | 1147711      5826073          1632.0     12751.8     21566065
1727.65/433.99	c ................................ | ***.......           21618000 | 1190717      6021289          1728.0     12510.2     22426697
1823.86/458.05	c ................................ | ***.......           22362000 | 1229803      6197381          1824.1     12259.4     22759914
1919.67/482.10	c ................................ | ****......           23238000 | 1272937      6399104          1920.1     12102.7     23320542
2015.85/506.15	c ................................ | ****......           24334000 | 1328966      6708123          2016.1     12069.9     24608507
2111.68/530.19	c ................................ | ****......           25334000 | 1382575      6962628          2112.0     11995.2     25753622
2207.86/554.24	c ................................ | ****......           26398000 | 1441467      7235789          2208.0     11955.4     26901370
2303.67/578.28	c ................................ | ****......           27345000 | 1494970      7485547          2304.0     11868.5     28063030
2399.87/602.33	c ................................ | *****.....           28297000 | 1543552      7721245          2400.0     11790.4     28530429
2495.69/626.38	c ................................ | *****.....           29271000 | 1596193      7941358          2496.0     11727.0     29038727
2591.91/650.42	c ................................ | *****.....           30214000 | 1643441      8167752          2592.0     11656.5     29455780
2687.72/674.48	c ................................ | *****.....           31169000 | 1690731      8412327          2688.1     11595.3     29816063
2783.94/698.53	c ................................ | *****.....           32148000 | 1742107      8653748          2784.0     11547.2     30793568
2879.74/722.59	c ................................ | ******....           33131000 | 1795411      8909840          2880.1     11503.4     30994499
2975.97/746.63	c ................................ | ******....           34000000 | 1846264      9117008          2976.1     11424.4     31140150
3071.80/770.67	c ................................ | ******....           34839000 | 1894000      9316026          3072.1     11340.6     31340223
3168.01/794.71	c ................................ | ******....           35701000 | 1940307      9521239          3168.1     11269.0     31645856
3263.84/818.74	c ................................ | ******....           36521000 | 1983856      9715595          3264.0     11189.0     32024713
3360.05/842.82	c ................................ | *******...           37245000 | 2017141      9919003          3360.1     11084.4     32302832
3455.88/866.85	c ................................ | *******...           37885000 | 2049422      10074731         3456.1     10961.8     32372170
3552.09/890.90	c ................................ | *******...           38434000 | 2075510      10207424         3552.1     10820.1     32762106
3647.88/914.97	c ................................ | *******...           39044000 | 2104222      10354626         3648.2     10702.3     33168501
3744.08/939.00	c ................................ | *******...           39629000 | 2139873      10496196         3744.1     10584.4     33457157
3840.31/963.10	c ................................ | ********..           40345000 | 2181829      10697594         3840.3     10505.6     33897224
3935.71/987.08	c ................................ | ********..           40819000 | 2210761      10817018         3936.0     10370.6     34278121
4032.33/1011.20	c ................................ | ********..           41285000 | 2233124      10947146         4032.4     10238.4     34617432
4128.55/1035.32	c ................................ | ********..           41797000 | 2261628      11091677         4128.6     10123.7     34868889
4224.02/1059.23	c ................................ | ********..           42252000 | 2285858      11208339         4224.1     10002.5     34879605
4319.82/1083.26	c ................................ | *********.           42718000 | 2312642      11314535         4320.1      9888.2     35079765
4416.04/1107.32	c ................................ | *********.           43195000 | 2336467      11423502         4416.1      9781.2     35319063
4512.26/1131.49	c ................................ | *********.           43710000 | 2361507      11546336         4512.6      9686.1     35750898
4607.69/1155.38	c ................................ | *********.           44215000 | 2387846      11661608         4608.0      9595.2     36208510
4703.90/1179.43	c ................................ | *********.           44817000 | 2424822      11804651         4704.1      9527.3     36664021
4799.75/1203.46	c CPU limit (4800) reached
4799.75/1203.46	c Time = 4800.00 seconds
4799.75/1203.46	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249196-1304060966/watcher-3249196-1304060966 -o /tmp/evaluation-result-3249196-1304060966/solver-3249196-1304060966 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249196-1304060966.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.91 7.22 7.45 2/177 15752
/proc/meminfo: memFree=17911332/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) R 15750 15752 12737 0 -1 4202496 415 0 0 0 0 0 0 0 21 0 1 0 517699272 9633792 358 33554432000 4194304 4338100 140733230247552 18446744073709551615 216634176197 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/15752/statm: 2352 359 147 36 0 257 0

[startup+0.0744551 s]
/proc/loadavg: 4.91 7.22 7.45 2/177 15752
/proc/meminfo: memFree=17911332/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=18004 CPUtime=0.07 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) R 15750 15752 12737 0 -1 4202496 2556 0 0 0 7 0 0 0 21 0 1 0 517699272 18436096 2499 33554432000 4194304 4338100 140733230247552 18446744073709551615 216634515200 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/15752/statm: 4501 2500 148 36 0 2406 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 18004

[startup+0.100461 s]
/proc/loadavg: 4.91 7.22 7.45 2/177 15752
/proc/meminfo: memFree=17911332/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=21268 CPUtime=0.09 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) R 15750 15752 12737 0 -1 4202496 3327 0 0 0 9 0 0 0 21 0 1 0 517699272 21778432 3270 33554432000 4194304 4338100 140733230247552 18446744073709551615 216634176197 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/15752/statm: 5317 3271 148 36 0 3222 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 21268

[startup+0.300416 s]
/proc/loadavg: 4.91 7.22 7.45 2/177 15752
/proc/meminfo: memFree=17911332/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=55176 CPUtime=0.29 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) R 15750 15752 12737 0 -1 4202496 10703 0 0 0 29 0 0 0 22 0 1 0 517699272 56500224 10645 33554432000 4194304 4338100 140733230247552 18446744073709551615 216633910466 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/15752/statm: 13794 10645 153 36 0 11699 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 55176

[startup+0.700358 s]
/proc/loadavg: 4.91 7.22 7.45 2/177 15752
/proc/meminfo: memFree=17911332/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=48976 CPUtime=0.69 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) R 15750 15752 12737 0 -1 4202496 11551 0 0 0 69 0 0 0 24 0 1 0 517699272 50151424 9732 33554432000 4194304 4338100 140733230247552 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/15752/statm: 12244 9732 155 36 0 10149 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 48976

[startup+1.50021 s]
/proc/loadavg: 4.91 7.22 7.45 7/182 15757
/proc/meminfo: memFree=17845084/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=76368 CPUtime=1.49 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) R 15750 15752 12737 0 -1 4202496 19243 0 0 0 147 2 0 0 25 0 1 0 517699272 78200832 17116 33554432000 4194304 4338100 140733230247552 18446744073709551615 4282397 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/15752/statm: 19092 17116 165 36 0 16997 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 76368

[startup+3.10089 s]
/proc/loadavg: 4.91 7.22 7.45 10/186 15761
/proc/meminfo: memFree=17796272/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=331704 CPUtime=7.85 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 22592 0 0 0 768 17 0 0 25 0 5 0 517699272 339664896 20465 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 82926 20466 176 36 0 80831 0
[pid=15752/tid=15758] ppid=15750 vsize=331704 CPUtime=1.58 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 652 0 0 0 154 4 0 0 20 0 5 0 517699422 339664896 20466 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=331704 CPUtime=1.58 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 494 0 0 0 154 4 0 0 18 0 5 0 517699422 339664896 20466 33554432000 4194304 4338100 140733230247552 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=331704 CPUtime=1.59 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 855 0 0 0 156 3 0 0 20 0 5 0 517699422 339664896 20466 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=331704 CPUtime=1.57 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 1341 0 0 0 155 2 0 0 25 0 5 0 517699423 339664896 20466 33554432000 4194304 4338100 140733230247552 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 7.85
Current children cumulated vsize (KiB) 331704

[startup+6.30032 s]
/proc/loadavg: 5.15 7.23 7.45 10/186 15761
/proc/meminfo: memFree=17735512/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=351528 CPUtime=20.62 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 27860 0 0 0 2015 47 0 0 25 0 5 0 517699272 359964672 25733 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 87882 25733 176 36 0 85787 0
[pid=15752/tid=15758] ppid=15750 vsize=351528 CPUtime=4.77 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 1632 0 0 0 466 11 0 0 25 0 5 0 517699422 359964672 25733 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=351528 CPUtime=4.78 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 1565 0 0 0 466 12 0 0 25 0 5 0 517699422 359964672 25733 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=351528 CPUtime=4.78 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 2391 0 0 0 468 10 0 0 25 0 5 0 517699422 359964672 25733 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=351528 CPUtime=4.75 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 3021 0 0 0 465 10 0 0 25 0 5 0 517699423 359964672 25733 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 20.62
Current children cumulated vsize (KiB) 351528

[startup+12.7002 s]
/proc/loadavg: 5.38 7.25 7.45 9/186 15761
/proc/meminfo: memFree=17661200/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=371352 CPUtime=46.17 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 32918 0 0 0 4510 107 0 0 25 0 5 0 517699272 380264448 30791 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 92838 30791 176 36 0 90743 0
[pid=15752/tid=15758] ppid=15750 vsize=371352 CPUtime=11.17 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 2453 0 0 0 1090 27 0 0 25 0 5 0 517699422 380264448 30791 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=371352 CPUtime=11.17 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 2284 0 0 0 1091 26 0 0 25 0 5 0 517699422 380264448 30791 33554432000 4194304 4338100 140733230247552 18446744073709551615 4256032 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=371352 CPUtime=11.18 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 4022 0 0 0 1092 26 0 0 25 0 5 0 517699422 380264448 30791 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=371352 CPUtime=11.11 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 4908 0 0 0 1087 24 0 0 25 0 5 0 517699423 380264448 30791 33554432000 4194304 4338100 140733230247552 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 46.17
Current children cumulated vsize (KiB) 371352

[startup+25.5009 s]
/proc/loadavg: 5.96 7.28 7.46 11/186 15761
/proc/meminfo: memFree=17543784/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=420652 CPUtime=97.26 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 46405 0 0 0 9481 245 0 0 25 0 5 0 517699272 430747648 44278 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 105163 44278 176 36 0 103068 0
[pid=15752/tid=15758] ppid=15750 vsize=420652 CPUtime=23.92 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 4633 0 0 0 2331 61 0 0 25 0 5 0 517699422 430747648 44278 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=420652 CPUtime=23.96 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 4446 0 0 0 2338 58 0 0 21 0 5 0 517699422 430747648 44278 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=420652 CPUtime=23.97 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 8266 0 0 0 2336 61 0 0 19 0 5 0 517699422 430747648 44278 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=420652 CPUtime=23.87 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 9809 0 0 0 2326 61 0 0 25 0 5 0 517699423 430747648 44278 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646340980 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 97.26
Current children cumulated vsize (KiB) 420652

[startup+51.1004 s]
/proc/loadavg: 6.66 7.34 7.47 10/186 15762
/proc/meminfo: memFree=17370304/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=498100 CPUtime=199.49 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 67324 0 0 0 19450 499 0 0 25 0 5 0 517699272 510054400 65197 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 124525 65197 176 36 0 122430 0
[pid=15752/tid=15758] ppid=15750 vsize=498100 CPUtime=49.37 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 7969 0 0 0 4820 117 0 0 25 0 5 0 517699422 510054400 65197 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=498100 CPUtime=49.55 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 7459 0 0 0 4842 113 0 0 25 0 5 0 517699422 510054400 65197 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=498100 CPUtime=49.56 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 14989 0 0 0 4825 131 0 0 25 0 5 0 517699422 510054400 65197 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=498100 CPUtime=49.46 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 17656 0 0 0 4812 134 0 0 25 0 5 0 517699423 510054400 65197 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 199.49
Current children cumulated vsize (KiB) 498100

[startup+102.306 s]
/proc/loadavg: 7.41 7.44 7.50 9/186 15763
/proc/meminfo: memFree=17074540/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=599264 CPUtime=403.94 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 95512 0 0 0 39370 1024 0 0 25 0 5 0 517699272 613646336 93385 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 149816 93385 176 36 0 147721 0
[pid=15752/tid=15758] ppid=15750 vsize=599264 CPUtime=100.37 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 12537 0 0 0 9797 240 0 0 18 0 5 0 517699422 613646336 93385 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646328014 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=599264 CPUtime=100.65 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 12616 0 0 0 9832 233 0 0 22 0 5 0 517699422 613646336 93385 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=599264 CPUtime=100.74 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 24330 0 0 0 9799 275 0 0 25 0 5 0 517699422 613646336 93385 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=599264 CPUtime=100.65 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 26778 0 0 0 9792 273 0 0 25 0 5 0 517699423 613646336 93385 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 403.94
Current children cumulated vsize (KiB) 599264

[startup+162.301 s]
/proc/loadavg: 7.78 7.54 7.53 7/186 15765
/proc/meminfo: memFree=16836680/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=724232 CPUtime=643.43 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 128965 0 0 0 62671 1672 0 0 25 0 5 0 517699272 741613568 126838 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 181058 126838 176 36 0 178963 0
[pid=15752/tid=15758] ppid=15750 vsize=724232 CPUtime=160.27 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 17850 0 0 0 15630 397 0 0 25 0 5 0 517699422 741613568 126838 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=724232 CPUtime=160.34 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 19184 0 0 0 15654 380 0 0 25 0 5 0 517699422 741613568 126838 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=724232 CPUtime=160.69 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 35129 0 0 0 15622 447 0 0 25 0 5 0 517699422 741613568 126838 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=724232 CPUtime=160.6 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 37551 0 0 0 15615 445 0 0 25 0 5 0 517699423 741613568 126838 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 643.43
Current children cumulated vsize (KiB) 724232

[startup+222.3 s]
/proc/loadavg: 7.97 7.63 7.56 9/186 15767
/proc/meminfo: memFree=16535456/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=830096 CPUtime=882.94 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 157665 0 0 0 86015 2279 0 0 25 0 5 0 517699272 850018304 155538 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 207524 155538 176 36 0 205429 0
[pid=15752/tid=15758] ppid=15750 vsize=830096 CPUtime=220.18 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 23361 0 0 0 21474 544 0 0 25 0 5 0 517699422 850018304 155538 33554432000 4194304 4338100 140733230247552 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 88832 0 0 0 68206 1822 0 0 25 0 5 0 517699423 1378545664 296782 33554432000 4194304 4338100 140733230247552 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2799.09
Current children cumulated vsize (KiB) 1346236

[startup+762.307 s]
/proc/loadavg: 8.04 8.03 7.80 9/186 15781
/proc/meminfo: memFree=15280652/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1369520 CPUtime=3038.68 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 305727 0 0 0 296477 7391 0 0 25 0 5 0 517699272 1402388480 303600 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 342380 303600 176 36 0 340285 0
[pid=15752/tid=15758] ppid=15750 vsize=1369520 CPUtime=758.34 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 48970 0 0 0 74078 1756 0 0 25 0 5 0 517699422 1402388480 303600 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1369520 CPUtime=758.26 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 56189 0 0 0 74120 1706 0 0 25 0 5 0 517699422 1402388480 303600 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1369520 CPUtime=760.27 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 90703 0 0 0 74061 1966 0 0 25 0 5 0 517699422 1402388480 303600 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1369520 CPUtime=760.26 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 90614 0 0 0 74067 1959 0 0 25 0 5 0 517699423 1402388480 303600 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3038.68
Current children cumulated vsize (KiB) 1369520

[startup+822.301 s]
/proc/loadavg: 8.12 8.05 7.82 10/186 15783
/proc/meminfo: memFree=15213552/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1396260 CPUtime=3278.22 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 313266 0 0 0 319982 7840 0 0 25 0 5 0 517699272 1429770240 311139 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 349065 311139 176 36 0 346970 0
[pid=15752/tid=15758] ppid=15750 vsize=1396260 CPUtime=817.99 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 49473 0 0 0 79946 1853 0 0 25 0 5 0 517699422 1429770240 311139 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1396260 CPUtime=818.23 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 56696 0 0 0 80018 1805 0 0 25 0 5 0 517699422 1429770240 311139 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1396260 CPUtime=820.23 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 93657 0 0 0 79930 2093 0 0 25 0 5 0 517699422 1429770240 311139 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1396260 CPUtime=820.23 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 94189 0 0 0 79937 2086 0 0 25 0 5 0 517699423 1429770240 311139 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3278.22
Current children cumulated vsize (KiB) 1396260

[startup+882.301 s]
/proc/loadavg: 8.10 8.05 7.83 10/186 15784
/proc/meminfo: memFree=15151284/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1413616 CPUtime=3517.74 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 319230 0 0 0 343455 8319 0 0 25 0 5 0 517699272 1447542784 317103 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 353404 317103 176 36 0 351309 0
[pid=15752/tid=15758] ppid=15750 vsize=1413616 CPUtime=877.92 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 50173 0 0 0 85820 1972 0 0 19 0 5 0 517699422 1447542784 317103 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1413616 CPUtime=877.96 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 57104 0 0 0 85881 1915 0 0 25 0 5 0 517699422 1447542784 317103 33554432000 4194304 4338100 140733230247552 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1413616 CPUtime=880.19 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 95095 0 0 0 85796 2223 0 0 25 0 5 0 517699422 1447542784 317103 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1413616 CPUtime=880.12 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 97607 0 0 0 85807 2205 0 0 25 0 5 0 517699423 1447542784 317103 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3517.74
Current children cumulated vsize (KiB) 1413616

[startup+942.301 s]
/proc/loadavg: 8.09 8.06 7.84 10/186 15786
/proc/meminfo: memFree=15073272/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1443484 CPUtime=3757.26 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 327987 0 0 0 366913 8813 0 0 25 0 5 0 517699272 1478127616 325860 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 360871 325860 176 36 0 358776 0
[pid=15752/tid=15758] ppid=15750 vsize=1443484 CPUtime=937.75 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 52533 0 0 0 91681 2094 0 0 25 0 5 0 517699422 1478127616 325860 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1443484 CPUtime=937.78 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 57467 0 0 0 91745 2033 0 0 25 0 5 0 517699422 1478127616 325860 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1443484 CPUtime=940.17 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 98464 0 0 0 91666 2351 0 0 21 0 5 0 517699422 1478127616 325860 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1443484 CPUtime=940.03 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 100272 0 0 0 91671 2332 0 0 25 0 5 0 517699423 1478127616 325860 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3757.26
Current children cumulated vsize (KiB) 1443484

[startup+1002.3 s]
/proc/loadavg: 8.13 8.08 7.86 10/186 15787
/proc/meminfo: memFree=14994396/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1476084 CPUtime=3996.8 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 336967 0 0 0 390427 9253 0 0 25 0 5 0 517699272 1511510016 334840 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 369021 334840 176 36 0 366926 0
[pid=15752/tid=15758] ppid=15750 vsize=1476084 CPUtime=997.38 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 54391 0 0 0 97545 2193 0 0 19 0 5 0 517699422 1511510016 334840 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1476084 CPUtime=997.74 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 60011 0 0 0 97637 2137 0 0 25 0 5 0 517699422 1511510016 334840 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1476084 CPUtime=1000.14 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 99225 0 0 0 97544 2470 0 0 25 0 5 0 517699422 1511510016 334840 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1476084 CPUtime=1000.01 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 104089 0 0 0 97551 2450 0 0 25 0 5 0 517699423 1511510016 334840 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3996.8
Current children cumulated vsize (KiB) 1476084

[startup+1062.3 s]
/proc/loadavg: 8.13 8.09 7.87 10/186 15789
/proc/meminfo: memFree=14919220/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1489520 CPUtime=4236.37 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 341068 0 0 0 414008 9629 0 0 25 0 5 0 517699272 1525268480 338941 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 372380 338941 176 36 0 370285 0
[pid=15752/tid=15758] ppid=15750 vsize=1489520 CPUtime=1057.02 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 55358 0 0 0 103422 2280 0 0 18 0 5 0 517699422 1525268480 338941 33554432000 4194304 4338100 140733230247552 18446744073709551615 216633910648 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1489520 CPUtime=1057.72 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 61025 0 0 0 103546 2226 0 0 25 0 5 0 517699422 1525268480 338941 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1489520 CPUtime=1060.11 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 100187 0 0 0 103441 2570 0 0 25 0 5 0 517699422 1525268480 338941 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1489520 CPUtime=1059.98 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 105247 0 0 0 103449 2549 0 0 25 0 5 0 517699423 1525268480 338941 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4236.37
Current children cumulated vsize (KiB) 1489520

[startup+1122.3 s]
/proc/loadavg: 8.13 8.10 7.89 10/186 15791
/proc/meminfo: memFree=14837476/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1512472 CPUtime=4475.91 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 347717 0 0 0 437565 10026 0 0 25 0 5 0 517699272 1548771328 345590 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 378118 345590 176 36 0 376023 0
[pid=15752/tid=15758] ppid=15750 vsize=1512472 CPUtime=1116.88 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 56396 0 0 0 109313 2375 0 0 25 0 5 0 517699422 1548771328 345590 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1512472 CPUtime=1117.69 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 61568 0 0 0 109447 2322 0 0 25 0 5 0 517699422 1548771328 345590 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1512472 CPUtime=1120.04 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 103339 0 0 0 109328 2676 0 0 25 0 5 0 517699422 1548771328 345590 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1512472 CPUtime=1119.77 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 107163 0 0 0 109327 2650 0 0 25 0 5 0 517699423 1548771328 345590 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4475.91
Current children cumulated vsize (KiB) 1512472

[startup+1182.3 s]
/proc/loadavg: 8.05 8.08 7.90 10/186 15792
/proc/meminfo: memFree=14773228/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=1550932 CPUtime=4715.48 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) S 15750 15752 12737 0 -1 4202496 357410 0 0 0 461110 10438 0 0 25 0 5 0 517699272 1588154368 355283 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 387733 355283 176 36 0 385638 0
[pid=15752/tid=15758] ppid=15750 vsize=1550932 CPUtime=1176.86 cores=1,3,5,7
/proc/15752/task/15758/stat : 15758 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 58677 0 0 0 115203 2483 0 0 25 0 5 0 517699422 1588154368 355283 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=1550932 CPUtime=1177.66 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 63822 0 0 0 115341 2425 0 0 25 0 5 0 517699422 1588154368 355283 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=15752/tid=15760] ppid=15750 vsize=1550932 CPUtime=1180.02 cores=1,3,5,7
/proc/15752/task/15760/stat : 15760 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 106934 0 0 0 115222 2780 0 0 25 0 5 0 517699422 1588154368 355283 33554432000 4194304 4338100 140733230247552 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=15752/tid=15761] ppid=15750 vsize=1550932 CPUtime=1179.41 cores=1,3,5,7
/proc/15752/task/15761/stat : 15761 (ttsth-5-4) R 15750 15752 12737 0 -1 4202560 108726 0 0 0 115195 2746 0 0 25 0 5 0 517699423 1588154368 355283 33554432000 4194304 4338100 140733230247552 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4715.48
Current children cumulated vsize (KiB) 1550932



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.5 s]
/proc/loadavg: 7.71 8.01 7.88 2/177 15794
/proc/meminfo: memFree=17240984/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=0 CPUtime=4800.04 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) Z 15750 15752 12737 0 -1 4203548 359645 0 0 0 469411 10593 0 0 18 0 2 0 517699272 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 0 0 0 0 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=0 CPUtime=1198.86 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4203588 64501 0 0 0 117419 2467 0 0 25 0 2 0 517699422 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.5 s]
/proc/loadavg: 7.71 8.01 7.88 2/177 15794
/proc/meminfo: memFree=17240984/32951124 swapFree=67111528/67111528
[pid=15752] ppid=15750 vsize=0 CPUtime=4800.04 cores=1,3,5,7
/proc/15752/stat : 15752 (ttsth-5-4) Z 15750 15752 12737 0 -1 4203548 359645 0 0 0 469411 10593 0 0 18 0 2 0 517699272 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/15752/statm: 0 0 0 0 0 0 0
[pid=15752/tid=15759] ppid=15750 vsize=0 CPUtime=1198.86 cores=1,3,5,7
/proc/15752/task/15759/stat : 15759 (ttsth-5-4) R 15750 15752 12737 0 -1 4203588 64501 0 0 0 117419 2467 0 0 25 0 2 0 517699422 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.53
CPU time (s): 4800.07
CPU user time (s): 4694.11
CPU system time (s): 105.956
CPU usage (%): 398.832
Max. virtual memory (cumulated for all children) (KiB): 1557584

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4694.11
system time used= 105.956
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 359645
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 521029
involuntary context switches= 43056

runsolver used 1.85072 second user time and 4.72128 second system time

The end

Launcher Data 

Begin job on node103 at 2011-04-29 09:09:27
IDJOB=3249196
IDBENCH=83159
IDSOLVER=1654
FILE ID=node103/3249196-1304060966
RUNJOBID= node103-1304060966-15733
PBS_JOBID= 13170359
Free space on /tmp= 71516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-13x32.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249196-1304060966/watcher-3249196-1304060966 -o /tmp/evaluation-result-3249196-1304060966/solver-3249196-1304060966 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249196-1304060966.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 73c8ae5bbdfe41463e8e40ea211c5e14
RANDOM SEED=1796088102

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17911488 kB
Buffers:       2146040 kB
Cached:       12242728 kB
SwapCached:          0 kB
Active:        5570512 kB
Inactive:      8887812 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17911488 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            4420 kB
Writeback:           4 kB
AnonPages:       69324 kB
Mapped:          15016 kB
Slab:           518188 kB
PageTables:       4664 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   211388 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71516 MiB
End job on node103 at 2011-04-29 09:29:32