Trace number 3260506

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.16 1206.72

General information on the benchmark

Namecrafted/mosoi/289/
289-sat-6x9.cnf
MD5SUM7cb6eb41697b41369a6f06ef320865a8
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.001999
Satisfiable
(Un)Satisfiability was proved
Number of variables216
Number of clauses2538
Sum of the clauses size9504
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2324
Number of clauses of size 30
Number of clauses of size 42214
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260506-1304071593.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 216
0.00/0.00	c Clauses = 2538
0.00/0.00	c Literals = 9504
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 216
0.00/0.00	c Clauses = 2538
0.00/0.00	c Literals = 9504
0.00/0.00	c a2
0.00/0.00	c a1
0.03/0.06	c Independent sub-problems = 1
0.03/0.06	c Sub-problem with 216 variables and 2538 clauses
0.03/0.06	c using permutation method 4
0.03/0.06	c count = 2
0.03/0.06	c Ordering finished 0.050 seconds
0.03/0.06	c mapped 0.050 seconds
0.03/0.06	c permuted 0.050 seconds
0.03/0.06	c about to sort
0.03/0.06	c sorted
0.03/0.06	c clauses now 2538
0.03/0.06	c init finished 0.060 seconds
0.03/0.06	c tree built 0.060 seconds
0.03/0.06	c 
0.03/0.06	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.03/0.06	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.03/0.06	c ................................ | ..........                    |                                  0.1         0.0             
0.03/0.06	c ................................ | ..........                  1 |                                  0.1        10.0             
0.03/0.06	c ................................ | ..........                  2 |                                  0.1        20.0             
0.03/0.06	c ................................ | ..........                  3 |                                  0.1        30.0             
0.03/0.06	c ................................ | ..........                  4 |                                  0.1        40.0             
0.03/0.06	c ................................ | ..........                  5 |                                  0.1        50.0             
0.03/0.07	c ................................ | ..........               1000 | 4            1904                0.1     10000.0         9607
95.82/24.16	c ................................ | ..........             830000 | 194919       1455327            96.1      8638.6      8526460
191.87/48.24	c ................................ | ..........            1357000 | 335145       2305675           192.0      7066.6     13078558
287.86/72.33	c ................................ | ..........            1885000 | 488644       2929938           288.0      6545.1     16345174
383.73/96.48	c ................................ | ..........            2425000 | 645296       3579679           384.1      6314.0     19363984
480.04/120.64	c ................................ | *.........            2915000 | 793422       4155630           480.2      6070.5     23146438
575.89/144.75	c ................................ | *.........            3446000 | 957318       4708177           576.1      5981.9     25300994
671.82/168.85	c ................................ | *.........            3886000 | 1084549      5163446           672.0      5782.7     28029311
767.93/192.92	c ................................ | *.........            4210000 | 1173795      5578401           768.0      5481.8     30077411
863.89/217.04	c ................................ | *.........            4546000 | 1266868      5991486           864.1      5261.1     31634381
960.19/241.21	c ................................ | **........            4888000 | 1360853      6434371           960.2      5090.4     32512291
1055.67/265.29	c ................................ | **........            5209000 | 1448959      6942907          1056.1      4932.5     34839440
1151.91/289.44	c ................................ | **........            5550000 | 1547026      7332812          1152.1      4817.3     36018225
1247.84/313.54	c ................................ | **........            5900000 | 1646941      7695457          1248.0      4727.5     37242936
1343.99/337.76	c ................................ | **........            6260000 | 1749838      8135549          1344.2      4656.9     37854207
1439.84/361.88	c ................................ | ***.......            6617000 | 1851530      8530980          1440.2      4594.6     39098292
1536.08/386.04	c ................................ | ***.......            6974000 | 1955634      8932979          1536.3      4539.6     40290729
1631.85/410.17	c ................................ | ***.......            7344000 | 2068298      9334712          1632.1      4499.6     40368027
1727.92/434.23	c ................................ | ***.......            7710000 | 2180824      9697919          1728.0      4461.7     42529284
1823.99/458.42	c ................................ | ***.......            8138000 | 2312718      10072762         1824.1      4461.5     43985317
1919.85/482.54	c ................................ | ****......            8547000 | 2435467      10431382         1920.0      4451.4     46241850
2015.80/506.67	c ................................ | ****......            8822000 | 2513036      10741828         2016.1      4375.8     47065193
2112.22/530.82	c ................................ | ****......            9108000 | 2594851      11052588         2112.3      4311.9     47851449
2207.81/554.85	c ................................ | ****......            9389000 | 2674649      11370737         2208.0      4252.2     49426827
2303.82/578.98	c ................................ | ****......            9670000 | 2753762      11735042         2304.2      4196.7     49986980
2400.05/603.10	c ................................ | *****.....            9986000 | 2846718      12062295         2400.1      4160.7     50225267
2495.89/627.29	c ................................ | *****.....           10298000 | 2937294      12365845         2496.3      4125.4     50383957
2591.76/651.39	c ................................ | *****.....           10627000 | 3031844      12725502         2592.1      4099.7     50548217
2688.05/675.50	c ................................ | *****.....           10949000 | 3123004      13063533         2688.1      4073.2     51036156
2783.89/699.66	c ................................ | *****.....           11279000 | 3221990      13403295         2784.2      4051.1     51332653
2879.74/723.76	c ................................ | ******....           11609000 | 3323462      13714365         2880.0      4030.9     51801539
2976.09/747.92	c ................................ | ******....           11952000 | 3427581      14035578         2976.2      4015.9     52062543
3071.71/772.09	c ................................ | ******....           12378000 | 3560031      14372675         3072.1      4029.2     52572236
3167.97/796.22	c ................................ | ******....           12692000 | 3650827      14667174         3168.1      4006.2     53021346
3263.88/820.39	c ................................ | ******....           12963000 | 3726729      14948563         3264.3      3971.2     53779454
3360.09/844.45	c ................................ | *******...           13230000 | 3798934      15288986         3360.3      3937.2     55438231
3455.88/868.58	c ................................ | *******...           13525000 | 3882660      15616581         3456.2      3913.2     55584765
3551.80/892.65	c ................................ | *******...           13787000 | 3954700      15918787         3552.0      3881.5     56523571
3647.97/916.89	c ................................ | *******...           14085000 | 4039408      16247891         3648.3      3860.7     57192164
3743.83/941.00	c ................................ | *******...           14338000 | 4107612      16591580         3744.2      3829.4     58912087
3839.73/965.07	c ................................ | ********..           14627000 | 4187908      17011799         3840.0      3809.1     60100999
3935.98/989.23	c ................................ | ********..           14937000 | 4278012      17405283         3936.1      3794.9     61334041
4032.00/1013.38	c ................................ | ********..           15232000 | 4362580      17672427         4032.3      3777.5     62546625
4128.07/1037.52	c ................................ | ********..           15534000 | 4449973      17949211         4128.2      3762.9     63393269
4223.70/1061.67	c ................................ | ********..           15822000 | 4532458      18264704         4224.0      3745.7     63949132
4319.75/1085.89	c ................................ | *********.           16141000 | 4625098      18622794         4320.1      3736.2     64169566
4416.13/1110.00	c ................................ | *********.           16446000 | 4711597      18939992         4416.2      3724.0     65086168
4512.21/1134.21	c ................................ | *********.           16763000 | 4804267      19271145         4512.3      3715.0     65627350
4607.91/1158.35	c ................................ | *********.           17080000 | 4897089      19595657         4608.1      3706.5     65629930
4703.70/1182.48	c ................................ | *********.           17385000 | 4987345      19948468         4704.0      3695.8     65790405
4799.99/1206.61	c CPU limit (4800) reached
4799.99/1206.61	c Time = 4800.05 seconds
4799.99/1206.61	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3260506-1304071593/watcher-3260506-1304071593 -o /tmp/evaluation-result-3260506-1304071593/solver-3260506-1304071593 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260506-1304071593.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.48 7.96 7.45 2/177 23726
/proc/meminfo: memFree=14410100/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=9508 CPUtime=0 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) R 23724 23726 23403 0 -1 4202496 434 0 0 0 0 0 0 0 21 0 1 0 518760241 9736192 376 33554432000 4194304 4338100 140737289283952 18446744073709551615 4206978 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/23726/statm: 2377 376 150 36 0 282 0

[startup+0.0369719 s]
/proc/loadavg: 7.48 7.96 7.45 2/177 23726
/proc/meminfo: memFree=14410100/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=9508 CPUtime=0.03 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) R 23724 23726 23403 0 -1 4202496 448 0 0 0 3 0 0 0 21 0 1 0 518760241 9736192 390 33554432000 4194304 4338100 140737289283952 18446744073709551615 4296083 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/23726/statm: 2377 390 153 36 0 282 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 9508

[startup+0.162924 s]
/proc/loadavg: 7.48 7.96 7.45 2/177 23726
/proc/meminfo: memFree=14410100/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=203480 CPUtime=1.04 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 4574 0 0 0 104 0 0 0 21 0 5 0 518760241 208363520 4516 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 54062 7372 174 36 0 51967 0
Current children cumulated CPU time (s) 1.04
Current children cumulated vsize (KiB) 203480

[startup+0.843113 s]
/proc/loadavg: 7.48 7.96 7.45 2/177 23726
/proc/meminfo: memFree=14410100/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=219632 CPUtime=3.27 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 8948 0 0 0 325 2 0 0 21 0 5 0 518760241 224903168 8890 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 54908 8891 174 36 0 52813 0
Current children cumulated CPU time (s) 3.27
Current children cumulated vsize (KiB) 219632

[startup+0.96411 s]
/proc/loadavg: 7.48 7.96 7.45 2/177 23726
/proc/meminfo: memFree=14410100/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=222760 CPUtime=3.68 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 9675 0 0 0 365 3 0 0 21 0 5 0 518760241 228106240 9617 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 55690 9617 174 36 0 53595 0
Current children cumulated CPU time (s) 3.68
Current children cumulated vsize (KiB) 222760

[startup+1.50064 s]
/proc/loadavg: 7.48 7.96 7.45 2/177 23726
/proc/meminfo: memFree=14410100/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=301192 CPUtime=5.77 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 13038 0 0 0 574 3 0 0 21 0 5 0 518760241 308420608 12980 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 75298 12980 174 36 0 73203 0
Current children cumulated CPU time (s) 5.77
Current children cumulated vsize (KiB) 301192

[startup+3.10035 s]
/proc/loadavg: 7.48 7.96 7.45 9/186 23735
/proc/meminfo: memFree=14278936/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=333240 CPUtime=12.16 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 22091 0 0 0 1208 8 0 0 21 0 5 0 518760241 341237760 22033 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 83310 22033 174 36 0 81215 0
[pid=23726/tid=23728] ppid=23724 vsize=333240 CPUtime=3.01 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 4767 0 0 0 300 1 0 0 18 0 5 0 518760247 341237760 22033 33554432000 4194304 4338100 140737289283952 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=333240 CPUtime=3.02 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 5593 0 0 0 300 2 0 0 21 0 5 0 518760247 341237760 22033 33554432000 4194304 4338100 140737289283952 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=333240 CPUtime=3 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 5557 0 0 0 298 2 0 0 25 0 5 0 518760247 341237760 22033 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=333240 CPUtime=3.02 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 5078 0 0 0 301 1 0 0 25 0 5 0 518760247 341237760 22033 33554432000 4194304 4338100 140737289283952 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 12.16
Current children cumulated vsize (KiB) 333240

[startup+6.30079 s]
/proc/loadavg: 7.52 7.96 7.46 9/186 23735
/proc/meminfo: memFree=14171060/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=379188 CPUtime=24.89 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 33878 0 0 0 2471 18 0 0 21 0 5 0 518760241 388288512 33812 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 94797 33812 175 36 0 92702 0
[pid=23726/tid=23728] ppid=23724 vsize=379188 CPUtime=6.21 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 7527 0 0 0 617 4 0 0 19 0 5 0 518760247 388288512 33812 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=379188 CPUtime=6.2 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 8422 0 0 0 616 4 0 0 19 0 5 0 518760247 388288512 33813 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=379188 CPUtime=6.18 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 8615 0 0 0 613 5 0 0 25 0 5 0 518760247 388288512 33814 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=379188 CPUtime=6.21 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 8218 0 0 0 618 3 0 0 25 0 5 0 518760247 388288512 33814 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.89
Current children cumulated vsize (KiB) 379188

[startup+12.7067 s]
/proc/loadavg: 7.56 7.96 7.46 9/186 23735
/proc/meminfo: memFree=14021784/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=447476 CPUtime=50.43 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 51243 0 0 0 5009 34 0 0 21 0 5 0 518760241 458215424 51177 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 111869 51177 175 36 0 109774 0
[pid=23726/tid=23728] ppid=23724 vsize=447476 CPUtime=12.59 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 12307 0 0 0 1250 9 0 0 18 0 5 0 518760247 458215424 51177 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=447476 CPUtime=12.58 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 13147 0 0 0 1251 7 0 0 21 0 5 0 518760247 458215424 51177 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=447476 CPUtime=12.54 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 13069 0 0 0 1244 10 0 0 25 0 5 0 518760247 458215424 51177 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=447476 CPUtime=12.62 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 11624 0 0 0 1256 6 0 0 25 0 5 0 518760247 458215424 51177 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 50.43
Current children cumulated vsize (KiB) 447476

[startup+25.5004 s]
/proc/loadavg: 7.65 7.96 7.47 10/186 23736
/proc/meminfo: memFree=13771184/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=537976 CPUtime=101.4 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 75887 0 0 0 10077 63 0 0 21 0 5 0 518760241 550887424 75821 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 134494 75821 175 36 0 132399 0
[pid=23726/tid=23728] ppid=23724 vsize=537976 CPUtime=25.36 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 18411 0 0 0 2518 18 0 0 23 0 5 0 518760247 550887424 75821 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=537976 CPUtime=25.32 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 19946 0 0 0 2520 12 0 0 20 0 5 0 518760247 550887424 75821 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=537976 CPUtime=25.24 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 18660 0 0 0 2505 19 0 0 20 0 5 0 518760247 550887424 75821 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=537976 CPUtime=25.38 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 17774 0 0 0 2526 12 0 0 25 0 5 0 518760247 550887424 75821 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 101.4
Current children cumulated vsize (KiB) 537976

[startup+51.1009 s]
/proc/loadavg: 7.77 7.97 7.48 9/186 23736
/proc/meminfo: memFree=13425468/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=690376 CPUtime=203.43 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 114911 0 0 0 20232 111 0 0 21 0 5 0 518760241 706945024 114845 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 172594 114845 175 36 0 170499 0
[pid=23726/tid=23728] ppid=23724 vsize=690376 CPUtime=50.88 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 28822 0 0 0 5056 32 0 0 18 0 5 0 518760247 706945024 114845 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=690376 CPUtime=50.87 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 28919 0 0 0 5065 22 0 0 23 0 5 0 518760247 706945024 114845 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=690376 CPUtime=50.66 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 28345 0 0 0 5032 34 0 0 25 0 5 0 518760247 706945024 114845 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=690376 CPUtime=50.93 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 27729 0 0 0 5071 22 0 0 18 0 5 0 518760247 706945024 114845 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 203.43
Current children cumulated vsize (KiB) 690376

[startup+102.301 s]
/proc/loadavg: 8.09 8.02 7.52 9/186 23738
/proc/meminfo: memFree=13044276/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=910764 CPUtime=407.19 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 173089 0 0 0 40507 212 0 0 21 0 5 0 518760241 932622336 173023 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 227691 173023 175 36 0 225596 0
[pid=23726/tid=23728] ppid=23724 vsize=910764 CPUtime=101.75 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 46216 0 0 0 10121 54 0 0 18 0 5 0 518760247 932622336 173023 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=910764 CPUtime=101.89 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 40513 0 0 0 10135 54 0 0 20 0 5 0 518760247 932622336 173023 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=910764 CPUtime=101.73 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 42402 0 0 0 10119 54 0 0 21 0 5 0 518760247 932622336 173023 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=910764 CPUtime=101.74 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 42862 0 0 0 10125 49 0 0 20 0 5 0 518760247 932622336 173023 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 407.19
Current children cumulated vsize (KiB) 910764

[startup+162.306 s]
/proc/loadavg: 8.08 8.03 7.55 9/186 23739
/proc/meminfo: memFree=12658628/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=1158148 CPUtime=645.95 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 237600 0 0 0 64252 343 0 0 21 0 5 0 518760241 1185943552 237534 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 289537 237534 175 36 0 287442 0
[pid=23726/tid=23728] ppid=23724 vsize=1158148 CPUtime=161.41 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 67871 0 0 0 16057 84 0 0 18 0 5 0 518760247 1185943552 237534 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=1158148 CPUtime=161.38 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 54459 0 0 0 16045 93 0 0 25 0 5 0 518760247 1185943552 237534 33554432000 4194304 4338100 140737289283952 18446744073709551615 270612807161 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=1158148 CPUtime=161.53 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 58703 0 0 0 16072 81 0 0 25 0 5 0 518760247 1185943552 237534 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=1158148 CPUtime=161.53 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 55471 0 0 0 16069 84 0 0 25 0 5 0 518760247 1185943552 237534 33554432000 4194304 4338100 140737289283952 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 645.95
Current children cumulated vsize (KiB) 1158148

[startup+222.301 s]
/proc/loadavg: 8.12 8.05 7.59 9/186 23741
/proc/meminfo: memFree=12459108/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=1291460 CPUtime=884.98 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 273203 0 0 0 88053 445 0 0 21 0 5 0 518760241 1322455040 273137 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 322865 273137 175 36 0 320770 0
[pid=23726/tid=23728] ppid=23724 vsize=1291460 CPUtime=221.18 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 79153 0 0 0 22013 105 0 0 25 0 5 0 518760247 1322455040 273137 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 97910 0 0 0 69677 292 0 0 25 0 5 0 518760247 1999429632 450596 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2794.66
Current children cumulated vsize (KiB) 1952568

[startup+762.301 s]
/proc/loadavg: 8.14 8.06 7.78 9/186 23755
/proc/meminfo: memFree=10755376/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=1979512 CPUtime=3033.21 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 459292 0 0 0 301899 1422 0 0 21 0 5 0 518760241 2027020288 459226 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 494878 459226 175 36 0 492783 0
[pid=23726/tid=23728] ppid=23724 vsize=1979512 CPUtime=756.79 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 154966 0 0 0 75296 383 0 0 21 0 5 0 518760247 2027020288 459226 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=1979512 CPUtime=758.38 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 98600 0 0 0 75528 310 0 0 21 0 5 0 518760247 2027020288 459226 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=1979512 CPUtime=758.46 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 106032 0 0 0 75441 405 0 0 18 0 5 0 518760247 2027020288 459226 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=1979512 CPUtime=759.49 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 98598 0 0 0 75626 323 0 0 25 0 5 0 518760247 2027020288 459226 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3033.21
Current children cumulated vsize (KiB) 1979512

[startup+822.306 s]
/proc/loadavg: 8.34 8.12 7.82 9/186 23757
/proc/meminfo: memFree=10500172/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=2031736 CPUtime=3271.89 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 474185 0 0 0 325666 1523 0 0 21 0 5 0 518760241 2080497664 474119 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 507934 474119 175 36 0 505839 0
[pid=23726/tid=23728] ppid=23724 vsize=2031736 CPUtime=816.43 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 166307 0 0 0 81236 407 0 0 25 0 5 0 518760247 2080497664 474119 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=2031736 CPUtime=817.95 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 99160 0 0 0 81459 336 0 0 18 0 5 0 518760247 2080497664 474119 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=2031736 CPUtime=818.21 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 108171 0 0 0 81394 427 0 0 18 0 5 0 518760247 2080497664 474119 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=2031736 CPUtime=819.22 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 99451 0 0 0 81570 352 0 0 25 0 5 0 518760247 2080497664 474119 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3271.89
Current children cumulated vsize (KiB) 2031736

[startup+882.301 s]
/proc/loadavg: 8.32 8.14 7.84 9/186 23758
/proc/meminfo: memFree=10225380/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=2096728 CPUtime=3510.83 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 492331 0 0 0 349468 1615 0 0 21 0 5 0 518760241 2147049472 492265 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 524182 492265 175 36 0 522087 0
[pid=23726/tid=23728] ppid=23724 vsize=2096728 CPUtime=876.21 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 171464 0 0 0 87195 426 0 0 25 0 5 0 518760247 2147049472 492265 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=2096728 CPUtime=877.76 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 102517 0 0 0 87415 361 0 0 25 0 5 0 518760247 2147049472 492265 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=2096728 CPUtime=877.69 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 112939 0 0 0 87323 446 0 0 20 0 5 0 518760247 2147049472 492265 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=2096728 CPUtime=879.07 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 104315 0 0 0 87526 381 0 0 25 0 5 0 518760247 2147049472 492265 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3510.83
Current children cumulated vsize (KiB) 2096728

[startup+942.3 s]
/proc/loadavg: 8.19 8.14 7.86 9/186 23760
/proc/meminfo: memFree=10001052/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=2196132 CPUtime=3749.33 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 518919 0 0 0 373217 1716 0 0 21 0 5 0 518760241 2248839168 518853 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 549033 518853 175 36 0 546938 0
[pid=23726/tid=23728] ppid=23724 vsize=2196132 CPUtime=935.83 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 180801 0 0 0 93139 444 0 0 25 0 5 0 518760247 2248839168 518853 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=2196132 CPUtime=937.48 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 110967 0 0 0 93361 387 0 0 25 0 5 0 518760247 2248839168 518853 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=2196132 CPUtime=937.07 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 119317 0 0 0 93236 471 0 0 23 0 5 0 518760247 2248839168 518853 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=2196132 CPUtime=938.87 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 106738 0 0 0 93474 413 0 0 25 0 5 0 518760247 2248839168 518853 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3749.33
Current children cumulated vsize (KiB) 2196132

[startup+1002.3 s]
/proc/loadavg: 8.20 8.16 7.88 9/186 23762
/proc/meminfo: memFree=9805248/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=2293184 CPUtime=3988.18 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 545499 0 0 0 397001 1817 0 0 21 0 5 0 518760241 2348220416 545433 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 573296 545433 175 36 0 571201 0
[pid=23726/tid=23728] ppid=23724 vsize=2293184 CPUtime=995.36 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 191539 0 0 0 99074 462 0 0 21 0 5 0 518760247 2348220416 545433 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=2293184 CPUtime=997.29 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 116302 0 0 0 99310 419 0 0 25 0 5 0 518760247 2348220416 545433 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=2293184 CPUtime=996.74 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 124769 0 0 0 99181 493 0 0 25 0 5 0 518760247 2348220416 545433 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=2293184 CPUtime=998.7 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 111793 0 0 0 99429 441 0 0 18 0 5 0 518760247 2348220416 545433 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3988.18
Current children cumulated vsize (KiB) 2293184

[startup+1062.3 s]
/proc/loadavg: 8.07 8.13 7.89 9/186 23763
/proc/meminfo: memFree=9687932/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=2358304 CPUtime=4226.48 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 563136 0 0 0 420731 1917 0 0 21 0 5 0 518760241 2414903296 563070 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 589576 563070 175 36 0 587481 0
[pid=23726/tid=23728] ppid=23724 vsize=2358304 CPUtime=1054.47 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 198512 0 0 0 104963 484 0 0 21 0 5 0 518760247 2414903296 563070 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=2358304 CPUtime=1057.07 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 118402 0 0 0 105258 449 0 0 25 0 5 0 518760247 2414903296 563070 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=2358304 CPUtime=1056.33 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 130706 0 0 0 105120 513 0 0 21 0 5 0 518760247 2414903296 563070 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=2358304 CPUtime=1058.52 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 114420 0 0 0 105382 470 0 0 23 0 5 0 518760247 2414903296 563070 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4226.48
Current children cumulated vsize (KiB) 2358304

[startup+1122.3 s]
/proc/loadavg: 7.94 8.07 7.88 9/186 23765
/proc/meminfo: memFree=9516436/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=2410792 CPUtime=4464.9 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 578600 0 0 0 444469 2021 0 0 21 0 5 0 518760241 2468651008 578534 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 602698 578534 175 36 0 600603 0
[pid=23726/tid=23728] ppid=23724 vsize=2410792 CPUtime=1113.68 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 207127 0 0 0 110859 509 0 0 19 0 5 0 518760247 2468651008 578534 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=2410792 CPUtime=1116.81 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 119392 0 0 0 111201 480 0 0 25 0 5 0 518760247 2468651008 578534 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=2410792 CPUtime=1116.08 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 131718 0 0 0 111075 533 0 0 25 0 5 0 518760247 2468651008 578534 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624936629 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=2410792 CPUtime=1118.25 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 119267 0 0 0 111327 498 0 0 25 0 5 0 518760247 2468651008 578534 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4464.9
Current children cumulated vsize (KiB) 2410792

[startup+1182.3 s]
/proc/loadavg: 7.98 8.06 7.89 10/186 23766
/proc/meminfo: memFree=9378788/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=2424604 CPUtime=4703.3 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) S 23724 23726 23403 0 -1 4202496 584821 0 0 0 468231 2099 0 0 21 0 5 0 518760241 2482794496 584755 33554432000 4194304 4338100 140737289283952 18446744073709551615 270624913381 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 606151 584755 175 36 0 604056 0
[pid=23726/tid=23728] ppid=23724 vsize=2424604 CPUtime=1172.85 cores=0,2,4,6
/proc/23726/task/23728/stat : 23728 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 208881 0 0 0 116761 524 0 0 19 0 5 0 518760247 2482794496 584755 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=23726/tid=23729] ppid=23724 vsize=2424604 CPUtime=1176.53 cores=0,2,4,6
/proc/23726/task/23729/stat : 23729 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 119927 0 0 0 117148 505 0 0 18 0 5 0 518760247 2482794496 584755 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=2424604 CPUtime=1175.9 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 133556 0 0 0 117039 551 0 0 25 0 5 0 518760247 2482794496 584755 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=23726/tid=23731] ppid=23724 vsize=2424604 CPUtime=1177.93 cores=0,2,4,6
/proc/23726/task/23731/stat : 23731 (ttsth-5-4) R 23724 23726 23403 0 -1 4202560 121361 0 0 0 117276 517 0 0 18 0 5 0 518760247 2482794496 584755 33554432000 4194304 4338100 140737289283952 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4703.3
Current children cumulated vsize (KiB) 2424604



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1206.7 s]
/proc/loadavg: 8.06 8.07 7.90 2/177 23768
/proc/meminfo: memFree=13928444/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=0 CPUtime=4800.14 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) Z 23724 23726 23403 0 -1 4203548 586354 0 0 0 477880 2134 0 0 18 0 2 0 518760241 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 0 0 0 0 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=0 CPUtime=1200.22 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4203588 134245 0 0 0 119455 567 0 0 25 0 2 0 518760247 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.14
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1206.7 s]
/proc/loadavg: 8.06 8.07 7.90 2/177 23768
/proc/meminfo: memFree=13928444/32951124 swapFree=67111528/67111528
[pid=23726] ppid=23724 vsize=0 CPUtime=4800.14 cores=0,2,4,6
/proc/23726/stat : 23726 (ttsth-5-4) Z 23724 23726 23403 0 -1 4203548 586354 0 0 0 477880 2134 0 0 18 0 2 0 518760241 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/23726/statm: 0 0 0 0 0 0 0
[pid=23726/tid=23730] ppid=23724 vsize=0 CPUtime=1200.22 cores=0,2,4,6
/proc/23726/task/23730/stat : 23730 (ttsth-5-4) R 23724 23726 23403 0 -1 4203588 134245 0 0 0 119455 567 0 0 25 0 2 0 518760247 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.14
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1206.72
CPU time (s): 4800.16
CPU user time (s): 4778.8
CPU system time (s): 21.3598
CPU usage (%): 397.785
Max. virtual memory (cumulated for all children) (KiB): 2425628

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4778.8
system time used= 21.3598
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 586354
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 96791
involuntary context switches= 20843

runsolver used 1.9417 second user time and 4.33434 second system time

The end

Launcher Data 

Begin job on node108 at 2011-04-29 12:06:33
IDJOB=3260506
IDBENCH=83186
IDSOLVER=1654
FILE ID=node108/3260506-1304071593
RUNJOBID= node108-1304071593-23706
PBS_JOBID= 13170476
Free space on /tmp= 72120 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-6x9.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3260506-1304071593/watcher-3260506-1304071593 -o /tmp/evaluation-result-3260506-1304071593/solver-3260506-1304071593 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260506-1304071593.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 7cb6eb41697b41369a6f06ef320865a8
RANDOM SEED=1789125529

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      14410488 kB
Buffers:       2272528 kB
Cached:       15620896 kB
SwapCached:          0 kB
Active:        9012120 kB
Inactive:      8946388 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      14410488 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             128 kB
Writeback:           0 kB
AnonPages:       63888 kB
Mapped:          14880 kB
Slab:           519084 kB
PageTables:       4896 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   208700 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72120 MiB
End job on node108 at 2011-04-29 12:26:42