Trace number 3266358

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.67 1202.25

General information on the benchmark

NameSAT02/handmade/pehoushek/plainoldcnf/
5cnf_4300_4300_110.shuffled.cnf
MD5SUM34f8b747bfe8e7b589b8832e56ae8363
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.022996
Satisfiable
(Un)Satisfiability was proved
Number of variables110
Number of clauses1892
Sum of the clauses size9308
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 33
Number of clauses of size 4146
Number of clauses of size 51743
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266358-1304104641.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 110
0.00/0.00	c Clauses = 1892
0.00/0.00	c Literals = 9308
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 110
0.00/0.00	c Clauses = 1892
0.00/0.00	c Literals = 9308
0.00/0.00	c a2
0.00/0.00	c a1
0.09/0.15	c Independent sub-problems = 1
0.09/0.15	c Sub-problem with 110 variables and 1892 clauses
0.09/0.15	c using permutation method 4
0.09/0.15	c count = 2
0.09/0.15	c Ordering finished 0.150 seconds
0.09/0.15	c mapped 0.150 seconds
0.09/0.15	c permuted 0.150 seconds
0.09/0.15	c about to sort
0.09/0.15	c sorted
0.09/0.16	c clauses now 1892
0.09/0.16	c init finished 0.150 seconds
0.09/0.16	c tree built 0.160 seconds
0.09/0.16	c 
0.09/0.16	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.16	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.16	c ................................ | ..........                    |                                  0.2         0.0             
0.09/0.16	c ................................ | ..........                  1 |                                  0.2         6.2             
0.09/0.16	c ................................ | ..........                  2 |                                  0.2        12.5             
0.09/0.16	c ................................ | ..........                  3 |                                  0.2        18.8             
0.09/0.16	c ................................ | ..........                  4 |                                  0.2        25.0             
0.09/0.16	c ................................ | ..........                  5 |                                  0.2        31.2             
0.09/0.17	c ................................ | ..........               1000 | 45           1338                0.2      5555.6        10923
96.18/24.26	c ................................ | ..........             520000 | 77410        840890             96.4      5394.2     18160836
192.03/48.27	c ................................ | ..........             804000 | 122659       1216594           192.3      4180.1     27776479
287.88/72.27	c ................................ | ..........             982000 | 154735       1488368           288.2      3407.8     34912605
384.12/96.36	c ................................ | ..........            1184000 | 188857       1752704           384.4      3080.4     42140015
480.34/120.41	c ................................ | *.........            1335000 | 214635       1984926           480.4      2778.9     47585193
576.22/144.47	c ................................ | *.........            1454000 | 235809       2175388           576.5      2522.1     52545245
672.47/168.56	c ................................ | *.........            1587000 | 259377       2342986           672.7      2359.0     57873685
768.37/192.59	c ................................ | *.........            1685000 | 278318       2497465           768.7      2192.0     62853922
863.81/216.50	c ................................ | *.........            1787000 | 298029       2649263           864.2      2067.8     67444424
960.47/240.68	c ................................ | **........            1898000 | 316800       2789119           960.8      1975.4     71401955
1055.92/264.57	c ................................ | **........            2014000 | 336742       2932618          1056.2      1906.8     74201165
1152.19/288.66	c ................................ | **........            2125000 | 356990       3084975          1152.5      1843.9     78585783
1248.45/312.75	c ................................ | **........            2240000 | 378890       3231209          1248.7      1793.9     82501339
1344.72/336.80	c ................................ | **........            2333000 | 395135       3358088          1344.7      1734.9     86268417
1440.21/360.72	c ................................ | ***.......            2428000 | 411811       3469197          1440.3      1685.8     90073246
1536.06/384.75	c ................................ | ***.......            2524000 | 431068       3593929          1536.3      1642.9     94266316
1632.71/408.92	c ................................ | ***.......            2619000 | 449376       3712656          1632.8      1604.0     97958455
1728.13/432.84	c ................................ | ***.......            2721000 | 463644       3834977          1728.3      1574.4    100576919
1823.98/456.85	c ................................ | ***.......            2825000 | 480114       3954559          1824.2      1548.6    103702817
1919.84/480.84	c ................................ | ****......            2915000 | 495377       4066336          1920.0      1518.2    106970424
2016.90/505.12	c ................................ | ****......            2989000 | 509839       4187394          2017.0      1481.9    110454703
2112.38/529.06	c ................................ | ****......            3064000 | 524291       4305472          2112.6      1450.3    113843777
2208.20/553.03	c ................................ | ****......            3149000 | 538174       4427433          2208.3      1426.0    116958031
2304.47/577.11	c ................................ | ****......            3226000 | 552178       4533461          2304.5      1399.9    119730815
2400.34/601.11	c ................................ | *****.....            3299000 | 566653       4641368          2400.4      1374.4    122675121
2496.98/625.36	c ................................ | *****.....            3390000 | 583029       4740950          2497.2      1357.5    124966149
2592.43/649.24	c ................................ | *****.....            3461000 | 595805       4851797          2592.6      1334.9    126941325
2688.70/673.31	c ................................ | *****.....            3532000 | 608168       4962123          2688.8      1313.6    128440141
2784.18/697.26	c ................................ | *****.....            3617000 | 623996       5065652          2784.4      1299.0    131354954
2880.01/721.28	c ................................ | ******....            3682000 | 634539       5166589          2880.3      1278.3    133872552
2976.28/745.38	c ................................ | ******....            3750000 | 646215       5279632          2976.6      1259.8    136937876
3072.14/769.34	c ................................ | ******....            3823000 | 658457       5386650          3072.3      1244.3    139833287
3168.75/793.56	c ................................ | ******....            3892000 | 669762       5486963          3169.0      1228.1    142691723
3264.22/817.48	c ................................ | ******....            3963000 | 682650       5574749          3264.6      1213.9    145434532
3360.87/841.65	c ................................ | *******...            4037000 | 695378       5657952          3361.1      1201.1    147045067
3456.72/865.61	c ................................ | *******...            4105000 | 708176       5735355          3456.8      1187.5    148715267
3553.38/889.86	c ................................ | *******...            4177000 | 720419       5834293          3553.6      1175.4    150229273
3648.04/913.58	c ................................ | *******...            4239000 | 731007       5925633          3648.4      1161.9    152628056
3745.09/937.85	c ................................ | *******...            4293000 | 740555       6013082          3745.3      1146.2    155162625
3841.36/961.90	c ................................ | ********..            4350000 | 751298       6092947          3841.4      1132.4    157599898
3936.06/985.69	c ................................ | ********..            4403000 | 762640       6166996          3936.4      1118.5    160133415
4033.10/1009.92	c ................................ | ********..            4463000 | 773721       6239724          4033.2      1106.6    162395913
4128.17/1033.71	c ................................ | ********..            4517000 | 783907       6322993          4128.2      1094.2    164887301
4224.05/1057.79	c ................................ | ********..            4571000 | 792844       6396124          4224.4      1082.0    167619893
4320.30/1081.88	c ................................ | *********.            4633000 | 804582       6472026          4320.6      1072.3    170064076
4416.93/1106.02	c ................................ | *********.            4687000 | 815393       6544480          4417.0      1061.1    172167157
4511.99/1129.81	c ................................ | *********.            4738000 | 826881       6612329          4512.0      1050.1    174663968
4608.66/1154.05	c ................................ | *********.            4796000 | 839292       6676920          4608.9      1040.6    177599114
4704.13/1177.98	c ................................ | *********.            4851000 | 850534       6743026          4704.5      1031.1    179676285

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3266358-1304104641/watcher-3266358-1304104641 -o /tmp/evaluation-result-3266358-1304104641/solver-3266358-1304104641 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266358-1304104641.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=9508 CPUtime=0 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) R 3789 3792 3320 0 -1 4202496 442 0 0 0 0 0 0 0 22 0 1 0 522063763 9736192 383 33554432000 4194304 4338100 140733401584688 18446744073709551615 274473362599 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3792/statm: 2377 383 152 36 0 282 0

[startup+0.0505541 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=9508 CPUtime=0.04 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) R 3789 3792 3320 0 -1 4202496 453 0 0 0 4 0 0 0 22 0 1 0 522063763 9736192 394 33554432000 4194304 4338100 140733401584688 18446744073709551615 4296035 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3792/statm: 2377 394 155 36 0 282 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 9508

[startup+0.100545 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=9508 CPUtime=0.09 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) R 3789 3792 3320 0 -1 4202496 453 0 0 0 9 0 0 0 22 0 1 0 522063763 9736192 394 33554432000 4194304 4338100 140733401584688 18446744073709551615 4296004 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/3792/statm: 2377 394 155 36 0 282 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 9508

[startup+0.300514 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=197992 CPUtime=0.7 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 4069 0 0 0 70 0 0 0 23 0 5 0 522063763 202743808 3986 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 49498 3987 175 36 0 47403 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 197992

[startup+0.700444 s]
/proc/loadavg: 7.16 7.91 7.99 2/176 3794
/proc/meminfo: memFree=19216800/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=226792 CPUtime=2.29 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 10855 0 0 0 228 1 0 0 23 0 5 0 522063763 232235008 10773 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 56698 10773 175 36 0 54603 0
Current children cumulated CPU time (s) 2.29
Current children cumulated vsize (KiB) 226792

[startup+1.5003 s]
/proc/loadavg: 7.16 7.91 7.99 10/186 3804
/proc/meminfo: memFree=19097696/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=271484 CPUtime=5.49 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 21509 0 0 0 546 3 0 0 23 0 5 0 522063763 277999616 21426 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 67871 21426 175 36 0 65776 0
[pid=3792/tid=3797] ppid=3789 vsize=271484 CPUtime=1.33 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 5138 0 0 0 133 0 0 0 25 0 5 0 522063779 277999616 21426 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=271484 CPUtime=1.32 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 4910 0 0 0 132 0 0 0 25 0 5 0 522063779 277999616 21426 33554432000 4194304 4338100 140733401584688 18446744073709551615 274473634747 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=271484 CPUtime=1.32 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 5045 0 0 0 131 1 0 0 25 0 5 0 522063779 277999616 21426 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=271484 CPUtime=1.32 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 5153 0 0 0 132 0 0 0 25 0 5 0 522063779 277999616 21426 33554432000 4194304 4338100 140733401584688 18446744073709551615 274473362878 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 5.49
Current children cumulated vsize (KiB) 271484

[startup+3.10102 s]
/proc/loadavg: 7.16 7.91 7.99 9/186 3804
/proc/meminfo: memFree=18982880/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=328948 CPUtime=11.88 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 37344 0 0 0 1182 6 0 0 23 0 5 0 522063763 336842752 37261 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 82237 37261 175 36 0 80142 0
[pid=3792/tid=3797] ppid=3789 vsize=328948 CPUtime=2.93 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 8986 0 0 0 292 1 0 0 25 0 5 0 522063779 336842752 37261 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=328948 CPUtime=2.93 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 9408 0 0 0 292 1 0 0 25 0 5 0 522063779 336842752 37262 33554432000 4194304 4338100 140733401584688 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=328948 CPUtime=2.92 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 8729 0 0 0 291 1 0 0 25 0 5 0 522063779 336842752 37262 33554432000 4194304 4338100 140733401584688 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=328948 CPUtime=2.91 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 8959 0 0 0 290 1 0 0 25 0 5 0 522063779 336842752 37262 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.88
Current children cumulated vsize (KiB) 328948

[startup+6.30046 s]
/proc/loadavg: 7.23 7.91 7.99 10/186 3804
/proc/meminfo: memFree=18738220/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=430456 CPUtime=24.67 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 62569 0 0 0 2456 11 0 0 23 0 5 0 522063763 440786944 62486 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 107614 62486 175 36 0 105519 0
[pid=3792/tid=3797] ppid=3789 vsize=430456 CPUtime=6.13 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 15012 0 0 0 611 2 0 0 25 0 5 0 522063779 440786944 62486 33554432000 4194304 4338100 140733401584688 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=430456 CPUtime=6.13 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 16786 0 0 0 611 2 0 0 25 0 5 0 522063779 440786944 62486 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=430456 CPUtime=6.13 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 14361 0 0 0 610 3 0 0 25 0 5 0 522063779 440786944 62486 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=430456 CPUtime=6.1 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 15147 0 0 0 607 3 0 0 25 0 5 0 522063779 440786944 62486 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.67
Current children cumulated vsize (KiB) 430456

[startup+12.7003 s]
/proc/loadavg: 7.29 7.91 7.99 10/186 3804
/proc/meminfo: memFree=18382196/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=582784 CPUtime=50.23 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 100956 0 0 0 5004 19 0 0 23 0 5 0 522063763 596770816 100873 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 145696 100873 175 36 0 143601 0
[pid=3792/tid=3797] ppid=3789 vsize=582784 CPUtime=12.53 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 24063 0 0 0 1249 4 0 0 25 0 5 0 522063779 596770816 100873 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=582784 CPUtime=12.52 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 26312 0 0 0 1248 4 0 0 25 0 5 0 522063779 596770816 100873 33554432000 4194304 4338100 140733401584688 18446744073709551615 4265599 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=582784 CPUtime=12.52 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 23069 0 0 0 1247 5 0 0 25 0 5 0 522063779 596770816 100873 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=582784 CPUtime=12.46 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 26249 0 0 0 1242 4 0 0 25 0 5 0 522063779 596770816 100873 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.23
Current children cumulated vsize (KiB) 582784

[startup+25.5011 s]
/proc/loadavg: 7.51 7.93 7.99 10/186 3804
/proc/meminfo: memFree=17872672/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=792188 CPUtime=101.35 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 154830 0 0 0 10106 29 0 0 23 0 5 0 522063763 811200512 154747 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 198047 154747 175 36 0 195952 0
[pid=3792/tid=3797] ppid=3789 vsize=792188 CPUtime=25.33 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 37528 0 0 0 2527 6 0 0 25 0 5 0 522063779 811200512 154747 33554432000 4194304 4338100 140733401584688 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=792188 CPUtime=25.32 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 40593 0 0 0 2525 7 0 0 25 0 5 0 522063779 811200512 154747 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=792188 CPUtime=25.28 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 37073 0 0 0 2520 8 0 0 25 0 5 0 522063779 811200512 154747 33554432000 4194304 4338100 140733401584688 18446744073709551615 4257123 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=792188 CPUtime=25.23 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 38373 0 0 0 2517 6 0 0 25 0 5 0 522063779 811200512 154748 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 101.35
Current children cumulated vsize (KiB) 792188

[startup+51.1006 s]
/proc/loadavg: 7.68 7.93 7.99 10/186 3805
/proc/meminfo: memFree=17124944/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=1099060 CPUtime=203.6 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 233726 0 0 0 20311 49 0 0 23 0 5 0 522063763 1125437440 233643 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 274765 233643 175 36 0 272670 0
[pid=3792/tid=3797] ppid=3789 vsize=1099060 CPUtime=50.92 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 57480 0 0 0 5081 11 0 0 25 0 5 0 522063779 1125437440 233643 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=1099060 CPUtime=50.92 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 62043 0 0 0 5079 13 0 0 25 0 5 0 522063779 1125437440 233643 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=1099060 CPUtime=50.73 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 55778 0 0 0 5061 12 0 0 25 0 5 0 522063779 1125437440 233643 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=1099060 CPUtime=50.84 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 57162 0 0 0 5073 11 0 0 25 0 5 0 522063779 1125437440 233643 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 203.6
Current children cumulated vsize (KiB) 1099060

[startup+102.307 s]
/proc/loadavg: 7.96 7.97 8.00 10/186 3806
/proc/meminfo: memFree=16068200/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=1582240 CPUtime=408.08 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 356258 0 0 0 40729 79 0 0 23 0 5 0 522063763 1620213760 356167 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 395560 356167 176 36 0 393465 0
[pid=3792/tid=3797] ppid=3789 vsize=1582240 CPUtime=102.11 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 87433 0 0 0 10191 20 0 0 25 0 5 0 522063779 1620213760 356167 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=1582240 CPUtime=102.1 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 94395 0 0 0 10190 20 0 0 25 0 5 0 522063779 1620213760 356167 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=1582240 CPUtime=101.86 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 82348 0 0 0 10167 19 0 0 25 0 5 0 522063779 1620213760 356167 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=1582240 CPUtime=101.82 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 90819 0 0 0 10163 19 0 0 25 0 5 0 522063779 1620213760 356167 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 408.08
Current children cumulated vsize (KiB) 1582240

[startup+162.301 s]
/proc/loadavg: 8.07 8.00 8.01 10/186 3808
/proc/meminfo: memFree=15085108/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=2002828 CPUtime=647.71 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 462947 0 0 0 64664 107 0 0 23 0 5 0 522063763 2050895872 462856 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 500707 462856 176 36 0 498612 0
[pid=3792/tid=3797] ppid=3789 vsize=2002828 CPUtime=162.1 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 112135 0 0 0 16185 25 0 0 25 0 5 0 522063779 2050895872 462856 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=2002828 CPUtime=162.07 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 126050 0 0 0 16178 29 0 0 25 0 5 0 522063779 2050895872 462856 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=2002828 CPUtime=161.85 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 111161 0 0 0 16159 26 0 0 25 0 5 0 522063779 2050895872 462856 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=2002828 CPUtime=161.49 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 112338 0 0 0 16124 25 0 0 25 0 5 0 522063779 2050895872 462856 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 647.71
Current children cumulated vsize (KiB) 2002828


################
# More data... #
################

/proc/meminfo: memFree=9173200/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=4631956 CPUtime=3044.2 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1131468 0 0 0 304115 305 0 0 23 0 5 0 522063763 4743122944 1131377 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1157989 1131377 176 36 0 1155894 0
[pid=3792/tid=3797] ppid=3789 vsize=4631956 CPUtime=762.06 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 267433 0 0 0 76128 78 0 0 25 0 5 0 522063779 4743122944 1131377 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=4631956 CPUtime=761.67 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 311933 0 0 0 76091 76 0 0 25 0 5 0 522063779 4743122944 1131377 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=4631956 CPUtime=758.8 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 266476 0 0 0 75808 72 0 0 25 0 5 0 522063779 4743122944 1131377 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=4631956 CPUtime=761.48 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 284363 0 0 0 76071 77 0 0 25 0 5 0 522063779 4743122944 1131377 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3044.2
Current children cumulated vsize (KiB) 4631956

[startup+822.301 s]
/proc/loadavg: 8.15 8.08 8.01 10/186 3825
/proc/meminfo: memFree=8706572/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=4848700 CPUtime=3283.8 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1186543 0 0 0 328058 322 0 0 23 0 5 0 522063763 4965068800 1186452 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1212175 1186452 176 36 0 1210080 0
[pid=3792/tid=3797] ppid=3789 vsize=4848700 CPUtime=822.03 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 279039 0 0 0 82121 82 0 0 25 0 5 0 522063779 4965068800 1186452 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=4848700 CPUtime=821.65 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 328146 0 0 0 82085 80 0 0 25 0 5 0 522063779 4965068800 1186452 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=4848700 CPUtime=818.47 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 280552 0 0 0 81770 77 0 0 25 0 5 0 522063779 4965068800 1186452 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=4848700 CPUtime=821.46 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 297543 0 0 0 82064 82 0 0 25 0 5 0 522063779 4965068800 1186452 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3283.8
Current children cumulated vsize (KiB) 4848700

[startup+882.3 s]
/proc/loadavg: 8.15 8.09 8.02 10/186 3827
/proc/meminfo: memFree=8290904/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=4985988 CPUtime=3523.39 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1220027 0 0 0 352003 336 0 0 23 0 5 0 522063763 5105651712 1219936 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1246497 1219936 176 36 0 1244402 0
[pid=3792/tid=3797] ppid=3789 vsize=4985988 CPUtime=882.01 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 290515 0 0 0 88115 86 0 0 25 0 5 0 522063779 5105651712 1219936 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=4985988 CPUtime=881.65 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 337913 0 0 0 88080 85 0 0 25 0 5 0 522063779 5105651712 1219936 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=4985988 CPUtime=878.12 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 291780 0 0 0 87731 81 0 0 25 0 5 0 522063779 5105651712 1219936 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=4985988 CPUtime=881.43 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 298556 0 0 0 88060 83 0 0 25 0 5 0 522063779 5105651712 1219936 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3523.39
Current children cumulated vsize (KiB) 4985988

[startup+942.301 s]
/proc/loadavg: 8.05 8.07 8.01 10/186 3829
/proc/meminfo: memFree=7832968/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=5175460 CPUtime=3763.06 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1267026 0 0 0 375957 349 0 0 23 0 5 0 522063763 5299671040 1266935 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1293865 1266935 176 36 0 1291770 0
[pid=3792/tid=3797] ppid=3789 vsize=5175460 CPUtime=942.01 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 298998 0 0 0 94112 89 0 0 25 0 5 0 522063779 5299671040 1266935 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=5175460 CPUtime=941.65 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 356078 0 0 0 94075 90 0 0 25 0 5 0 522063779 5299671040 1266935 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=5175460 CPUtime=937.8 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 306255 0 0 0 93695 85 0 0 25 0 5 0 522063779 5299671040 1266935 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=5175460 CPUtime=941.43 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 304432 0 0 0 94059 84 0 0 25 0 5 0 522063779 5299671040 1266935 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3763.06
Current children cumulated vsize (KiB) 5175460

[startup+1002.3 s]
/proc/loadavg: 8.02 8.06 8.01 9/186 3830
/proc/meminfo: memFree=7437248/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=5361804 CPUtime=4002.75 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1315825 0 0 0 399911 364 0 0 23 0 5 0 522063763 5490487296 1315734 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1340451 1315734 176 36 0 1338356 0
[pid=3792/tid=3797] ppid=3789 vsize=5361804 CPUtime=1002.01 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 308825 0 0 0 100109 92 0 0 25 0 5 0 522063779 5490487296 1315734 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=5361804 CPUtime=1001.65 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 370266 0 0 0 100071 94 0 0 25 0 5 0 522063779 5490487296 1315734 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=5361804 CPUtime=997.46 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 318466 0 0 0 99658 88 0 0 25 0 5 0 522063779 5490487296 1315734 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=5361804 CPUtime=1001.43 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 317005 0 0 0 100055 88 0 0 25 0 5 0 522063779 5490487296 1315734 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4002.75
Current children cumulated vsize (KiB) 5361804

[startup+1062.3 s]
/proc/loadavg: 8.05 8.06 8.01 10/186 3832
/proc/meminfo: memFree=6985252/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=5563404 CPUtime=4242.42 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1366917 0 0 0 423861 381 0 0 23 0 5 0 522063763 5696925696 1366826 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1390851 1366826 176 36 0 1388756 0
[pid=3792/tid=3797] ppid=3789 vsize=5563404 CPUtime=1062.01 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 316187 0 0 0 106106 95 0 0 25 0 5 0 522063779 5696925696 1366826 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=5563404 CPUtime=1061.65 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 385955 0 0 0 106066 99 0 0 25 0 5 0 522063779 5696925696 1366826 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=5563404 CPUtime=1057.14 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 332064 0 0 0 105622 92 0 0 25 0 5 0 522063779 5696925696 1366826 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=5563404 CPUtime=1061.43 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 331448 0 0 0 106050 93 0 0 25 0 5 0 522063779 5696925696 1366826 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4242.42
Current children cumulated vsize (KiB) 5563404

[startup+1122.3 s]
/proc/loadavg: 8.10 8.07 8.01 10/186 3834
/proc/meminfo: memFree=6586820/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=5743364 CPUtime=4482.03 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1412298 0 0 0 447808 395 0 0 23 0 5 0 522063763 5881204736 1412207 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1435841 1412207 176 36 0 1433746 0
[pid=3792/tid=3797] ppid=3789 vsize=5743364 CPUtime=1121.99 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 327349 0 0 0 112100 99 0 0 25 0 5 0 522063779 5881204736 1412207 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=5743364 CPUtime=1121.63 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 398549 0 0 0 112061 102 0 0 25 0 5 0 522063779 5881204736 1412207 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=5743364 CPUtime=1116.81 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 340038 0 0 0 111586 95 0 0 25 0 5 0 522063779 5881204736 1412207 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=5743364 CPUtime=1121.41 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 345099 0 0 0 112044 97 0 0 25 0 5 0 522063779 5881204736 1412207 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4482.03
Current children cumulated vsize (KiB) 5743364

[startup+1182.3 s]
/proc/loadavg: 8.11 8.09 8.02 10/186 3835
/proc/meminfo: memFree=6145352/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=5938708 CPUtime=4721.69 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1462651 0 0 0 471760 409 0 0 23 0 5 0 522063763 6081236992 1462560 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1484677 1462560 176 36 0 1482582 0
[pid=3792/tid=3797] ppid=3789 vsize=5938708 CPUtime=1181.99 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 339726 0 0 0 118095 104 0 0 25 0 5 0 522063779 6081236992 1462560 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=5938708 CPUtime=1181.64 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 411116 0 0 0 118058 106 0 0 25 0 5 0 522063779 6081236992 1462560 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=5938708 CPUtime=1176.47 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 350782 0 0 0 117549 98 0 0 25 0 5 0 522063779 6081236992 1462560 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=5938708 CPUtime=1181.41 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 359764 0 0 0 118041 100 0 0 25 0 5 0 522063779 6081236992 1462560 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4721.69
Current children cumulated vsize (KiB) 5938708



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 8.21 8.11 8.02 10/186 3836
/proc/meminfo: memFree=6027660/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=5989268 CPUtime=4800.4 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) S 3789 3792 3320 0 -1 4202496 1475260 0 0 0 479626 414 0 0 23 0 5 0 522063763 6133010432 1475169 33554432000 4194304 4338100 140733401584688 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 1497317 1475169 176 36 0 1495222 0
[pid=3792/tid=3797] ppid=3789 vsize=5989268 CPUtime=1201.68 cores=0,2,4,6
/proc/3792/task/3797/stat : 3797 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 343249 0 0 0 120063 105 0 0 25 0 5 0 522063779 6133010432 1475169 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=3792/tid=3798] ppid=3789 vsize=5989268 CPUtime=1201.34 cores=0,2,4,6
/proc/3792/task/3798/stat : 3798 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 415042 0 0 0 120027 107 0 0 25 0 5 0 522063779 6133010432 1475169 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=5989268 CPUtime=1196.07 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 353018 0 0 0 119508 99 0 0 25 0 5 0 522063779 6133010432 1475169 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=3792/tid=3800] ppid=3789 vsize=5989268 CPUtime=1201.12 cores=0,2,4,6
/proc/3792/task/3800/stat : 3800 (ttsth-5-4) R 3789 3792 3320 0 -1 4202560 362688 0 0 0 120011 101 0 0 25 0 5 0 522063779 6133010432 1475169 33554432000 4194304 4338100 140733401584688 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.4
Current children cumulated vsize (KiB) 5989268

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.2 s]
/proc/loadavg: 8.21 8.11 8.02 4/182 3838
/proc/meminfo: memFree=15550004/32950928 swapFree=67111528/67111528
[pid=3792] ppid=3789 vsize=0 CPUtime=4800.61 cores=0,2,4,6
/proc/3792/stat : 3792 (ttsth-5-4) Z 3789 3792 3320 0 -1 4203548 1475263 0 0 0 479628 433 0 0 18 0 2 0 522063763 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/3792/statm: 0 0 0 0 0 0 0
[pid=3792/tid=3799] ppid=3789 vsize=0 CPUtime=1196.26 cores=0,2,4,6
/proc/3792/task/3799/stat : 3799 (ttsth-5-4) R 3789 3792 3320 0 -1 4203588 353018 0 0 0 119508 118 0 0 25 0 2 0 522063779 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.61
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.25
CPU time (s): 4800.67
CPU user time (s): 4796.28
CPU system time (s): 4.38133
CPU usage (%): 399.306
Max. virtual memory (cumulated for all children) (KiB): 5989268

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.28
system time used= 4.38133
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1475263
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1662
involuntary context switches= 36698

runsolver used 1.82372 second user time and 4.73528 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 21:17:21
IDJOB=3266358
IDBENCH=87455
IDSOLVER=1654
FILE ID=node125/3266358-1304104641
RUNJOBID= node125-1304104641-3775
PBS_JOBID= 13170627
Free space on /tmp= 72764 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT02/handmade/pehoushek/plainoldcnf/5cnf_4300_4300_110.shuffled.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3266358-1304104641/watcher-3266358-1304104641 -o /tmp/evaluation-result-3266358-1304104641/solver-3266358-1304104641 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266358-1304104641.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 34f8b747bfe8e7b589b8832e56ae8363
RANDOM SEED=1217696561

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19217096 kB
Buffers:       2218808 kB
Cached:       10918988 kB
SwapCached:          0 kB
Active:        8428992 kB
Inactive:      4773004 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19217096 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             156 kB
Writeback:           0 kB
AnonPages:       63172 kB
Mapped:          14400 kB
Slab:           467500 kB
PageTables:       4556 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   188008 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72764 MiB
End job on node125 at 2011-04-29 21:37:25