Trace number 3260066

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.11 1207.4

General information on the benchmark

Namecrafted/mosoi/
289/289-sat-14x8.cnf
MD5SUM8ef06ba92b977ea1ab969c2727216040
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.010998
Satisfiable
(Un)Satisfiability was proved
Number of variables448
Number of clauses10976
Sum of the clauses size42560
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2672
Number of clauses of size 30
Number of clauses of size 410304
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260066-1304062369.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 448
0.00/0.01	c Clauses = 10976
0.00/0.01	c Literals = 42560
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 448
0.00/0.01	c Clauses = 10976
0.00/0.01	c Literals = 42560
0.00/0.01	c a2
0.00/0.03	c a1
0.49/0.51	c Independent sub-problems = 1
0.49/0.51	c Sub-problem with 448 variables and 10976 clauses
0.49/0.51	c using permutation method 4
0.49/0.51	c count = 2
0.49/0.51	c Ordering finished 0.510 seconds
0.49/0.51	c mapped 0.510 seconds
0.49/0.51	c permuted 0.510 seconds
0.49/0.51	c about to sort
0.49/0.51	c sorted
0.49/0.53	c clauses now 10976
0.49/0.53	c init finished 0.520 seconds
0.49/0.53	c tree built 0.530 seconds
0.49/0.53	c 
0.49/0.53	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.53	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.53	c ................................ | ..........                    |                                  0.5         0.0             
0.49/0.53	c ................................ | ..........                  1 |                                  0.5         1.9             
0.49/0.53	c ................................ | ..........                  2 |                                  0.5         3.8             
0.49/0.53	c ................................ | ..........                  3 |                                  0.5         5.7             
0.49/0.53	c ................................ | ..........                  4 |                                  0.5         7.5             
0.49/0.53	c ................................ | ..........                  5 |                                  0.5         9.4             
0.49/0.54	c ................................ | ..........               1000 | 18           2093                0.6      1818.2         4983
95.77/24.47	c ................................ | ..........             740000 | 169928       1269889            96.1      7703.5      7229044
191.91/48.53	c ................................ | ..........            1243000 | 309986       1939933           192.1      6472.3     10641028
288.03/72.64	c ................................ | ..........            1765000 | 475056       2487337           288.2      6124.6     12900273
383.93/96.73	c ................................ | ..........            2445000 | 710188       2911250           384.1      6366.4     15317622
480.04/120.93	c ................................ | *.........            3214000 | 964673       3600159           480.1      6693.7     19681734
576.01/145.03	c ................................ | *.........            3755000 | 1135100      4166095           576.1      6517.6     22260416
671.73/169.19	c ................................ | *.........            4324000 | 1280448      4991830           672.1      6433.6     26835553
768.01/193.31	c ................................ | *.........            4782000 | 1400183      5647581           768.1      6226.0     29959941
863.95/217.44	c ................................ | *.........            5209000 | 1518351      6221249           864.1      6028.0     31627411
959.88/241.55	c ................................ | **........            5620000 | 1637172      6727410           960.1      5853.7     32454295
1055.80/265.65	c ................................ | **........            6019000 | 1753150      7198109          1056.0      5699.8     34060922
1151.91/289.76	c ................................ | **........            6438000 | 1880772      7660746          1152.2      5587.7     37059110
1248.05/313.91	c ................................ | **........            6831000 | 2004807      8045789          1248.1      5473.2     37946970
1343.84/338.03	c ................................ | **........            7227000 | 2131462      8428592          1344.0      5377.2     38401812
1439.79/362.17	c ................................ | ***.......            7659000 | 2269901      8808470          1440.1      5318.5     38944855
1535.78/386.37	c ................................ | ***.......            8117000 | 2421894      9174761          1536.1      5284.2     39253028
1631.94/410.55	c ................................ | ***.......            8579000 | 2581016      9504037          1632.2      5256.2     40544985
1727.75/434.66	c ................................ | ***.......            9055000 | 2742630      9835751          1728.0      5240.1     41943524
1823.80/458.89	c ................................ | ***.......            9588000 | 2925830      10149349         1824.2      5256.1     42603852
1919.90/483.04	c ................................ | ****......           10137000 | 3115117      10426159         1920.1      5279.4     44023698
2015.93/507.13	c ................................ | ****......           10476000 | 3233595      10680910         2016.0      5196.3     44518882
2112.20/531.30	c ................................ | ****......           10846000 | 3363099      10913181         2112.2      5134.9     45755907
2207.79/555.49	c ................................ | ****......           11233000 | 3485446      11264854         2208.1      5087.1     48148630
2303.92/579.63	c ................................ | ****......           11569000 | 3585192      11655604         2304.0      5021.2     49600550
2399.85/603.79	c ................................ | *****.....           11870000 | 3671829      12004084         2400.2      4945.4     51250336
2496.16/627.91	c ................................ | *****.....           12174000 | 3758627      12369919         2496.2      4877.0     53268601
2592.02/652.02	c ................................ | *****.....           12501000 | 3863701      12674867         2592.1      4822.7     54941195
2688.02/676.11	c ................................ | *****.....           12823000 | 3959525      13008794         2688.1      4770.3     55948028
2784.15/700.30	c ................................ | *****.....           13130000 | 4054332      13321287         2784.2      4716.0     56814376
2879.93/724.48	c ................................ | ******....           13439000 | 4151942      13620743         2880.3      4665.9     58238646
2975.87/748.56	c ................................ | ******....           13746000 | 4250480      13903232         2976.1      4618.7     58396024
3072.09/772.70	c ................................ | ******....           14091000 | 4361688      14207838         3072.1      4586.8     58571235
3168.23/796.96	c ................................ | ******....           14478000 | 4489472      14498550         3168.2      4569.8     58796688
3264.08/821.10	c ................................ | ******....           14830000 | 4607314      14760457         3264.1      4543.4     58796688
3359.81/845.36	c ................................ | *******...           15208000 | 4733079      15031647         3360.1      4526.1     59322295
3455.96/869.52	c ................................ | *******...           15646000 | 4881435      15297710         3456.1      4527.1     59758599
3551.98/893.78	c ................................ | *******...           16129000 | 5046877      15526327         3552.3      4540.4     61135679
3647.72/917.78	c ................................ | *******...           16360000 | 5118484      15769087         3648.1      4484.6     63246973
3743.95/941.97	c ................................ | *******...           16591000 | 5188128      16017406         3744.2      4431.1     65581969
3839.86/966.04	c ................................ | ********..           16806000 | 5251969      16247559         3840.0      4376.5     66804941
3936.05/990.24	c ................................ | ********..           17037000 | 5320245      16498642         3936.2      4328.2     67790213
4032.02/1014.31	c ................................ | ********..           17282000 | 5397908      16732734         4032.1      4286.1     68123278
4127.94/1038.46	c ................................ | ********..           17529000 | 5474066      16962980         4128.2      4246.2     68704125
4223.82/1062.56	c ................................ | ********..           17793000 | 5552715      17231478         4224.1      4212.3     69256761
4320.29/1086.76	c ................................ | *********.           18032000 | 5626600      17468813         4320.5      4173.6     70011860
4416.37/1110.83	c ................................ | *********.           18265000 | 5700643      17680957         4416.5      4135.6     71077729
4511.90/1134.96	c ................................ | *********.           18495000 | 5774638      17880927         4512.2      4098.9     72532082
4607.94/1159.08	c ................................ | *********.           18749000 | 5856279      18105958         4608.3      4068.6     72705730
4704.00/1183.23	c ................................ | *********.           19025000 | 5942308      18360693         4704.2      4044.3     72764479
4799.66/1207.29	c CPU limit (4800) reached
4799.66/1207.29	c Time = 4800.00 seconds
4799.66/1207.29	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3260066-1304062369/watcher-3260066-1304062369 -o /tmp/evaluation-result-3260066-1304062369/solver-3260066-1304062369 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260066-1304062369.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.70 4.62 6.21 2/173 31587
/proc/meminfo: memFree=15118440/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) R 31584 31587 31098 0 -1 4202496 402 0 0 0 0 0 0 0 24 0 1 0 517836533 9633792 345 33554432000 4194304 4338100 140734173725504 18446744073709551615 274473967360 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/31587/statm: 2352 345 147 36 0 257 0

[startup+0.0553119 s]
/proc/loadavg: 3.70 4.62 6.21 2/173 31587
/proc/meminfo: memFree=15118440/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=12020 CPUtime=0.05 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) R 31584 31587 31098 0 -1 4202496 1091 0 0 0 5 0 0 0 24 0 1 0 517836533 12308480 1033 33554432000 4194304 4338100 140734173725504 18446744073709551615 4295863 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/31587/statm: 3005 1033 155 36 0 910 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 12020

[startup+0.100305 s]
/proc/loadavg: 3.70 4.62 6.21 2/173 31587
/proc/meminfo: memFree=15118440/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=12020 CPUtime=0.09 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) R 31584 31587 31098 0 -1 4202496 1091 0 0 0 9 0 0 0 24 0 1 0 517836533 12308480 1033 33554432000 4194304 4338100 140734173725504 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/31587/statm: 3005 1033 155 36 0 910 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12020

[startup+0.300269 s]
/proc/loadavg: 3.70 4.62 6.21 2/173 31587
/proc/meminfo: memFree=15118440/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=12020 CPUtime=0.29 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) R 31584 31587 31098 0 -1 4202496 1091 0 0 0 29 0 0 0 25 0 1 0 517836533 12308480 1033 33554432000 4194304 4338100 140734173725504 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/31587/statm: 3005 1033 155 36 0 910 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 12020

[startup+0.700201 s]
/proc/loadavg: 3.70 4.62 6.21 2/173 31587
/proc/meminfo: memFree=15118440/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=201580 CPUtime=1.18 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 3963 0 0 0 118 0 0 0 25 0 5 0 517836533 206417920 3883 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 50395 3883 175 36 0 48300 0
Current children cumulated CPU time (s) 1.18
Current children cumulated vsize (KiB) 201580

[startup+1.50106 s]
/proc/loadavg: 3.70 4.62 6.21 9/186 31601
/proc/meminfo: memFree=15070388/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=224136 CPUtime=4.38 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 9062 0 0 0 436 2 0 0 25 0 5 0 517836533 229515264 8982 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 56034 8982 175 36 0 53939 0
[pid=31587/tid=31593] ppid=31584 vsize=224136 CPUtime=0.96 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 1250 0 0 0 96 0 0 0 25 0 5 0 517836587 229515264 8982 33554432000 4194304 4338100 140734173725504 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=224136 CPUtime=0.95 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 1935 0 0 0 95 0 0 0 25 0 5 0 517836587 229515264 8982 33554432000 4194304 4338100 140734173725504 18446744073709551615 274473362913 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=224136 CPUtime=0.95 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 1819 0 0 0 95 0 0 0 25 0 5 0 517836587 229515264 8982 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=224136 CPUtime=0.95 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 1819 0 0 0 95 0 0 0 25 0 5 0 517836587 229515264 8982 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4.38
Current children cumulated vsize (KiB) 224136

[startup+3.10078 s]
/proc/loadavg: 3.70 4.62 6.21 9/186 31601
/proc/meminfo: memFree=15011784/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=322140 CPUtime=10.75 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 18627 0 0 0 1071 4 0 0 25 0 5 0 517836533 329871360 18547 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 80535 18547 175 36 0 78440 0
[pid=31587/tid=31593] ppid=31584 vsize=322140 CPUtime=2.54 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 3178 0 0 0 254 0 0 0 25 0 5 0 517836587 329871360 18547 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=322140 CPUtime=2.55 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 4535 0 0 0 254 1 0 0 25 0 5 0 517836587 329871360 18547 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=322140 CPUtime=2.55 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 4241 0 0 0 254 1 0 0 25 0 5 0 517836587 329871360 18548 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=322140 CPUtime=2.54 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 4435 0 0 0 254 0 0 0 25 0 5 0 517836587 329871360 18549 33554432000 4194304 4338100 140734173725504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 10.75
Current children cumulated vsize (KiB) 322140

[startup+6.30022 s]
/proc/loadavg: 4.05 4.67 6.21 9/186 31601
/proc/meminfo: memFree=14926972/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=363956 CPUtime=23.53 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 29346 0 0 0 2341 12 0 0 25 0 5 0 517836533 372690944 29266 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 90989 29266 175 36 0 88894 0
[pid=31587/tid=31593] ppid=31584 vsize=363956 CPUtime=5.74 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 5518 0 0 0 572 2 0 0 25 0 5 0 517836587 372690944 29266 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=363956 CPUtime=5.75 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 7167 0 0 0 572 3 0 0 25 0 5 0 517836587 372690944 29266 33554432000 4194304 4338100 140734173725504 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=363956 CPUtime=5.73 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 7301 0 0 0 570 3 0 0 25 0 5 0 517836587 372690944 29266 33554432000 4194304 4338100 140734173725504 18446744073709551615 274473362913 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=363956 CPUtime=5.75 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 7121 0 0 0 573 2 0 0 25 0 5 0 517836587 372690944 29266 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 23.53
Current children cumulated vsize (KiB) 363956

[startup+12.7011 s]
/proc/loadavg: 4.36 4.73 6.22 9/186 31601
/proc/meminfo: memFree=14780244/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=428028 CPUtime=49.07 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 46356 0 0 0 4877 30 0 0 25 0 5 0 517836533 438300672 46276 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 107007 46276 175 36 0 104912 0
[pid=31587/tid=31593] ppid=31584 vsize=428028 CPUtime=12.15 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 9549 0 0 0 1207 8 0 0 25 0 5 0 517836587 438300672 46277 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485780174 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=428028 CPUtime=12.11 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 11124 0 0 0 1202 9 0 0 25 0 5 0 517836587 438300672 46277 33554432000 4194304 4338100 140734173725504 18446744073709551615 4284164 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=428028 CPUtime=12.12 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 11877 0 0 0 1206 6 0 0 25 0 5 0 517836587 438300672 46277 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=428028 CPUtime=12.13 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 11568 0 0 0 1208 5 0 0 25 0 5 0 517836587 438300672 46277 33554432000 4194304 4338100 140734173725504 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 49.07
Current children cumulated vsize (KiB) 428028

[startup+25.5008 s]
/proc/loadavg: 5.17 4.89 6.25 9/186 31601
/proc/meminfo: memFree=14564012/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=505788 CPUtime=100.12 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 66263 0 0 0 9942 70 0 0 25 0 5 0 517836533 517926912 66183 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 126447 66183 176 36 0 124352 0
[pid=31587/tid=31593] ppid=31584 vsize=505788 CPUtime=24.92 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 14126 0 0 0 2471 21 0 0 25 0 5 0 517836587 517926912 66183 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254244 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=505788 CPUtime=24.83 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 15656 0 0 0 2464 19 0 0 25 0 5 0 517836587 517926912 66183 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=505788 CPUtime=24.89 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 17671 0 0 0 2475 14 0 0 25 0 5 0 517836587 517926912 66183 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=505788 CPUtime=24.91 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 16571 0 0 0 2477 14 0 0 25 0 5 0 517836587 517926912 66183 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 100.12
Current children cumulated vsize (KiB) 505788

[startup+51.1003 s]
/proc/loadavg: 6.13 5.14 6.30 9/186 31602
/proc/meminfo: memFree=14247060/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=614080 CPUtime=202.27 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 95847 0 0 0 20059 168 0 0 25 0 5 0 517836533 628817920 95767 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 153520 95767 176 36 0 151425 0
[pid=31587/tid=31593] ppid=31584 vsize=614080 CPUtime=50.49 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 22993 0 0 0 5000 49 0 0 25 0 5 0 517836587 628817920 95767 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=614080 CPUtime=50.27 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 21566 0 0 0 4980 47 0 0 25 0 5 0 517836587 628817920 95767 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=614080 CPUtime=50.45 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 25594 0 0 0 5010 35 0 0 25 0 5 0 517836587 628817920 95767 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=614080 CPUtime=50.5 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 23455 0 0 0 5014 36 0 0 25 0 5 0 517836587 628817920 95767 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 202.27
Current children cumulated vsize (KiB) 614080

[startup+102.306 s]
/proc/loadavg: 7.27 5.59 6.39 9/186 31603
/proc/meminfo: memFree=13719568/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=830624 CPUtime=406.2 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 152370 0 0 0 40256 364 0 0 25 0 5 0 517836533 850558976 152290 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 207656 152290 176 36 0 205561 0
[pid=31587/tid=31593] ppid=31584 vsize=830624 CPUtime=101.25 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 44234 0 0 0 10019 106 0 0 25 0 5 0 517836587 850558976 152290 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=830624 CPUtime=101.33 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 28653 0 0 0 10037 96 0 0 25 0 5 0 517836587 850558976 152290 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=830624 CPUtime=101.53 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 41632 0 0 0 10075 78 0 0 25 0 5 0 517836587 850558976 152290 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=830624 CPUtime=101.51 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 35612 0 0 0 10070 81 0 0 25 0 5 0 517836587 850558976 152290 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 406.2
Current children cumulated vsize (KiB) 830624

[startup+162.301 s]
/proc/loadavg: 7.84 6.06 6.50 9/186 31605
/proc/meminfo: memFree=13182276/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=1105188 CPUtime=644.79 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 224831 0 0 0 63926 553 0 0 25 0 5 0 517836533 1131712512 224751 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 276297 224751 176 36 0 274202 0
[pid=31587/tid=31593] ppid=31584 vsize=1105188 CPUtime=160.58 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 76135 0 0 0 15908 150 0 0 25 0 5 0 517836587 1131712512 224751 33554432000 4194304 4338100 140734173725504 18446744073709551615 274473664137 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=1105188 CPUtime=161.02 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 43832 0 0 0 15950 152 0 0 25 0 5 0 517836587 1131712512 224751 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=1105188 CPUtime=161.34 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 52160 0 0 0 16015 119 0 0 25 0 5 0 517836587 1131712512 224751 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=1105188 CPUtime=161.28 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 50465 0 0 0 15998 130 0 0 25 0 5 0 517836587 1131712512 224751 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 644.79
Current children cumulated vsize (KiB) 1105188


################
# More data... #
################

/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 98497 0 0 0 69331 554 0 0 25 0 5 0 517836587 2221195264 506145 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2792.14
Current children cumulated vsize (KiB) 2169136

[startup+762.307 s]
/proc/loadavg: 8.02 7.78 7.22 9/186 31621
/proc/meminfo: memFree=11316220/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2221428 CPUtime=3030.79 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 520925 0 0 0 300586 2493 0 0 25 0 5 0 517836533 2274742272 520845 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 555357 520845 176 36 0 553262 0
[pid=31587/tid=31593] ppid=31584 vsize=2221428 CPUtime=755.73 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 226072 0 0 0 75000 573 0 0 25 0 5 0 517836587 2274742272 520845 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2221428 CPUtime=757.28 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 67912 0 0 0 74980 748 0 0 25 0 5 0 517836587 2274742272 520845 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2221428 CPUtime=758.62 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 122321 0 0 0 75280 582 0 0 25 0 5 0 517836587 2274742272 520845 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2221428 CPUtime=758.59 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 102381 0 0 0 75271 588 0 0 25 0 5 0 517836587 2274742272 520845 33554432000 4194304 4338100 140734173725504 18446744073709551615 4286331 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3030.79
Current children cumulated vsize (KiB) 2221428

[startup+822.301 s]
/proc/loadavg: 8.04 7.83 7.27 9/186 31622
/proc/meminfo: memFree=11245164/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2232924 CPUtime=3268.88 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 525958 0 0 0 324183 2705 0 0 25 0 5 0 517836533 2286514176 525878 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 558231 525878 176 36 0 556136 0
[pid=31587/tid=31593] ppid=31584 vsize=2232924 CPUtime=814.81 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 229201 0 0 0 80855 626 0 0 20 0 5 0 517836587 2286514176 525878 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2232924 CPUtime=816.93 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 68524 0 0 0 80887 806 0 0 25 0 5 0 517836587 2286514176 525878 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2232924 CPUtime=818.28 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 122864 0 0 0 81197 631 0 0 25 0 5 0 517836587 2286514176 525878 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2232924 CPUtime=818.3 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 103130 0 0 0 81190 640 0 0 25 0 5 0 517836587 2286514176 525878 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3268.88
Current children cumulated vsize (KiB) 2232924

[startup+882.301 s]
/proc/loadavg: 8.04 7.87 7.32 9/186 31624
/proc/meminfo: memFree=11161324/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2282680 CPUtime=3506.78 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 538597 0 0 0 347755 2923 0 0 25 0 5 0 517836533 2337464320 538517 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 570670 538517 176 36 0 568575 0
[pid=31587/tid=31593] ppid=31584 vsize=2282680 CPUtime=873.89 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 239109 0 0 0 86703 686 0 0 25 0 5 0 517836587 2337464320 538517 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2282680 CPUtime=876.53 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 69129 0 0 0 86788 865 0 0 25 0 5 0 517836587 2337464320 538517 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2282680 CPUtime=877.9 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 124314 0 0 0 87111 679 0 0 25 0 5 0 517836587 2337464320 538517 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2282680 CPUtime=877.9 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 103806 0 0 0 87098 692 0 0 25 0 5 0 517836587 2337464320 538517 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254226 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3506.78
Current children cumulated vsize (KiB) 2282680

[startup+942.3 s]
/proc/loadavg: 8.16 7.93 7.37 9/186 31625
/proc/meminfo: memFree=10881704/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2459828 CPUtime=3745.55 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 584172 0 0 0 371480 3075 0 0 25 0 5 0 517836533 2518863872 584092 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 614957 584092 176 36 0 612862 0
[pid=31587/tid=31593] ppid=31584 vsize=2459828 CPUtime=933.27 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 256937 0 0 0 92596 731 0 0 25 0 5 0 517836587 2518863872 584092 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2459828 CPUtime=936.32 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 83317 0 0 0 92724 908 0 0 25 0 5 0 517836587 2518863872 584092 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2459828 CPUtime=937.68 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 128339 0 0 0 93058 710 0 0 25 0 5 0 517836587 2518863872 584092 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254244 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2459828 CPUtime=937.71 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 113340 0 0 0 93047 724 0 0 25 0 5 0 517836587 2518863872 584092 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3745.55
Current children cumulated vsize (KiB) 2459828

[startup+1002.3 s]
/proc/loadavg: 8.15 7.95 7.41 9/186 31627
/proc/meminfo: memFree=10747892/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2533488 CPUtime=3984.12 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 603851 0 0 0 395218 3194 0 0 25 0 5 0 517836533 2594291712 603771 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 633372 603771 176 36 0 631277 0
[pid=31587/tid=31593] ppid=31584 vsize=2533488 CPUtime=992.55 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 267023 0 0 0 98488 767 0 0 25 0 5 0 517836587 2594291712 603771 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2533488 CPUtime=996.06 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 87639 0 0 0 98667 939 0 0 25 0 5 0 517836587 2594291712 603771 33554432000 4194304 4338100 140734173725504 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2533488 CPUtime=997.49 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 130309 0 0 0 99014 735 0 0 25 0 5 0 517836587 2594291712 603771 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2533488 CPUtime=997.47 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 116641 0 0 0 98995 752 0 0 25 0 5 0 517836587 2594291712 603771 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3984.12
Current children cumulated vsize (KiB) 2533488

[startup+1062.3 s]
/proc/loadavg: 8.12 7.98 7.45 9/186 31629
/proc/meminfo: memFree=10663924/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2578996 CPUtime=4223.01 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 616742 0 0 0 418941 3360 0 0 25 0 5 0 517836533 2640891904 616662 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 644749 616662 176 36 0 642654 0
[pid=31587/tid=31593] ppid=31584 vsize=2578996 CPUtime=1051.93 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 277438 0 0 0 104378 815 0 0 25 0 5 0 517836587 2640891904 616662 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2578996 CPUtime=1055.86 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 88095 0 0 0 104604 982 0 0 25 0 5 0 517836587 2640891904 616662 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2578996 CPUtime=1057.26 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 131734 0 0 0 104954 772 0 0 25 0 5 0 517836587 2640891904 616662 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2578996 CPUtime=1057.4 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 117236 0 0 0 104951 789 0 0 18 0 5 0 517836587 2640891904 616662 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4223.01
Current children cumulated vsize (KiB) 2578996

[startup+1122.3 s]
/proc/loadavg: 8.04 7.98 7.48 9/186 31630
/proc/meminfo: memFree=10513368/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2665432 CPUtime=4462.17 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 640368 0 0 0 442726 3491 0 0 25 0 5 0 517836533 2729402368 640288 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 666358 640288 176 36 0 664263 0
[pid=31587/tid=31593] ppid=31584 vsize=2665432 CPUtime=1111.41 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 286233 0 0 0 110289 852 0 0 25 0 5 0 517836587 2729402368 640288 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2665432 CPUtime=1115.74 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 90400 0 0 0 110558 1016 0 0 25 0 5 0 517836587 2729402368 640288 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2665432 CPUtime=1117.13 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 142096 0 0 0 110911 802 0 0 25 0 5 0 517836587 2729402368 640288 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2665432 CPUtime=1117.31 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 119400 0 0 0 110912 819 0 0 25 0 5 0 517836587 2729402368 640288 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4462.17
Current children cumulated vsize (KiB) 2665432

[startup+1182.3 s]
/proc/loadavg: 8.10 8.01 7.52 9/186 31632
/proc/meminfo: memFree=10419864/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=2760836 CPUtime=4700.41 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) S 31584 31587 31098 0 -1 4202496 648970 0 0 0 466403 3638 0 0 25 0 5 0 517836533 2827096064 648890 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 690209 648890 176 36 0 688114 0
[pid=31587/tid=31593] ppid=31584 vsize=2760836 CPUtime=1170.69 cores=1,3,5,7
/proc/31587/task/31593/stat : 31593 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 290106 0 0 0 116176 893 0 0 25 0 5 0 517836587 2827096064 648890 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=31587/tid=31594] ppid=31584 vsize=2760836 CPUtime=1175.44 cores=1,3,5,7
/proc/31587/task/31594/stat : 31594 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 91913 0 0 0 116489 1055 0 0 25 0 5 0 517836587 2827096064 648890 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=31587/tid=31595] ppid=31584 vsize=2760836 CPUtime=1176.88 cores=1,3,5,7
/proc/31587/task/31595/stat : 31595 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 143206 0 0 0 116852 836 0 0 25 0 5 0 517836587 2827096064 648890 33554432000 4194304 4338100 140734173725504 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=2760836 CPUtime=1176.83 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4202560 121506 0 0 0 116831 852 0 0 25 0 5 0 517836587 2827096064 648890 33554432000 4194304 4338100 140734173725504 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4700.41
Current children cumulated vsize (KiB) 2760836



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1207.3 s]
/proc/loadavg: 8.07 8.00 7.53 3/179 31661
/proc/meminfo: memFree=12543084/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=0 CPUtime=4800.01 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) Z 31584 31587 31098 0 -1 4203548 649732 0 0 0 476272 3729 0 0 18 0 2 0 517836533 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 0 0 0 0 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=0 CPUtime=1201.68 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4203588 121731 0 0 0 119294 874 0 0 25 0 2 0 517836587 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.01
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1207.3 s]
/proc/loadavg: 8.07 8.00 7.53 3/179 31661
/proc/meminfo: memFree=12543084/32950928 swapFree=67111528/67111528
[pid=31587] ppid=31584 vsize=0 CPUtime=4800.01 cores=1,3,5,7
/proc/31587/stat : 31587 (ttsth-5-4) Z 31584 31587 31098 0 -1 4203548 649732 0 0 0 476272 3729 0 0 18 0 2 0 517836533 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/31587/statm: 0 0 0 0 0 0 0
[pid=31587/tid=31596] ppid=31584 vsize=0 CPUtime=1201.68 cores=1,3,5,7
/proc/31587/task/31596/stat : 31596 (ttsth-5-4) R 31584 31587 31098 0 -1 4203588 121731 0 0 0 119294 874 0 0 25 0 2 0 517836587 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.01
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1207.4
CPU time (s): 4800.11
CPU user time (s): 4762.72
CPU system time (s): 37.3893
CPU usage (%): 397.558
Max. virtual memory (cumulated for all children) (KiB): 2761628

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4762.72
system time used= 37.3893
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 649732
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 125787
involuntary context switches= 33225

runsolver used 1.9427 second user time and 4.64229 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 09:32:49
IDJOB=3260066
IDBENCH=83168
IDSOLVER=1654
FILE ID=node125/3260066-1304062369
RUNJOBID= node125-1304062369-31572
PBS_JOBID= 13170441
Free space on /tmp= 72764 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-sat-14x8.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3260066-1304062369/watcher-3260066-1304062369 -o /tmp/evaluation-result-3260066-1304062369/solver-3260066-1304062369 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3260066-1304062369.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 8ef06ba92b977ea1ab969c2727216040
RANDOM SEED=1372614850

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      15118480 kB
Buffers:       2219216 kB
Cached:       15017984 kB
SwapCached:          0 kB
Active:        9119324 kB
Inactive:      8181956 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      15118480 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             140 kB
Writeback:           0 kB
AnonPages:       63420 kB
Mapped:          14492 kB
Slab:           468716 kB
PageTables:       4316 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   186700 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72756 MiB
End job on node125 at 2011-04-29 09:52:58