Trace number 3266490

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.25 1206.82

General information on the benchmark

NameSAT02/handmade/simon/satex-challenges/
par32-3.shuffled.cnf
MD5SUM0115a0e26236f3f01aa829c2004452c1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.6469
Satisfiable
(Un)Satisfiability was proved
Number of variables3176
Number of clauses10297
Sum of the clauses size27581
Maximum clause length3
Minimum clause length1
Number of clauses of size 1141
Number of clauses of size 23028
Number of clauses of size 37128
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266490-1304105233.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 3176
0.00/0.04	c Clauses = 10297
0.00/0.04	c Literals = 27581
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 3176
0.00/0.04	c Clauses = 7434
0.00/0.04	c Literals = 19790
0.00/0.04	c a2
0.00/0.05	c a1
0.09/0.18	c Independent sub-problems = 1
0.09/0.18	c Sub-problem with 2395 variables and 7434 clauses
0.09/0.18	c using permutation method 4
0.09/0.18	c count = 2
0.09/0.18	c Ordering finished 0.180 seconds
0.09/0.18	c mapped 0.180 seconds
0.09/0.18	c permuted 0.180 seconds
0.09/0.18	c about to sort
0.09/0.18	c sorted
0.19/0.21	c clauses now 7434
0.19/0.21	c init finished 0.190 seconds
0.19/0.21	c tree built 0.210 seconds
0.19/0.21	c 
0.19/0.21	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.21	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.21	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.21	c ................................ | ..........                  1 |                                  0.2         4.8             
0.19/0.21	c ................................ | ..........                  2 |                                  0.2         9.5             
0.19/0.21	c ................................ | ..........                  3 |                                  0.2        14.3             
0.19/0.21	c ................................ | ..........                  4 |                                  0.2        19.0             
0.19/0.21	c ................................ | ..........                  5 |                                  0.2        23.8             
0.19/0.22	c ................................ | ..........               1000 | 1            153                 0.2      4545.5           21
95.61/24.20	c ................................ | ..........            5490000 | 391644       1367556            96.0     57181.5      2406041
191.79/48.25	c ................................ | ..........            9509000 | 746105       2415162           192.0     49520.9      4620985
287.96/72.31	c ................................ | ..........           12003000 | 1008865      3152518           288.0     41677.1      7395187
383.66/96.38	c ................................ | ..........           13743000 | 1209422      3703166           384.0     35787.2      8691795
479.80/120.45	c ................................ | *.........           15759000 | 1427513      4286345           480.0     32829.9     10502571
575.84/144.54	c ................................ | *.........           17660000 | 1635470      4857512           576.0     30658.1     11383779
671.87/168.63	c ................................ | *.........           19293000 | 1819278      5331594           672.0     28709.4     12156433
767.87/192.73	c ................................ | *.........           20773000 | 2008658      5855558           768.0     27047.8     13112341
863.95/216.81	c ................................ | *.........           22147000 | 2182237      6342727           864.0     25632.2     13934295
959.74/240.98	c ................................ | **........           23466000 | 2350012      6829608           960.0     24442.7     14588267
1055.80/265.07	c ................................ | **........           24948000 | 2539000      7347240          1056.1     23623.7     15613150
1152.09/289.20	c ................................ | **........           26182000 | 2695701      7770763          1152.1     22725.5     16361842
1247.68/313.28	c ................................ | **........           27326000 | 2841204      8162448          1248.0     21895.8     16780611
1343.93/337.43	c ................................ | **........           28502000 | 3002290      8648230          1344.0     21206.1     17574028
1439.66/361.59	c ................................ | ***.......           29622000 | 3154729      9105780          1440.0     20570.1     18289414
1535.64/385.69	c ................................ | ***.......           30614000 | 3288438      9499326          1536.0     19930.6     18918698
1631.67/409.79	c ................................ | ***.......           31700000 | 3439387      9971612          1632.0     19423.5     19608266
1727.82/433.85	c ................................ | ***.......           32725000 | 3583569      10396818         1728.0     18937.6     21106838
1823.86/457.95	c ................................ | ***.......           33688000 | 3714381      10787148         1824.1     18468.6     22045707
1919.81/482.05	c ................................ | ****......           34820000 | 3874356      11272984         1920.0     18135.2     22748846
2016.01/506.20	c ................................ | ****......           36026000 | 4042269      11743448         2016.0     17869.9     23446542
2111.76/530.37	c ................................ | ****......           37113000 | 4196742      12194907         2112.1     17571.9     23455915
2207.82/554.67	c ................................ | ****......           38340000 | 4368249      12691095         2208.1     17363.3     23646182
2303.90/578.92	c ................................ | ****......           39423000 | 4525839      13155707         2304.0     17110.6     23979128
2399.72/603.07	c ................................ | *****.....           40522000 | 4685250      13627821         2400.0     16884.0     24057939
2495.86/627.23	c ................................ | *****.....           41687000 | 4856139      14150697         2496.0     16701.4     24656216
2591.75/651.46	c ................................ | *****.....           42670000 | 4999998      14573176         2592.0     16462.1     25250251
2687.89/675.63	c ................................ | *****.....           43810000 | 5168226      15079677         2688.0     16298.2     26382700
2783.98/699.82	c ................................ | *****.....           44865000 | 5319911      15525439         2784.1     16114.8     27310763
2879.63/723.99	c ................................ | ******....           45855000 | 5464526      15954529         2880.0     15921.9     28047166
2975.89/748.24	c ................................ | ******....           46797000 | 5604180      16366303         2976.1     15724.4     28566949
3071.92/772.35	c ................................ | ******....           47643000 | 5731457      16761565         3072.2     15508.0     28566949
3167.95/796.52	c ................................ | ******....           48490000 | 5857728      17146449         3168.1     15305.9     28566949
3263.77/820.67	c ................................ | ******....           49360000 | 5991385      17558537         3264.0     15122.4     29060794
3359.98/844.82	c ................................ | *******...           50140000 | 6110239      17931453         3360.1     14922.3     29663502
3455.89/868.94	c ................................ | *******...           50901000 | 6225704      18282863         3456.1     14728.0     29889584
3551.81/893.06	c ................................ | *******...           51716000 | 6351091      18665748         3552.1     14559.4     30364431
3647.87/917.16	c ................................ | *******...           52465000 | 6467469      19032703         3648.1     14381.3     31011132
3743.72/941.29	c ................................ | *******...           53201000 | 6582730      19378363         3744.1     14209.4     31567664
3839.63/965.39	c ................................ | ********..           54005000 | 6704669      19747027         3840.0     14063.8     31942656
3935.65/989.49	c ................................ | ********..           54759000 | 6820562      20097027         3936.0     13912.2     32488029
4031.77/1013.58	c ................................ | ********..           55484000 | 6934207      20449454         4032.1     13760.6     33178502
4127.64/1037.79	c ................................ | ********..           56137000 | 7037579      20771634         4128.0     13598.9     33323216
4223.88/1061.95	c ................................ | ********..           56777000 | 7140944      21088408         4224.1     13441.2     33533556
4320.11/1086.10	c ................................ | *********.           57406000 | 7242991      21404423         4320.1     13288.0     34230638
4415.90/1110.33	c ................................ | *********.           58122000 | 7357807      21770169         4416.0     13161.6     34468908
4512.03/1134.53	c ................................ | *********.           58815000 | 7469595      22120155         4512.2     13034.7     34628532
4607.84/1158.56	c ................................ | *********.           59472000 | 7576015      22462904         4608.1     12906.0     35734683
4704.07/1182.61	c ................................ | *********.           60183000 | 7691434      22829950         4704.1     12793.6     36935903
4799.96/1206.76	c CPU limit (4800) reached
4799.96/1206.76	c Time = 4800.17 seconds
4799.96/1206.76	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3266490-1304105233/watcher-3266490-1304105233 -o /tmp/evaluation-result-3266490-1304105233/solver-3266490-1304105233 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266490-1304105233.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.08 8.12 8.03 2/173 2669
/proc/meminfo: memFree=25622684/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=9112 CPUtime=0 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) R 2666 2669 2480 0 -1 4202496 348 0 0 0 0 0 0 0 18 0 1 0 502938732 9330688 290 33554432000 4194304 4338100 140734591754176 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2669/statm: 2278 291 147 36 0 183 0

[startup+0.0631721 s]
/proc/loadavg: 8.08 8.12 8.03 2/173 2669
/proc/meminfo: memFree=25622684/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=11244 CPUtime=0.06 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) R 2666 2669 2480 0 -1 4202496 900 0 0 0 6 0 0 0 18 0 1 0 502938732 11513856 841 33554432000 4194304 4338100 140734591754176 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2669/statm: 2811 841 155 36 0 716 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 11244

[startup+0.101142 s]
/proc/loadavg: 8.08 8.12 8.03 2/173 2669
/proc/meminfo: memFree=25622684/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=11376 CPUtime=0.09 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) R 2666 2669 2480 0 -1 4202496 910 0 0 0 9 0 0 0 18 0 1 0 502938732 11649024 851 33554432000 4194304 4338100 140734591754176 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2669/statm: 2844 851 155 36 0 749 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 11376

[startup+0.330063 s]
/proc/loadavg: 8.08 8.12 8.03 2/173 2669
/proc/meminfo: memFree=25622684/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=224188 CPUtime=0.66 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 8404 0 0 0 65 1 0 0 18 0 5 0 502938732 229568512 8344 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 56047 8344 175 36 0 53952 0
Current children cumulated CPU time (s) 0.66
Current children cumulated vsize (KiB) 224188

[startup+0.700994 s]
/proc/loadavg: 8.08 8.12 8.03 2/173 2669
/proc/meminfo: memFree=25622684/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=224452 CPUtime=2.14 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 8981 0 0 0 213 1 0 0 18 0 5 0 502938732 229838848 8921 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 56113 8921 175 36 0 54018 0
Current children cumulated CPU time (s) 2.14
Current children cumulated vsize (KiB) 224452

[startup+1.50085 s]
/proc/loadavg: 8.08 8.12 8.03 6/182 2685
/proc/meminfo: memFree=25586684/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=224716 CPUtime=5.33 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 9869 0 0 0 532 1 0 0 18 0 5 0 502938732 230109184 9809 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 56179 9809 175 36 0 54084 0
[pid=2669/tid=2682] ppid=2666 vsize=224716 CPUtime=1.28 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 504 0 0 0 128 0 0 0 19 0 5 0 502938754 230109184 9809 33554432000 4194304 4338100 140734591754176 18446744073709551615 4256993 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=224716 CPUtime=1.27 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 556 0 0 0 127 0 0 0 19 0 5 0 502938754 230109184 9809 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=224716 CPUtime=1.27 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 514 0 0 0 127 0 0 0 22 0 5 0 502938754 230109184 9809 33554432000 4194304 4338100 140734591754176 18446744073709551615 4241466 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=224716 CPUtime=1.28 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 549 0 0 0 128 0 0 0 19 0 5 0 502938754 230109184 9809 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107265283 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5.33
Current children cumulated vsize (KiB) 224716

[startup+3.10055 s]
/proc/loadavg: 7.83 8.07 8.02 6/182 2685
/proc/meminfo: memFree=25581600/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=228648 CPUtime=11.72 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 11785 0 0 0 1170 2 0 0 18 0 5 0 502938732 234135552 11725 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 57162 11725 175 36 0 55067 0
[pid=2669/tid=2682] ppid=2666 vsize=228648 CPUtime=2.87 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 963 0 0 0 287 0 0 0 24 0 5 0 502938754 234135552 11725 33554432000 4194304 4338100 140734591754176 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=228648 CPUtime=2.87 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 1080 0 0 0 287 0 0 0 23 0 5 0 502938754 234135552 11725 33554432000 4194304 4338100 140734591754176 18446744073709551615 228094846338 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=228648 CPUtime=2.86 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 979 0 0 0 286 0 0 0 25 0 5 0 502938754 234135552 11725 33554432000 4194304 4338100 140734591754176 18446744073709551615 4256131 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=228648 CPUtime=2.87 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 1017 0 0 0 287 0 0 0 19 0 5 0 502938754 234135552 11725 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.72
Current children cumulated vsize (KiB) 228648

[startup+6.30096 s]
/proc/loadavg: 7.83 8.07 8.02 9/186 2689
/proc/meminfo: memFree=25423596/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=239044 CPUtime=24.51 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 14754 0 0 0 2447 4 0 0 18 0 5 0 502938732 244781056 14694 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 59761 14694 175 36 0 57666 0
[pid=2669/tid=2682] ppid=2666 vsize=239044 CPUtime=6.07 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 1708 0 0 0 606 1 0 0 25 0 5 0 502938754 244781056 14694 33554432000 4194304 4338100 140734591754176 18446744073709551615 4286738 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=239044 CPUtime=6.07 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 1876 0 0 0 607 0 0 0 25 0 5 0 502938754 244781056 14694 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=239044 CPUtime=6.04 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 1683 0 0 0 604 0 0 0 25 0 5 0 502938754 244781056 14694 33554432000 4194304 4338100 140734591754176 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=239044 CPUtime=6.07 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 1741 0 0 0 607 0 0 0 25 0 5 0 502938754 244781056 14694 33554432000 4194304 4338100 140734591754176 18446744073709551615 228094846714 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.51
Current children cumulated vsize (KiB) 239044

[startup+12.7008 s]
/proc/loadavg: 7.94 8.08 8.02 9/186 2689
/proc/meminfo: memFree=25063236/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=262128 CPUtime=50.07 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 20117 0 0 0 5000 7 0 0 18 0 5 0 502938732 268419072 20057 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 65532 20057 175 36 0 63437 0
[pid=2669/tid=2682] ppid=2666 vsize=262128 CPUtime=12.47 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 3061 0 0 0 1245 2 0 0 25 0 5 0 502938754 268419072 20057 33554432000 4194304 4338100 140734591754176 18446744073709551615 4240513 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=262128 CPUtime=12.47 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 3399 0 0 0 1246 1 0 0 25 0 5 0 502938754 268419072 20057 33554432000 4194304 4338100 140734591754176 18446744073709551615 4287704 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=262128 CPUtime=12.4 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 2559 0 0 0 1240 0 0 0 25 0 5 0 502938754 268419072 20057 33554432000 4194304 4338100 140734591754176 18446744073709551615 4256284 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=262128 CPUtime=12.48 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 3352 0 0 0 1247 1 0 0 25 0 5 0 502938754 268419072 20057 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.07
Current children cumulated vsize (KiB) 262128

[startup+25.5004 s]
/proc/loadavg: 7.95 8.08 8.02 9/186 2690
/proc/meminfo: memFree=24811864/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=305120 CPUtime=101.18 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 31598 0 0 0 10106 12 0 0 18 0 5 0 502938732 312442880 31538 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 76280 31538 176 36 0 74185 0
[pid=2669/tid=2682] ppid=2666 vsize=305120 CPUtime=25.27 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 5799 0 0 0 2522 5 0 0 25 0 5 0 502938754 312442880 31538 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254426 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=305120 CPUtime=25.27 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 6959 0 0 0 2525 2 0 0 25 0 5 0 502938754 312442880 31538 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=305120 CPUtime=25.13 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 4613 0 0 0 2512 1 0 0 25 0 5 0 502938754 312442880 31538 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=305120 CPUtime=25.27 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 6481 0 0 0 2525 2 0 0 25 0 5 0 502938754 312442880 31538 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.18
Current children cumulated vsize (KiB) 305120

[startup+51.1007 s]
/proc/loadavg: 8.03 8.09 8.02 9/186 2690
/proc/meminfo: memFree=24474188/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=379740 CPUtime=203.36 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 52040 0 0 0 20314 22 0 0 18 0 5 0 502938732 388853760 51980 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 94935 51980 176 36 0 92840 0
[pid=2669/tid=2682] ppid=2666 vsize=379740 CPUtime=50.84 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 12269 0 0 0 5075 9 0 0 25 0 5 0 502938754 388853760 51980 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=379740 CPUtime=50.72 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 10961 0 0 0 5068 4 0 0 25 0 5 0 502938754 388853760 51980 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=379740 CPUtime=50.71 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 8998 0 0 0 5069 2 0 0 25 0 5 0 502938754 388853760 51980 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=379740 CPUtime=50.84 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 12066 0 0 0 5080 4 0 0 25 0 5 0 502938754 388853760 51980 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 203.36
Current children cumulated vsize (KiB) 379740

[startup+102.306 s]
/proc/loadavg: 7.33 7.89 7.96 9/186 2692
/proc/meminfo: memFree=23955964/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=523336 CPUtime=407.61 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 86711 0 0 0 40719 42 0 0 18 0 5 0 502938732 535896064 86651 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 130834 86651 176 36 0 128739 0
[pid=2669/tid=2682] ppid=2666 vsize=523336 CPUtime=101.96 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 25751 0 0 0 10178 18 0 0 25 0 5 0 502938754 535896064 86651 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=523336 CPUtime=101.59 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 17101 0 0 0 10151 8 0 0 25 0 5 0 502938754 535896064 86651 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=523336 CPUtime=101.86 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 15614 0 0 0 10180 6 0 0 25 0 5 0 502938754 535896064 86651 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=523336 CPUtime=101.97 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 20499 0 0 0 10189 8 0 0 25 0 5 0 502938754 535896064 86651 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254231 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.61
Current children cumulated vsize (KiB) 523336

[startup+162.301 s]
/proc/loadavg: 7.45 7.82 7.93 9/186 2693
/proc/meminfo: memFree=23496772/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=630492 CPUtime=646.73 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 114768 0 0 0 64607 66 0 0 18 0 5 0 502938732 645623808 114708 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 157623 114708 176 36 0 155528 0
[pid=2669/tid=2682] ppid=2666 vsize=630492 CPUtime=161.79 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 40832 0 0 0 16147 32 0 0 25 0 5 0 502938754 645623808 114708 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=630492 CPUtime=161.28 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 20909 0 0 0 16117 11 0 0 25 0 5 0 502938754 645623808 114708 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=630492 CPUtime=161.75 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 20766 0 0 0 16166 9 0 0 25 0 5 0 502938754 645623808 114708 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=630492 CPUtime=161.66 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 24515 0 0 0 16155 11 0 0 25 0 5 0 502938754 645623808 114708 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.73
Current children cumulated vsize (KiB) 630492


################
# More data... #
################

/proc/meminfo: memFree=19364268/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1253544 CPUtime=2793.95 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 250376 0 0 0 279166 229 0 0 18 0 5 0 502938732 1283629056 250316 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 313386 250316 176 36 0 311291 0
[pid=2669/tid=2682] ppid=2666 vsize=1253544 CPUtime=698.18 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 140483 0 0 0 69694 124 0 0 25 0 5 0 502938754 1283629056 250316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1253544 CPUtime=697.23 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 38237 0 0 0 69688 35 0 0 25 0 5 0 502938754 1283629056 250316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1253544 CPUtime=699.89 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 34896 0 0 0 69955 34 0 0 25 0 5 0 502938754 1283629056 250316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1253544 CPUtime=698.4 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 29014 0 0 0 69807 33 0 0 25 0 5 0 502938754 1283629056 250316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4258054 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2793.95
Current children cumulated vsize (KiB) 1253544

[startup+762.306 s]
/proc/loadavg: 7.40 7.51 7.71 9/186 2709
/proc/meminfo: memFree=18931616/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1312072 CPUtime=3032.01 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 263151 0 0 0 302957 244 0 0 18 0 5 0 502938732 1343561728 263091 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 328018 263091 176 36 0 325923 0
[pid=2669/tid=2682] ppid=2666 vsize=1312072 CPUtime=757.73 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 150219 0 0 0 75639 134 0 0 25 0 5 0 502938754 1343561728 263091 33554432000 4194304 4338100 140734591754176 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1312072 CPUtime=756.71 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 39269 0 0 0 75634 37 0 0 25 0 5 0 502938754 1343561728 263091 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1312072 CPUtime=759.51 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 36780 0 0 0 75915 36 0 0 25 0 5 0 502938754 1343561728 263091 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1312072 CPUtime=757.8 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 29137 0 0 0 75746 34 0 0 25 0 5 0 502938754 1343561728 263091 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3032.01
Current children cumulated vsize (KiB) 1312072

[startup+822.301 s]
/proc/loadavg: 7.90 7.64 7.74 9/186 2711
/proc/meminfo: memFree=18506640/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1328224 CPUtime=3270.54 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 268376 0 0 0 326797 257 0 0 18 0 5 0 502938732 1360101376 268316 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 332056 268316 176 36 0 329961 0
[pid=2669/tid=2682] ppid=2666 vsize=1328224 CPUtime=817.37 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 154628 0 0 0 81595 142 0 0 25 0 5 0 502938754 1360101376 268316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1328224 CPUtime=816.38 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 39389 0 0 0 81599 39 0 0 25 0 5 0 502938754 1360101376 268316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1328224 CPUtime=819.16 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 37353 0 0 0 81879 37 0 0 25 0 5 0 502938754 1360101376 268316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1328224 CPUtime=817.37 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 29260 0 0 0 81701 36 0 0 25 0 5 0 502938754 1360101376 268316 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3270.54
Current children cumulated vsize (KiB) 1328224

[startup+882.301 s]
/proc/loadavg: 7.72 7.63 7.73 9/186 2713
/proc/meminfo: memFree=18087132/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1359888 CPUtime=3509.3 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 276747 0 0 0 350658 272 0 0 18 0 5 0 502938732 1392525312 276687 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 339972 276687 176 36 0 337877 0
[pid=2669/tid=2682] ppid=2666 vsize=1359888 CPUtime=877.13 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 162677 0 0 0 87563 150 0 0 25 0 5 0 502938754 1392525312 276687 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1359888 CPUtime=876.14 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 39489 0 0 0 87573 41 0 0 25 0 5 0 502938754 1392525312 276687 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1359888 CPUtime=878.6 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 37464 0 0 0 87820 40 0 0 25 0 5 0 502938754 1392525312 276687 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1359888 CPUtime=877.18 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 29371 0 0 0 87680 38 0 0 25 0 5 0 502938754 1392525312 276687 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3509.3
Current children cumulated vsize (KiB) 1359888

[startup+942.301 s]
/proc/loadavg: 7.44 7.59 7.71 9/186 2714
/proc/meminfo: memFree=17717832/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1407196 CPUtime=3748.12 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 288352 0 0 0 374522 290 0 0 18 0 5 0 502938732 1440968704 288292 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 351799 288292 176 36 0 349704 0
[pid=2669/tid=2682] ppid=2666 vsize=1407196 CPUtime=936.79 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 173997 0 0 0 93516 163 0 0 25 0 5 0 502938754 1440968704 288292 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1407196 CPUtime=935.58 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 39578 0 0 0 93515 43 0 0 25 0 5 0 502938754 1440968704 288292 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1407196 CPUtime=938.47 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 37568 0 0 0 93805 42 0 0 25 0 5 0 502938754 1440968704 288292 33554432000 4194304 4338100 140734591754176 18446744073709551615 228095147465 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1407196 CPUtime=937.04 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 29463 0 0 0 93664 40 0 0 25 0 5 0 502938754 1440968704 288292 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3748.12
Current children cumulated vsize (KiB) 1407196

[startup+1002.3 s]
/proc/loadavg: 7.35 7.52 7.67 9/186 2716
/proc/meminfo: memFree=17361308/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1445120 CPUtime=3987.04 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 298689 0 0 0 398396 308 0 0 18 0 5 0 502938732 1479802880 298629 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 361280 298629 176 36 0 359185 0
[pid=2669/tid=2682] ppid=2666 vsize=1445120 CPUtime=996.51 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 184046 0 0 0 99477 174 0 0 25 0 5 0 502938754 1479802880 298629 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1445120 CPUtime=995.46 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 39676 0 0 0 99501 45 0 0 25 0 5 0 502938754 1479802880 298629 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1445120 CPUtime=998.26 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 37664 0 0 0 99782 44 0 0 25 0 5 0 502938754 1479802880 298629 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1445120 CPUtime=996.55 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 29557 0 0 0 99613 42 0 0 25 0 5 0 502938754 1479802880 298629 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3987.04
Current children cumulated vsize (KiB) 1445120

[startup+1062.3 s]
/proc/loadavg: 7.14 7.44 7.64 9/186 2717
/proc/meminfo: memFree=16973036/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1467404 CPUtime=4225.48 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 305546 0 0 0 422229 319 0 0 18 0 5 0 502938732 1502621696 305486 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 366851 305486 176 36 0 364756 0
[pid=2669/tid=2682] ppid=2666 vsize=1467404 CPUtime=1056.05 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 188647 0 0 0 105426 179 0 0 25 0 5 0 502938754 1502621696 305486 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1467404 CPUtime=1055.22 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 39871 0 0 0 105475 47 0 0 25 0 5 0 502938754 1502621696 305486 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1467404 CPUtime=1058.01 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 38697 0 0 0 105754 47 0 0 25 0 5 0 502938754 1502621696 305486 33554432000 4194304 4338100 140734591754176 18446744073709551615 228094845999 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1467404 CPUtime=1055.96 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 30585 0 0 0 105552 44 0 0 25 0 5 0 502938754 1502621696 305486 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4225.48
Current children cumulated vsize (KiB) 1467404

[startup+1122.3 s]
/proc/loadavg: 7.16 7.40 7.61 9/186 2719
/proc/meminfo: memFree=16518672/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1495940 CPUtime=4463.61 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 315485 0 0 0 446029 332 0 0 18 0 5 0 502938732 1531842560 315425 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 373985 315425 176 36 0 371890 0
[pid=2669/tid=2682] ppid=2666 vsize=1495940 CPUtime=1115.33 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 195019 0 0 0 111347 186 0 0 25 0 5 0 502938754 1531842560 315425 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1495940 CPUtime=1114.84 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 40765 0 0 0 111436 48 0 0 25 0 5 0 502938754 1531842560 315425 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1495940 CPUtime=1117.65 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 40296 0 0 0 111716 49 0 0 25 0 5 0 502938754 1531842560 315425 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1495940 CPUtime=1115.54 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 31659 0 0 0 111507 47 0 0 25 0 5 0 502938754 1531842560 315425 33554432000 4194304 4338100 140734591754176 18446744073709551615 228094846380 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4463.61
Current children cumulated vsize (KiB) 1495940

[startup+1182.3 s]
/proc/loadavg: 7.50 7.44 7.61 7/186 2721
/proc/meminfo: memFree=16139340/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=1574396 CPUtime=4702.88 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) S 2666 2669 2480 0 -1 4202496 335799 0 0 0 469940 348 0 0 18 0 5 0 502938732 1612181504 335739 33554432000 4194304 4338100 140734591754176 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 393599 335739 176 36 0 391504 0
[pid=2669/tid=2682] ppid=2666 vsize=1574396 CPUtime=1174.9 cores=1,3,5,7
/proc/2669/task/2682/stat : 2682 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 206109 0 0 0 117296 194 0 0 25 0 5 0 502938754 1612181504 335739 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=1574396 CPUtime=1174.76 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 41348 0 0 0 117426 50 0 0 25 0 5 0 502938754 1612181504 335739 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2669/tid=2684] ppid=2666 vsize=1574396 CPUtime=1177.56 cores=1,3,5,7
/proc/2669/task/2684/stat : 2684 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 40623 0 0 0 117706 50 0 0 25 0 5 0 502938754 1612181504 335739 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2669/tid=2685] ppid=2666 vsize=1574396 CPUtime=1175.42 cores=1,3,5,7
/proc/2669/task/2685/stat : 2685 (ttsth-5-4) R 2666 2669 2480 0 -1 4202560 39973 0 0 0 117490 52 0 0 25 0 5 0 502938754 1612181504 335739 33554432000 4194304 4338100 140734591754176 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4702.88
Current children cumulated vsize (KiB) 1574396



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1206.8 s]
/proc/loadavg: 7.53 7.44 7.60 6/183 2721
/proc/meminfo: memFree=16840444/32951124 swapFree=67111524/67111528
[pid=2669] ppid=2666 vsize=0 CPUtime=4800.22 cores=1,3,5,7
/proc/2669/stat : 2669 (ttsth-5-4) Z 2666 2669 2480 0 -1 4203548 338516 0 0 0 479664 358 0 0 18 0 2 0 502938732 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2669/statm: 0 0 0 0 0 0 0
[pid=2669/tid=2683] ppid=2666 vsize=0 CPUtime=1199.2 cores=1,3,5,7
/proc/2669/task/2683/stat : 2683 (ttsth-5-4) R 2666 2669 2480 0 -1 4203588 41379 0 0 0 119865 55 0 0 25 0 2 0 502938754 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 1206.82
CPU time (s): 4800.25
CPU user time (s): 4796.65
CPU system time (s): 3.60145
CPU usage (%): 397.759
Max. virtual memory (cumulated for all children) (KiB): 1583908

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.65
system time used= 3.60145
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 338516
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10216
involuntary context switches= 29189

runsolver used 2.02969 second user time and 4.55031 second system time

The end

Launcher Data 

Begin job on node142 at 2011-04-29 21:27:13
IDJOB=3266490
IDBENCH=87739
IDSOLVER=1654
FILE ID=node142/3266490-1304105233
RUNJOBID= node142-1304104028-2499
PBS_JOBID= 13170645
Free space on /tmp= 71316 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT02/handmade/simon/satex-challenges/par32-3.shuffled.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3266490-1304105233/watcher-3266490-1304105233 -o /tmp/evaluation-result-3266490-1304105233/solver-3266490-1304105233 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266490-1304105233.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0115a0e26236f3f01aa829c2004452c1
RANDOM SEED=76911704

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25622948 kB
Buffers:       1682720 kB
Cached:        5038844 kB
SwapCached:          4 kB
Active:         874328 kB
Inactive:      5911444 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25622948 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:              32 kB
Writeback:           0 kB
AnonPages:       63960 kB
Mapped:          14700 kB
Slab:           478124 kB
PageTables:       4676 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   188432 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71312 MiB
End job on node142 at 2011-04-29 21:47:21