Trace number 3273639

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.39 1214.35

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n150-s89334302-unsat.cnf
MD5SUMe0b064ae815c6a8bb8a9898180562bec
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables153
Number of clauses316
Sum of the clauses size948
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3316
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273639-1304058268.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 153
0.00/0.00	c Clauses = 316
0.00/0.00	c Literals = 948
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 153
0.00/0.00	c Clauses = 316
0.00/0.00	c Literals = 948
0.00/0.00	c a2
0.00/0.00	c a1
1.10/1.19	c Independent sub-problems = 1
1.10/1.19	c Sub-problem with 153 variables and 316 clauses
1.10/1.19	c using permutation method 3
1.10/1.19	c Ordering finished 1.190 seconds
1.10/1.19	c mapped 1.190 seconds
1.10/1.19	c permuted 1.190 seconds
1.10/1.19	c about to sort
1.10/1.19	c sorted
1.10/1.20	c clauses now 316
1.10/1.20	c init finished 1.190 seconds
1.10/1.20	c tree built 1.190 seconds
1.10/1.20	c 
1.10/1.20	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.10/1.20	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.10/1.20	c ................................ | ..........                    |                                  1.2         0.0             
1.10/1.20	c ................................ | ..........                  1 |                                  1.2         0.8             
1.10/1.20	c ................................ | ..........                  2 |                                  1.2         1.7             
1.10/1.20	c ................................ | ..........                  3 |                                  1.2         2.5             
1.10/1.20	c ................................ | ..........                  4 |                                  1.2         3.4             
1.10/1.20	c ................................ | ..........                  5 |                                  1.2         4.2             
1.19/1.20	c ................................ | ..........               1000 | 88           420                 1.2       833.3        10628
95.64/25.08	c ................................ | ..........            2205000 | 362953       361726             96.0     22968.8      6980896
191.69/49.38	c ................................ | ..........            4315000 | 714697       695344            192.0     22472.8      9788212
287.79/73.66	c ................................ | ..........            5768000 | 970807       887407            288.0     20026.4     11977384
383.78/97.88	c ................................ | ..........            6922000 | 1168726      1062529           384.1     18021.8     13794423
480.04/122.11	c ................................ | *.........            7872000 | 1331201      1183863           480.1     16398.0     15300664
575.81/146.44	c ................................ | *.........            9127000 | 1555739      1338619           576.0     15845.5     16559873
671.88/170.83	c ................................ | *.........           10300000 | 1769720      1497930           672.0     15327.2     17016851
767.84/195.14	c ................................ | *.........           11372000 | 1960432      1658705           768.0     14806.5     18331244
864.41/219.51	c ................................ | *.........           12300000 | 2122292      1809221           864.5     14228.7     19522089
959.83/243.64	c ................................ | **........           13093000 | 2262441      1915585           960.0     13638.4     19620544
1055.95/267.92	c ................................ | **........           14099000 | 2441428      2031874          1056.0     13350.8     20633906
1151.70/292.18	c ................................ | **........           15094000 | 2641515      2130449          1152.0     13102.4     21494583
1247.69/316.48	c ................................ | **........           15887000 | 2787769      2264984          1248.0     12729.7     22029753
1343.72/340.79	c ................................ | **........           16944000 | 2977213      2415137          1344.0     12606.9     23039752
1439.91/365.04	c ................................ | ***.......           17825000 | 3143992      2562708          1440.1     12377.9     23039752
1536.04/389.31	c ................................ | ***.......           18497000 | 3253652      2701074          1536.1     12041.8     24042047
1631.88/413.53	c ................................ | ***.......           19175000 | 3375996      2804078          1632.0     11749.4     24042047
1727.92/437.84	c ................................ | ***.......           19833000 | 3495187      2895298          1728.1     11477.0     24351378
1823.85/462.14	c ................................ | ***.......           20738000 | 3655246      3048499          1824.0     11369.4     25014317
1919.89/486.54	c ................................ | ****......           21652000 | 3802614      3232123          1920.0     11276.8     25336012
2016.00/510.88	c ................................ | ****......           22405000 | 3934330      3350280          2016.3     11111.8     25834853
2111.77/535.06	c ................................ | ****......           23088000 | 4061928      3447850          2112.0     10931.8     26451230
2207.87/559.23	c ................................ | ****......           23682000 | 4176765      3529812          2208.0     10725.4     27152513
2304.13/583.55	c ................................ | ****......           24490000 | 4320549      3617106          2304.3     10627.7     27823489
2399.70/607.78	c ................................ | *****.....           25159000 | 4434478      3714243          2400.1     10482.7     28357551
2495.98/632.13	c ................................ | *****.....           25911000 | 4555555      3844545          2496.1     10380.6     28781971
2591.95/656.35	c ................................ | *****.....           26517000 | 4659125      3957655          2592.2     10229.7     29287017
2687.74/680.67	c ................................ | *****.....           27039000 | 4751488      4071418          2688.1     10059.0     29750419
2783.93/704.87	c ................................ | *****.....           27540000 | 4843304      4148391          2784.2      9891.4     30187698
2879.77/729.15	c ................................ | ******....           28199000 | 4954905      4236466          2880.0      9791.3     30562834
2975.91/753.43	c ................................ | ******....           28960000 | 5088944      4320304          2976.0      9731.1     30562834
3072.05/777.80	c ................................ | ******....           29909000 | 5246343      4422908          3072.1      9735.7     30967652
3167.76/802.16	c ................................ | ******....           30865000 | 5406116      4538701          3168.0      9742.7     31833278
3264.61/826.61	c ................................ | ******....           31600000 | 5536679      4625004          3264.7      9679.4     31833278
3360.03/850.70	c ................................ | *******...           32524000 | 5692202      4717241          3360.0      9679.6     31833278
3455.74/874.98	c ................................ | *******...           33210000 | 5813461      4800399          3456.1      9609.2     31833278
3552.05/899.31	c ................................ | *******...           34100000 | 5977063      4901807          3552.1      9599.9     32193489
3647.80/923.57	c ................................ | *******...           34937000 | 6124234      5004606          3648.1      9576.8     33092965
3743.86/947.83	c ................................ | *******...           35670000 | 6252795      5086361          3744.0      9527.2     33281251
3839.78/972.06	c ................................ | ********..           36388000 | 6378848      5155727          3840.0      9476.0     33895159
3936.12/996.40	c ................................ | ********..           37351000 | 6546851      5249873          3936.1      9489.2     34170349
4031.89/1020.63	c ................................ | ********..           37981000 | 6662115      5340374          4032.0      9419.8     34219536
4127.87/1044.85	c ................................ | ********..           38516000 | 6759751      5415815          4128.1      9330.2     34365121
4223.88/1069.04	c ................................ | ********..           39103000 | 6866420      5496653          4224.1      9257.2     34365121
4319.89/1093.26	c ................................ | *********.           39693000 | 6973415      5575394          4320.2      9187.8     34782377
4415.94/1117.43	c ................................ | *********.           40390000 | 7098176      5678395          4416.1      9146.1     34979064
4511.91/1141.62	c ................................ | *********.           41018000 | 7212346      5759146          4512.0      9090.9     34980003
4607.65/1165.89	c ................................ | *********.           41738000 | 7352701      5820841          4608.0      9057.7     35197080
4703.92/1190.04	c ................................ | *********.           42345000 | 7472951      5873237          4704.1      9001.8     35575517
4799.97/1214.28	c CPU limit (4800) reached
4799.97/1214.28	c Time = 4800.31 seconds
4799.97/1214.28	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3273639-1304058268/watcher-3273639-1304058268 -o /tmp/evaluation-result-3273639-1304058268/solver-3273639-1304058268 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273639-1304058268.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.19 8.09 8.01 5/181 22358
/proc/meminfo: memFree=21879596/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=8708 CPUtime=0 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) R 22356 22358 22182 0 -1 4202496 280 0 0 0 0 0 0 0 25 0 1 0 517423863 8916992 222 33554432000 4194304 4338100 140735812327168 18446744073709551615 239476096496 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22358/statm: 2177 222 163 36 0 82 0

[startup+0.019694 s]
/proc/loadavg: 8.19 8.09 8.01 5/181 22358
/proc/meminfo: memFree=21879596/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=8708 CPUtime=0.01 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) R 22356 22358 22182 0 -1 4202496 280 0 0 0 1 0 0 0 25 0 1 0 517423863 8916992 222 33554432000 4194304 4338100 140735812327168 18446744073709551615 239480133617 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22358/statm: 2177 222 163 36 0 82 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 8708

[startup+0.100681 s]
/proc/loadavg: 8.19 8.09 8.01 5/181 22358
/proc/meminfo: memFree=21879596/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=8708 CPUtime=0.09 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) R 22356 22358 22182 0 -1 4202496 280 0 0 0 9 0 0 0 25 0 1 0 517423863 8916992 222 33554432000 4194304 4338100 140735812327168 18446744073709551615 239476360027 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22358/statm: 2177 222 163 36 0 82 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 8708

[startup+0.30064 s]
/proc/loadavg: 8.19 8.09 8.01 5/181 22358
/proc/meminfo: memFree=21879596/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=8708 CPUtime=0.29 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) R 22356 22358 22182 0 -1 4202496 280 0 0 0 29 0 0 0 25 0 1 0 517423863 8916992 222 33554432000 4194304 4338100 140735812327168 18446744073709551615 4290902 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22358/statm: 2177 222 163 36 0 82 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 8708

[startup+0.700564 s]
/proc/loadavg: 8.19 8.09 8.01 5/181 22358
/proc/meminfo: memFree=21879596/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=8708 CPUtime=0.69 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) R 22356 22358 22182 0 -1 4202496 291 0 0 0 69 0 0 0 25 0 1 0 517423863 8916992 233 33554432000 4194304 4338100 140735812327168 18446744073709551615 4290870 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/22358/statm: 2177 233 173 36 0 82 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 8708

[startup+1.50035 s]
/proc/loadavg: 8.19 8.09 8.01 6/182 22359
/proc/meminfo: memFree=21879612/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=201504 CPUtime=2.39 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 4120 0 0 0 238 1 0 0 25 0 5 0 517423863 206340096 4062 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 50376 4064 196 36 0 48281 0
Current children cumulated CPU time (s) 2.39
Current children cumulated vsize (KiB) 201504

[startup+3.10104 s]
/proc/loadavg: 8.19 8.09 8.01 9/186 22363
/proc/meminfo: memFree=21847464/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=239632 CPUtime=8.77 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 13217 0 0 0 873 4 0 0 25 0 5 0 517423863 245383168 13159 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 59908 13159 198 36 0 57813 0
[pid=22358/tid=22360] ppid=22356 vsize=239632 CPUtime=1.89 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 3228 0 0 0 188 1 0 0 21 0 5 0 517423983 245383168 13159 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=239632 CPUtime=1.88 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 2784 0 0 0 188 0 0 0 20 0 5 0 517423983 245383168 13159 33554432000 4194304 4338100 140735812327168 18446744073709551615 4242669 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=239632 CPUtime=1.89 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 3768 0 0 0 188 1 0 0 21 0 5 0 517423983 245383168 13159 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=239632 CPUtime=1.89 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 2697 0 0 0 188 1 0 0 20 0 5 0 517423983 245383168 13159 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 8.77
Current children cumulated vsize (KiB) 239632

[startup+6.30043 s]
/proc/loadavg: 8.18 8.09 8.01 9/186 22363
/proc/meminfo: memFree=21775168/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=278860 CPUtime=21.49 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 23388 0 0 0 2142 7 0 0 25 0 5 0 517423863 285552640 23292 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 69715 23293 198 36 0 67620 0
[pid=22358/tid=22360] ppid=22356 vsize=278860 CPUtime=5.06 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 7617 0 0 0 504 2 0 0 20 0 5 0 517423983 285552640 23293 33554432000 4194304 4338100 140735812327168 18446744073709551615 239476090242 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=278860 CPUtime=5.06 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 5263 0 0 0 505 1 0 0 25 0 5 0 517423983 285552640 23293 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254226 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=278860 CPUtime=5.07 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 6442 0 0 0 506 1 0 0 25 0 5 0 517423983 285552640 23293 33554432000 4194304 4338100 140735812327168 18446744073709551615 239476090606 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=278860 CPUtime=5.07 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 3327 0 0 0 506 1 0 0 20 0 5 0 517423983 285552640 23293 33554432000 4194304 4338100 140735812327168 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 21.49
Current children cumulated vsize (KiB) 278860

[startup+12.7012 s]
/proc/loadavg: 8.16 8.09 8.01 9/186 22363
/proc/meminfo: memFree=21663960/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=360448 CPUtime=46.93 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 43670 0 0 0 4681 12 0 0 25 0 5 0 517423863 369098752 43574 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 90112 43574 198 36 0 88017 0
[pid=22358/tid=22360] ppid=22356 vsize=360448 CPUtime=11.41 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 13196 0 0 0 1137 4 0 0 25 0 5 0 517423983 369098752 43574 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=360448 CPUtime=11.39 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 9934 0 0 0 1138 1 0 0 25 0 5 0 517423983 369098752 43574 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=360448 CPUtime=11.46 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 11798 0 0 0 1143 3 0 0 25 0 5 0 517423983 369098752 43574 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=360448 CPUtime=11.43 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 8002 0 0 0 1141 2 0 0 25 0 5 0 517423983 369098752 43574 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 46.93
Current children cumulated vsize (KiB) 360448

[startup+25.5008 s]
/proc/loadavg: 8.13 8.08 8.01 9/186 22364
/proc/meminfo: memFree=21521096/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=421560 CPUtime=97.64 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 58978 0 0 0 9744 20 0 0 25 0 5 0 517423863 431677440 58882 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 105390 58882 198 36 0 103295 0
[pid=22358/tid=22360] ppid=22356 vsize=421560 CPUtime=23.97 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 18498 0 0 0 2390 7 0 0 25 0 5 0 517423983 431677440 58882 33554432000 4194304 4338100 140735812327168 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=421560 CPUtime=24.11 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 11915 0 0 0 2409 2 0 0 25 0 5 0 517423983 431677440 58882 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=421560 CPUtime=24.16 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 16330 0 0 0 2411 5 0 0 25 0 5 0 517423983 431677440 58882 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=421560 CPUtime=24.14 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 11495 0 0 0 2411 3 0 0 25 0 5 0 517423983 431677440 58882 33554432000 4194304 4338100 140735812327168 18446744073709551615 239476090081 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 97.64
Current children cumulated vsize (KiB) 421560

[startup+51.1009 s]
/proc/loadavg: 8.08 8.08 8.01 7/186 22364
/proc/meminfo: memFree=21339656/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=537348 CPUtime=198.79 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 86060 0 0 0 19845 34 0 0 25 0 5 0 517423863 550244352 85963 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 134337 85963 199 36 0 132242 0
[pid=22358/tid=22360] ppid=22356 vsize=537348 CPUtime=49.04 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 32026 0 0 0 4891 13 0 0 25 0 5 0 517423983 550244352 85963 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=537348 CPUtime=49.49 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 20110 0 0 0 4943 6 0 0 25 0 5 0 517423983 550244352 85963 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=537348 CPUtime=49.56 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 20255 0 0 0 4947 9 0 0 24 0 5 0 517423983 550244352 85963 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254257 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=537348 CPUtime=49.48 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 12929 0 0 0 4943 5 0 0 25 0 5 0 517423983 550244352 85963 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 198.79
Current children cumulated vsize (KiB) 537348

[startup+102.307 s]
/proc/loadavg: 8.03 8.06 8.00 9/186 22366
/proc/meminfo: memFree=21044364/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=687980 CPUtime=401.6 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 122425 0 0 0 40106 54 0 0 25 0 5 0 517423863 704491520 122328 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 171995 122328 199 36 0 169900 0
[pid=22358/tid=22360] ppid=22356 vsize=687980 CPUtime=99.36 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 49047 0 0 0 9914 22 0 0 25 0 5 0 517423983 704491520 122328 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=687980 CPUtime=100.33 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 22766 0 0 0 10025 8 0 0 25 0 5 0 517423983 704491520 122328 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=687980 CPUtime=100.41 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 36855 0 0 0 10026 15 0 0 25 0 5 0 517423983 704491520 122328 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=687980 CPUtime=100.26 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 13017 0 0 0 10020 6 0 0 25 0 5 0 517423983 704491520 122328 33554432000 4194304 4338100 140735812327168 18446744073709551615 239476090252 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 401.6
Current children cumulated vsize (KiB) 687980

[startup+162.301 s]
/proc/loadavg: 8.01 8.05 8.00 9/186 22367
/proc/meminfo: memFree=20749808/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=773460 CPUtime=638.33 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 144350 0 0 0 63762 71 0 0 25 0 5 0 517423863 792023040 144253 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 193365 144253 199 36 0 191270 0
[pid=22358/tid=22360] ppid=22356 vsize=773460 CPUtime=158.09 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 64737 0 0 0 15775 34 0 0 25 0 5 0 517423983 792023040 144253 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=773460 CPUtime=159.63 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 23205 0 0 0 15953 10 0 0 25 0 5 0 517423983 792023040 144253 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=773460 CPUtime=159.84 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 42525 0 0 0 15967 17 0 0 25 0 5 0 517423983 792023040 144253 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=773460 CPUtime=159.53 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 13143 0 0 0 15945 8 0 0 25 0 5 0 517423983 792023040 144253 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 638.33
Current children cumulated vsize (KiB) 773460

[startup+222.301 s]
/proc/loadavg: 8.08 8.07 8.01 9/186 22369
/proc/meminfo: memFree=20525828/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=883148 CPUtime=875.39 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 168375 0 0 0 87448 91 0 0 25 0 5 0 517423863 904343552 168278 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 220787 168278 199 36 0 218692 0
[pid=22358/tid=22360] ppid=22356 vsize=883148 CPUtime=216.87 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 72747 0 0 0 21644 43 0 0 25 0 5 0 517423983 904343552 168278 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 31242 0 0 0 69451 27 0 0 25 0 5 0 517423983 1296826368 265660 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2774
Current children cumulated vsize (KiB) 1266432

[startup+762.307 s]
/proc/loadavg: 8.02 7.99 7.99 9/186 22383
/proc/meminfo: memFree=19431812/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1279584 CPUtime=3011.1 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 271253 0 0 0 300918 192 0 0 25 0 5 0 517423863 1310294016 271156 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 319896 271156 199 36 0 317801 0
[pid=22358/tid=22360] ppid=22356 vsize=1279584 CPUtime=748.21 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 115787 0 0 0 74735 86 0 0 25 0 5 0 517423983 1310294016 271156 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1279584 CPUtime=753.56 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 37113 0 0 0 75328 28 0 0 25 0 5 0 517423983 1310294016 271156 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1279584 CPUtime=754.04 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 86274 0 0 0 75357 47 0 0 25 0 5 0 517423983 1310294016 271156 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1279584 CPUtime=754.06 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 31339 0 0 0 75378 28 0 0 25 0 5 0 517423983 1310294016 271156 33554432000 4194304 4338100 140735812327168 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3011.1
Current children cumulated vsize (KiB) 1279584

[startup+822.3 s]
/proc/loadavg: 8.02 8.00 7.99 9/186 22385
/proc/meminfo: memFree=19354356/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1341372 CPUtime=3247.66 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 287396 0 0 0 324562 204 0 0 25 0 5 0 517423863 1373564928 287299 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 335343 287299 199 36 0 333248 0
[pid=22358/tid=22360] ppid=22356 vsize=1341372 CPUtime=807.1 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 126241 0 0 0 80616 94 0 0 25 0 5 0 517423983 1373564928 287299 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1341372 CPUtime=812.64 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 40227 0 0 0 81235 29 0 0 25 0 5 0 517423983 1373564928 287299 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1341372 CPUtime=813.4 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 88407 0 0 0 81291 49 0 0 25 0 5 0 517423983 1373564928 287299 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1341372 CPUtime=813.28 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 31781 0 0 0 81298 30 0 0 25 0 5 0 517423983 1373564928 287299 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3247.66
Current children cumulated vsize (KiB) 1341372

[startup+882.301 s]
/proc/loadavg: 8.18 8.04 8.01 9/186 22387
/proc/meminfo: memFree=19203252/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1345012 CPUtime=3485.02 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 290595 0 0 0 348292 210 0 0 25 0 5 0 517423863 1377292288 290498 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 336253 290498 199 36 0 334158 0
[pid=22358/tid=22360] ppid=22356 vsize=1345012 CPUtime=865.93 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 127421 0 0 0 86496 97 0 0 25 0 5 0 517423983 1377292288 290498 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1345012 CPUtime=872.07 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 41456 0 0 0 87177 30 0 0 25 0 5 0 517423983 1377292288 290498 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1345012 CPUtime=872.98 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 88486 0 0 0 87248 50 0 0 25 0 5 0 517423983 1377292288 290498 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1345012 CPUtime=872.81 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 32492 0 0 0 87250 31 0 0 25 0 5 0 517423983 1377292288 290498 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3485.02
Current children cumulated vsize (KiB) 1345012

[startup+942.3 s]
/proc/loadavg: 8.14 8.06 8.01 9/186 22388
/proc/meminfo: memFree=19033428/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1468108 CPUtime=3722.1 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 302383 0 0 0 371987 223 0 0 25 0 5 0 517423863 1503342592 302286 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 367027 302286 199 36 0 364932 0
[pid=22358/tid=22360] ppid=22356 vsize=1468108 CPUtime=925 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 137595 0 0 0 92396 104 0 0 25 0 5 0 517423983 1503342592 302286 33554432000 4194304 4338100 140735812327168 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1468108 CPUtime=931.13 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 42486 0 0 0 93080 33 0 0 25 0 5 0 517423983 1503342592 302286 33554432000 4194304 4338100 140735812327168 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1468108 CPUtime=932.41 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 88990 0 0 0 93190 51 0 0 25 0 5 0 517423983 1503342592 302286 33554432000 4194304 4338100 140735812327168 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1468108 CPUtime=932.31 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 32572 0 0 0 93199 32 0 0 25 0 5 0 517423983 1503342592 302286 33554432000 4194304 4338100 140735812327168 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3722.1
Current children cumulated vsize (KiB) 1468108

[startup+1002.3 s]
/proc/loadavg: 8.09 8.06 8.01 9/186 22390
/proc/meminfo: memFree=18892740/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1520380 CPUtime=3959.42 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 312041 0 0 0 395710 232 0 0 25 0 5 0 517423863 1556869120 311944 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 380095 311944 199 36 0 378000 0
[pid=22358/tid=22360] ppid=22356 vsize=1520380 CPUtime=984.18 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 145480 0 0 0 98308 110 0 0 25 0 5 0 517423983 1556869120 311944 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1520380 CPUtime=990.3 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 42547 0 0 0 98996 34 0 0 25 0 5 0 517423983 1556869120 311944 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1520380 CPUtime=991.84 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 90160 0 0 0 99131 53 0 0 23 0 5 0 517423983 1556869120 311944 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1520380 CPUtime=991.87 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 33114 0 0 0 99154 33 0 0 25 0 5 0 517423983 1556869120 311944 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3959.42
Current children cumulated vsize (KiB) 1520380

[startup+1062.3 s]
/proc/loadavg: 8.07 8.06 8.01 6/182 22419
/proc/meminfo: memFree=21357540/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1544500 CPUtime=4197.3 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 314609 0 0 0 419491 239 0 0 25 0 5 0 517423863 1581568000 314512 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 386125 314512 199 36 0 384030 0
[pid=22358/tid=22360] ppid=22356 vsize=1544500 CPUtime=1043.34 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 146711 0 0 0 104221 113 0 0 25 0 5 0 517423983 1581568000 314512 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1544500 CPUtime=1049.83 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 42671 0 0 0 104948 35 0 0 18 0 5 0 517423983 1581568000 314512 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1544500 CPUtime=1051.38 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 90447 0 0 0 105084 54 0 0 25 0 5 0 517423983 1581568000 314512 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1544500 CPUtime=1051.53 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 34040 0 0 0 105117 36 0 0 25 0 5 0 517423983 1581568000 314512 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4197.3
Current children cumulated vsize (KiB) 1544500

[startup+1122.3 s]
/proc/loadavg: 7.94 8.03 8.00 9/186 22425
/proc/meminfo: memFree=21054828/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1563908 CPUtime=4435.23 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 319851 0 0 0 443278 245 0 0 25 0 5 0 517423863 1601441792 319754 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 390977 319754 199 36 0 388882 0
[pid=22358/tid=22360] ppid=22356 vsize=1563908 CPUtime=1102.75 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 151614 0 0 0 110158 117 0 0 25 0 5 0 517423983 1601441792 319754 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1563908 CPUtime=1109.47 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 42826 0 0 0 110911 36 0 0 25 0 5 0 517423983 1601441792 319754 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1563908 CPUtime=1111.1 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 90540 0 0 0 111056 54 0 0 25 0 5 0 517423983 1601441792 319754 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1563908 CPUtime=1110.68 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 34131 0 0 0 111032 36 0 0 25 0 5 0 517423983 1601441792 319754 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4435.23
Current children cumulated vsize (KiB) 1563908

[startup+1182.3 s]
/proc/loadavg: 7.69 7.96 7.98 9/186 22427
/proc/meminfo: memFree=20860480/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=1583060 CPUtime=4673.33 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) S 22356 22358 22182 0 -1 4202496 324566 0 0 0 467081 252 0 0 25 0 5 0 517423863 1621053440 324469 33554432000 4194304 4338100 140735812327168 18446744073709551615 239488497637 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 395765 324469 199 36 0 393670 0
[pid=22358/tid=22360] ppid=22356 vsize=1583060 CPUtime=1161.92 cores=0,2,4,6
/proc/22358/task/22360/stat : 22360 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 156092 0 0 0 116070 122 0 0 25 0 5 0 517423983 1621053440 324469 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=22358/tid=22361] ppid=22356 vsize=1583060 CPUtime=1169.16 cores=0,2,4,6
/proc/22358/task/22361/stat : 22361 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 42954 0 0 0 116880 36 0 0 25 0 5 0 517423983 1621053440 324469 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=1583060 CPUtime=1170.8 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 90593 0 0 0 117025 55 0 0 25 0 5 0 517423983 1621053440 324469 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=22358/tid=22363] ppid=22356 vsize=1583060 CPUtime=1170.22 cores=0,2,4,6
/proc/22358/task/22363/stat : 22363 (ttsth-5-4) R 22356 22358 22182 0 -1 4202560 34187 0 0 0 116985 37 0 0 25 0 5 0 517423983 1621053440 324469 33554432000 4194304 4338100 140735812327168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4673.33
Current children cumulated vsize (KiB) 1583060



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1214.3 s]
/proc/loadavg: 7.76 7.94 7.97 6/183 22427
/proc/meminfo: memFree=21232584/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=0 CPUtime=4800.34 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) Z 22356 22358 22182 0 -1 4203548 326217 0 0 0 479777 257 0 0 18 0 2 0 517423863 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 0 0 0 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=0 CPUtime=1202.73 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4203588 90618 0 0 0 120215 58 0 0 25 0 2 0 517423983 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1214.3 s]
/proc/loadavg: 7.76 7.94 7.97 6/183 22427
/proc/meminfo: memFree=21232584/32950928 swapFree=67087032/67111528
[pid=22358] ppid=22356 vsize=0 CPUtime=4800.34 cores=0,2,4,6
/proc/22358/stat : 22358 (ttsth-5-4) Z 22356 22358 22182 0 -1 4203548 326217 0 0 0 479777 257 0 0 18 0 2 0 517423863 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/22358/statm: 0 0 0 0 0 0 0
[pid=22358/tid=22362] ppid=22356 vsize=0 CPUtime=1202.73 cores=0,2,4,6
/proc/22358/task/22362/stat : 22362 (ttsth-5-4) R 22356 22358 22182 0 -1 4203588 90618 0 0 0 120215 58 0 0 25 0 2 0 517423983 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1214.35
CPU time (s): 4800.39
CPU user time (s): 4797.78
CPU system time (s): 2.6076
CPU usage (%): 395.306
Max. virtual memory (cumulated for all children) (KiB): 1589444

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.78
system time used= 2.6076
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 326217
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 22531
involuntary context switches= 31621

runsolver used 2.16367 second user time and 4.46532 second system time

The end

Launcher Data 

Begin job on node148 at 2011-04-29 08:24:28
IDJOB=3273639
IDBENCH=83143
IDSOLVER=1654
FILE ID=node148/3273639-1304058268
RUNJOBID= node148-1304056916-22200
PBS_JOBID= 13170435
Free space on /tmp= 71436 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n150-s89334302-unsat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3273639-1304058268/watcher-3273639-1304058268 -o /tmp/evaluation-result-3273639-1304058268/solver-3273639-1304058268 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3273639-1304058268.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e0b064ae815c6a8bb8a9898180562bec
RANDOM SEED=1479301399

node148.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      21879752 kB
Buffers:       2017960 kB
Cached:        7709992 kB
SwapCached:       3548 kB
Active:        2476504 kB
Inactive:      8031896 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      21879752 kB
SwapTotal:    67111528 kB
SwapFree:     67087032 kB
Dirty:              52 kB
Writeback:           0 kB
AnonPages:      778488 kB
Mapped:          14204 kB
Slab:           498320 kB
PageTables:       6220 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  1013216 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71436 MiB
End job on node148 at 2011-04-29 08:44:44