Trace number 3262750

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.12 1202.11

General information on the benchmark

NameSAT03/handmade/gomes/bqwh/
bqwh.60.1152.shuffled-as.sat03-1650.cnf
MD5SUM98d6317b259165bb4fb7a39e783668e0
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark118.001
Satisfiable
(Un)Satisfiability was proved
Number of variables7534
Number of clauses72812
Sum of the clauses size161368
Maximum clause length14
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 269406
Number of clauses of size 3128
Number of clauses of size 4291
Number of clauses of size 5538
Number of clauses of size over 52449

Solver Data 

0.04/0.05	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262750-1304094101.cnf 
0.04/0.05	c Using SATTIMEOUT of 4800
0.04/0.05	c Using SATRAM of 15500
0.04/0.05	c Size of subset node = 32
0.04/0.05	c Variables = 7534
0.04/0.05	c Clauses = 72812
0.04/0.05	c Literals = 161368
0.04/0.05	c Reduced size:
0.04/0.05	c Variables = 7534
0.04/0.05	c Clauses = 72812
0.04/0.05	c Literals = 161368
0.04/0.05	c a2
0.09/0.12	c a1
0.19/0.28	c Independent sub-problems = 1
0.19/0.28	c Sub-problem with 7534 variables and 72812 clauses
0.19/0.28	c using permutation method 4
0.19/0.28	c count = 1
0.19/0.28	c Ordering finished 0.260 seconds
0.19/0.28	c mapped 0.280 seconds
0.19/0.28	c permuted 0.280 seconds
0.19/0.28	c about to sort
0.19/0.28	c sorted
0.39/0.49	c clauses now 72812
0.39/0.49	c init finished 0.360 seconds
0.39/0.49	c tree built 0.480 seconds
0.39/0.49	c 
0.39/0.49	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.49	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.49	c ................................ | ..........                    |                                  0.5         0.0             
0.39/0.49	c ................................ | ..........                  1 |                                  0.5         2.1             
0.39/0.49	c ................................ | ..........                  2 |                                  0.5         4.2             
0.39/0.49	c ................................ | ..........                  3 |                                  0.5         6.2             
0.39/0.49	c ................................ | ..........                  4 |                                  0.5         8.3             
0.39/0.49	c ................................ | ..........                  5 |                                  0.5        10.4             
0.50/0.50	c ................................ | ..........               1000 |              58                  0.5      2000.0           52
95.96/24.41	c ................................ | ..........            4634000 | 86547        2679494            96.0     48255.8       584768
191.85/48.44	c ................................ | ..........            8323000 | 152899       4819690           192.0     43344.4      1003300
287.70/72.47	c ................................ | ..........           12083000 | 227784       7018797           288.0     41954.9      1396301
383.97/96.50	c ................................ | ..........           15511000 | 301318       9281190           384.0     40393.2      1690657
479.83/120.54	c ................................ | *.........           18477000 | 377112       11150906          480.0     38492.9      1987067
575.70/144.58	c ................................ | *.........           20870000 | 472013       12834068          576.0     36232.0      2312618
671.97/168.61	c ................................ | *.........           22932000 | 557230       14307690          672.0     34124.0      2706209
767.83/192.65	c ................................ | *.........           25036000 | 641221       15551143          768.0     32597.3      3058853
863.73/216.67	c ................................ | *.........           27624000 | 692373       16883283          864.0     31971.9      3489126
959.98/240.71	c ................................ | **........           29667000 | 732191       18038931          960.0     30901.8      3945315
1055.85/264.74	c ................................ | **........           31420000 | 775340       19269968         1056.0     29752.9      4331465
1151.71/288.77	c ................................ | **........           33414000 | 823231       20536441         1152.0     29005.2      4696874
1247.98/312.82	c ................................ | **........           35100000 | 854023       21668911         1248.1     28123.4      5079878
1343.84/336.84	c ................................ | **........           37119000 | 887499       22901514         1344.0     27618.3      5368962
1439.74/360.87	c ................................ | ***.......           39462000 | 952526       24086768         1440.0     27404.0      5761269
1535.59/384.90	c ................................ | ***.......           41071000 | 992598       25057605         1536.0     26738.9      6178317
1631.85/408.94	c ................................ | ***.......           42445000 | 1019600      25983992         1632.0     26007.6      6821685
1727.72/432.97	c ................................ | ***.......           43356000 | 1040518      26728580         1728.0     25090.0      7804968
1823.98/457.00	c ................................ | ***.......           44257000 | 1064159      27481167         1824.0     24263.6      8334450
1919.85/481.05	c ................................ | ****......           45340000 | 1092698      28389555         1920.1     23613.8      8839261
2015.72/505.09	c ................................ | ****......           46099000 | 1112557      29031553         2016.1     22865.4      9775014
2112.01/529.12	c ................................ | ****......           46895000 | 1134466      29727017         2112.1     22203.3     10339741
2207.86/553.14	c ................................ | ****......           47700000 | 1157963      30362665         2208.0     21602.9     10959249
2304.13/577.20	c ................................ | ****......           48479000 | 1182305      31071031         2304.1     21040.0     11289890
2400.00/601.24	c ................................ | *****.....           49386000 | 1206722      31780643         2400.2     20576.0     11902981
2495.87/625.27	c ................................ | *****.....           49935000 | 1225072      32352866         2496.2     20004.7     12749167
2591.74/649.28	c ................................ | *****.....           50592000 | 1243333      33115523         2592.1     19518.0     13483923
2688.01/673.33	c ................................ | *****.....           51346000 | 1265547      33930843         2688.1     19100.9     14064881
2783.89/697.33	c ................................ | *****.....           52263000 | 1291157      34699775         2784.0     18772.5     14608389
2879.74/721.39	c ................................ | ******....           53138000 | 1312333      35361027         2880.1     18450.0     14858005
2976.01/745.49	c ................................ | ******....           54838000 | 1348384      36163775         2976.4     18424.5     15139412
3071.88/769.46	c ................................ | ******....           55662000 | 1370517      36723057         3072.1     18118.4     15747134
3167.75/793.46	c ................................ | ******....           56167000 | 1387022      37242453         3168.0     17729.4     16163725
3264.02/817.55	c ................................ | ******....           56971000 | 1407055      37815274         3264.2     17453.0     16368890
3359.91/841.57	c ................................ | *******...           57420000 | 1421411      38338014         3360.2     17088.4     16765209
3456.18/865.63	c ................................ | *******...           57800000 | 1434234      38885883         3456.3     16723.1     17218999
3551.63/889.59	c ................................ | *******...           58144000 | 1446065      39357332         3552.0     16369.4     17821398
3648.30/913.74	c ................................ | *******...           58564000 | 1457859      39761639         3648.5     16051.6     18355117
3743.77/937.69	c ................................ | *******...           58906000 | 1468635      40170723         3744.1     15732.9     18670838
3840.04/961.70	c ................................ | ********..           59351000 | 1480930      40782732         3840.1     15455.7     19095545
3935.91/985.78	c ................................ | ********..           59870000 | 1495682      41208276         3936.2     15209.9     19691048
4032.20/1009.84	c ................................ | ********..           60361000 | 1509136      41560771         4032.3     14969.3     20524120
4128.05/1033.82	c ................................ | ********..           60626000 | 1517003      42084436         4128.2     14686.0     20663521
4223.92/1057.84	c ................................ | ********..           61526000 | 1537684      42914263         4224.1     14565.6     20846204
4320.19/1081.94	c ................................ | *********.           62682000 | 1563791      43857631         4320.4     14508.5     21115989
4416.05/1105.94	c ................................ | *********.           63288000 | 1578374      44732882         4416.2     14330.8     21448570
4511.93/1129.95	c ................................ | *********.           63827000 | 1594964      45344440         4512.1     14145.7     22166215
4608.19/1154.04	c ................................ | *********.           64133000 | 1606638      45997815         4608.4     13916.6     22370080
4704.09/1178.01	c ................................ | *********.           64399000 | 1617078      46642930         4704.1     13689.9     22478432
4799.93/1202.02	c CPU limit (4800) reached
4799.93/1202.02	c Time = 4800.01 seconds
4799.93/1202.02	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3262750-1304094101/watcher-3262750-1304094101 -o /tmp/evaluation-result-3262750-1304094101/solver-3262750-1304094101 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262750-1304094101.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.27 7.66 7.88 2/177 16356
/proc/meminfo: memFree=30925380/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=9544 CPUtime=0 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) R 16354 16356 16318 0 -1 4202496 452 0 0 0 0 0 0 0 24 0 1 0 521008820 9773056 395 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/16356/statm: 2386 396 147 36 0 291 0

[startup+0.047425 s]
/proc/loadavg: 6.27 7.66 7.88 2/177 16356
/proc/meminfo: memFree=30925380/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=17084 CPUtime=0.04 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) R 16354 16356 16318 0 -1 4202496 2288 0 0 0 4 0 0 0 24 0 1 0 521008820 17494016 2231 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/16356/statm: 4271 2233 148 36 0 2176 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 17084

[startup+0.10045 s]
/proc/loadavg: 6.27 7.66 7.88 2/177 16356
/proc/meminfo: memFree=30925380/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=26124 CPUtime=0.09 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) R 16354 16356 16318 0 -1 4202496 4558 0 0 0 9 0 0 0 24 0 1 0 521008820 26750976 4501 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635302 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/16356/statm: 6531 4501 151 36 0 4436 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26124

[startup+0.300416 s]
/proc/loadavg: 6.27 7.66 7.88 2/177 16356
/proc/meminfo: memFree=30925380/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=32728 CPUtime=0.29 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) R 16354 16356 16318 0 -1 4202496 6691 0 0 0 28 1 0 0 25 0 1 0 521008820 33513472 6205 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813900997 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/16356/statm: 8182 6218 160 36 0 6087 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 32728

[startup+0.700291 s]
/proc/loadavg: 6.27 7.66 7.88 2/177 16356
/proc/meminfo: memFree=30925380/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=322348 CPUtime=1.29 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 33328 0 0 0 123 6 0 0 25 0 5 0 521008820 330084352 32841 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 80587 32841 175 36 0 78492 0
Current children cumulated CPU time (s) 1.29
Current children cumulated vsize (KiB) 322348

[startup+1.50116 s]
/proc/loadavg: 6.27 7.66 7.88 6/182 16361
/proc/meminfo: memFree=30784980/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=324196 CPUtime=4.49 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 34368 0 0 0 439 10 0 0 25 0 5 0 521008820 331976704 33880 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 81049 33880 176 36 0 78954 0
[pid=16356/tid=16358] ppid=16354 vsize=324196 CPUtime=1 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 663 0 0 0 97 3 0 0 19 0 5 0 521008869 331976704 33880 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=324196 CPUtime=1 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 689 0 0 0 100 0 0 0 19 0 5 0 521008869 331976704 33880 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256084 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=324196 CPUtime=0.99 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 594 0 0 0 97 2 0 0 18 0 5 0 521008869 331976704 33880 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826066101 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=324196 CPUtime=1 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 580 0 0 0 100 0 0 0 19 0 5 0 521008869 331976704 33880 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635209 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4.49
Current children cumulated vsize (KiB) 324196

[startup+3.10089 s]
/proc/loadavg: 6.27 7.66 7.88 10/186 16365
/proc/meminfo: memFree=30698268/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=326308 CPUtime=10.88 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 35878 0 0 0 1074 14 0 0 25 0 5 0 521008820 334139392 35390 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 81577 35390 176 36 0 79482 0
[pid=16356/tid=16358] ppid=16354 vsize=326308 CPUtime=2.6 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 1005 0 0 0 255 5 0 0 23 0 5 0 521008869 334139392 35390 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=326308 CPUtime=2.59 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 1166 0 0 0 259 0 0 0 22 0 5 0 521008869 334139392 35390 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635209 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=326308 CPUtime=2.58 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 918 0 0 0 254 4 0 0 21 0 5 0 521008869 334139392 35390 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=326308 CPUtime=2.6 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 947 0 0 0 260 0 0 0 22 0 5 0 521008869 334139392 35390 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635302 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 10.88
Current children cumulated vsize (KiB) 326308

[startup+6.30035 s]
/proc/loadavg: 6.41 7.67 7.88 10/186 16365
/proc/meminfo: memFree=30654264/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=330928 CPUtime=23.67 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 38939 0 0 0 2340 27 0 0 25 0 5 0 521008820 338870272 38451 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 82732 38451 176 36 0 80637 0
[pid=16356/tid=16358] ppid=16354 vsize=330928 CPUtime=5.79 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 1734 0 0 0 568 11 0 0 25 0 5 0 521008869 338870272 38451 33554432000 4194304 4338100 140735546313344 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=330928 CPUtime=5.8 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 1966 0 0 0 579 1 0 0 25 0 5 0 521008869 338870272 38451 33554432000 4194304 4338100 140735546313344 18446744073709551615 4257010 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=330928 CPUtime=5.76 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 1593 0 0 0 568 8 0 0 25 0 5 0 521008869 338870272 38451 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635152 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=330928 CPUtime=5.8 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 1804 0 0 0 579 1 0 0 25 0 5 0 521008869 338870272 38451 33554432000 4194304 4338100 140735546313344 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 23.67
Current children cumulated vsize (KiB) 330928

[startup+12.7003 s]
/proc/loadavg: 6.53 7.67 7.88 10/186 16365
/proc/meminfo: memFree=30583336/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=342504 CPUtime=49.23 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 45533 0 0 0 4876 47 0 0 25 0 5 0 521008820 350724096 45045 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 85626 45045 176 36 0 83531 0
[pid=16356/tid=16358] ppid=16354 vsize=342504 CPUtime=12.19 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 3153 0 0 0 1198 21 0 0 25 0 5 0 521008869 350724096 45045 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=342504 CPUtime=12.19 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 3858 0 0 0 1218 1 0 0 25 0 5 0 521008869 350724096 45045 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635321 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=342504 CPUtime=12.12 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 3007 0 0 0 1195 17 0 0 25 0 5 0 521008869 350724096 45045 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=342504 CPUtime=12.19 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 3673 0 0 0 1218 1 0 0 25 0 5 0 521008869 350724096 45045 33554432000 4194304 4338100 140735546313344 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 49.23
Current children cumulated vsize (KiB) 342504

[startup+25.5011 s]
/proc/loadavg: 6.86 7.69 7.88 10/186 16365
/proc/meminfo: memFree=30470344/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=364296 CPUtime=100.36 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 56237 0 0 0 9948 88 0 0 25 0 5 0 521008820 373039104 55749 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 91074 55749 176 36 0 88979 0
[pid=16356/tid=16358] ppid=16354 vsize=364296 CPUtime=24.99 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 5576 0 0 0 2458 41 0 0 25 0 5 0 521008869 373039104 55749 33554432000 4194304 4338100 140735546313344 18446744073709551615 4207478 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=364296 CPUtime=24.99 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 7089 0 0 0 2496 3 0 0 25 0 5 0 521008869 373039104 55749 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=364296 CPUtime=24.86 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 5440 0 0 0 2452 34 0 0 25 0 5 0 521008869 373039104 55749 33554432000 4194304 4338100 140735546313344 18446744073709551615 4286383 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=364296 CPUtime=24.99 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 6290 0 0 0 2496 3 0 0 25 0 5 0 521008869 373039104 55749 33554432000 4194304 4338100 140735546313344 18446744073709551615 4284144 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 100.36
Current children cumulated vsize (KiB) 364296

[startup+51.1008 s]
/proc/loadavg: 7.25 7.71 7.88 10/186 16366
/proc/meminfo: memFree=30296588/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=401452 CPUtime=202.63 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 74769 0 0 0 20119 144 0 0 25 0 5 0 521008820 411086848 74281 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 100363 74281 176 36 0 98268 0
[pid=16356/tid=16358] ppid=16354 vsize=401452 CPUtime=50.59 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 9991 0 0 0 4992 67 0 0 25 0 5 0 521008869 411086848 74281 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256161 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=401452 CPUtime=50.6 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 11378 0 0 0 5053 7 0 0 25 0 5 0 521008869 411086848 74281 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=401452 CPUtime=50.31 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 9738 0 0 0 4976 55 0 0 25 0 5 0 521008869 411086848 74281 33554432000 4194304 4338100 140735546313344 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=401452 CPUtime=50.6 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 11820 0 0 0 5052 8 0 0 25 0 5 0 521008869 411086848 74281 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 202.63
Current children cumulated vsize (KiB) 401452

[startup+102.306 s]
/proc/loadavg: 7.81 7.79 7.90 10/186 16368
/proc/meminfo: memFree=29979500/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=651656 CPUtime=407.16 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 110145 0 0 0 40372 344 0 0 25 0 5 0 521008820 667295744 109657 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 162914 109658 176 36 0 160819 0
[pid=16356/tid=16358] ppid=16354 vsize=651656 CPUtime=101.79 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 17291 0 0 0 10008 171 0 0 25 0 5 0 521008869 667295744 109658 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635336 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=651656 CPUtime=101.57 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 20999 0 0 0 10145 12 0 0 25 0 5 0 521008869 667295744 109658 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826052814 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=651656 CPUtime=101.46 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 16951 0 0 0 10007 139 0 0 25 0 5 0 521008869 667295744 109658 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813901345 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=651656 CPUtime=101.8 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 23063 0 0 0 10165 15 0 0 25 0 5 0 521008869 667295744 109658 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635302 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.16
Current children cumulated vsize (KiB) 651656

[startup+162.301 s]
/proc/loadavg: 8.05 7.85 7.91 10/186 16369
/proc/meminfo: memFree=29644796/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=702420 CPUtime=646.81 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 144586 0 0 0 64123 558 0 0 25 0 5 0 521008820 719278080 144098 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 175605 144098 176 36 0 173510 0
[pid=16356/tid=16358] ppid=16354 vsize=702420 CPUtime=161.78 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 24817 0 0 0 15903 275 0 0 25 0 5 0 521008869 719278080 144098 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=702420 CPUtime=161.38 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 27838 0 0 0 16121 17 0 0 25 0 5 0 521008869 719278080 144098 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=702420 CPUtime=161.32 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 24219 0 0 0 15900 232 0 0 25 0 5 0 521008869 719278080 144098 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=702420 CPUtime=161.8 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 35870 0 0 0 16153 27 0 0 25 0 5 0 521008869 719278080 144098 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 646.81
Current children cumulated vsize (KiB) 702420


################
# More data... #
################

/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 114060 0 0 0 70092 89 0 0 25 0 5 0 521008869 1677094912 375018 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2803.85
Current children cumulated vsize (KiB) 1637788

[startup+762.306 s]
/proc/loadavg: 8.25 8.21 8.07 10/186 16385
/proc/meminfo: memFree=27482392/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=1747472 CPUtime=3043.54 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 394109 0 0 0 302496 1858 0 0 25 0 5 0 521008820 1789411328 393621 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 436868 393621 176 36 0 434773 0
[pid=16356/tid=16358] ppid=16354 vsize=1747472 CPUtime=760.7 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 75787 0 0 0 75164 906 0 0 25 0 5 0 521008869 1789411328 393621 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=1747472 CPUtime=761.4 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 92590 0 0 0 76074 66 0 0 25 0 5 0 521008869 1789411328 393621 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=1747472 CPUtime=759.1 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 74557 0 0 0 75123 787 0 0 25 0 5 0 521008869 1789411328 393621 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=1747472 CPUtime=761.81 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 119333 0 0 0 76088 93 0 0 25 0 5 0 521008869 1789411328 393621 33554432000 4194304 4338100 140735546313344 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3043.54
Current children cumulated vsize (KiB) 1747472

[startup+822.3 s]
/proc/loadavg: 8.09 8.17 8.06 10/186 16387
/proc/meminfo: memFree=27348080/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=1779004 CPUtime=3283.19 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 409251 0 0 0 326409 1910 0 0 25 0 5 0 521008820 1821700096 408763 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 444751 408763 176 36 0 442656 0
[pid=16356/tid=16358] ppid=16354 vsize=1779004 CPUtime=820.37 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 78391 0 0 0 81107 930 0 0 25 0 5 0 521008869 1821700096 408763 33554432000 4194304 4338100 140735546313344 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=1779004 CPUtime=821.39 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 97900 0 0 0 82069 70 0 0 25 0 5 0 521008869 1821700096 408763 33554432000 4194304 4338100 140735546313344 18446744073709551615 4229660 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=1779004 CPUtime=819.1 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 78408 0 0 0 81102 808 0 0 25 0 5 0 521008869 1821700096 408763 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254864 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=1779004 CPUtime=821.81 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 122710 0 0 0 82084 97 0 0 25 0 5 0 521008869 1821700096 408763 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3283.19
Current children cumulated vsize (KiB) 1779004

[startup+882.3 s]
/proc/loadavg: 8.09 8.15 8.06 10/186 16389
/proc/meminfo: memFree=27184252/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=1820936 CPUtime=3522.87 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 426487 0 0 0 350325 1962 0 0 25 0 5 0 521008820 1864638464 425999 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 455234 425999 176 36 0 453139 0
[pid=16356/tid=16358] ppid=16354 vsize=1820936 CPUtime=880.04 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 80992 0 0 0 87054 950 0 0 25 0 5 0 521008869 1864638464 425999 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=1820936 CPUtime=881.39 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 103404 0 0 0 88065 74 0 0 25 0 5 0 521008869 1864638464 425999 33554432000 4194304 4338100 140735546313344 18446744073709551615 251813635119 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=1820936 CPUtime=879.09 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 82227 0 0 0 87082 827 0 0 25 0 5 0 521008869 1864638464 425999 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=1820936 CPUtime=881.81 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 128022 0 0 0 88077 104 0 0 25 0 5 0 521008869 1864638464 425999 33554432000 4194304 4338100 140735546313344 18446744073709551615 4242344 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3522.87
Current children cumulated vsize (KiB) 1820936

[startup+942.3 s]
/proc/loadavg: 8.29 8.19 8.08 10/186 16390
/proc/meminfo: memFree=27043480/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=1872424 CPUtime=3762.54 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 442889 0 0 0 374252 2002 0 0 25 0 5 0 521008820 1917362176 442401 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 468106 442401 176 36 0 466011 0
[pid=16356/tid=16358] ppid=16354 vsize=1872424 CPUtime=939.72 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 83015 0 0 0 93004 968 0 0 25 0 5 0 521008869 1917362176 442401 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=1872424 CPUtime=941.39 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 108638 0 0 0 94061 78 0 0 25 0 5 0 521008869 1917362176 442401 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=1872424 CPUtime=939.09 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 85735 0 0 0 93067 842 0 0 25 0 5 0 521008869 1917362176 442401 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=1872424 CPUtime=941.81 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 133659 0 0 0 94073 108 0 0 25 0 5 0 521008869 1917362176 442401 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3762.54
Current children cumulated vsize (KiB) 1872424

[startup+1002.3 s]
/proc/loadavg: 8.30 8.22 8.10 10/186 16392
/proc/meminfo: memFree=26911636/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=1992352 CPUtime=4002.22 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 462017 0 0 0 398180 2042 0 0 25 0 5 0 521008820 2040168448 461529 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 498088 461529 176 36 0 495993 0
[pid=16356/tid=16358] ppid=16354 vsize=1992352 CPUtime=999.39 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 84522 0 0 0 98954 985 0 0 25 0 5 0 521008869 2040168448 461529 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=1992352 CPUtime=1001.4 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 115990 0 0 0 100057 83 0 0 25 0 5 0 521008869 2040168448 461529 33554432000 4194304 4338100 140735546313344 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=1992352 CPUtime=999.1 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 88340 0 0 0 99054 856 0 0 25 0 5 0 521008869 2040168448 461529 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=1992352 CPUtime=1001.82 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 141323 0 0 0 100069 113 0 0 25 0 5 0 521008869 2040168448 461529 33554432000 4194304 4338100 140735546313344 18446744073709551615 4240554 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4002.22
Current children cumulated vsize (KiB) 1992352

[startup+1062.3 s]
/proc/loadavg: 8.21 8.20 8.10 10/186 16393
/proc/meminfo: memFree=26759212/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=2025824 CPUtime=4241.9 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 477882 0 0 0 422078 2112 0 0 25 0 5 0 521008820 2074443776 477394 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 506456 477394 176 36 0 504361 0
[pid=16356/tid=16358] ppid=16354 vsize=2025824 CPUtime=1059.07 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 86420 0 0 0 104892 1015 0 0 25 0 5 0 521008869 2074443776 477394 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=2025824 CPUtime=1061.39 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 118006 0 0 0 106055 84 0 0 25 0 5 0 521008869 2074443776 477394 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=2025824 CPUtime=1059.09 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 91045 0 0 0 105027 882 0 0 25 0 5 0 521008869 2074443776 477394 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=2025824 CPUtime=1061.82 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 150569 0 0 0 106058 124 0 0 25 0 5 0 521008869 2074443776 477394 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4241.9
Current children cumulated vsize (KiB) 2025824

[startup+1122.3 s]
/proc/loadavg: 8.21 8.21 8.10 10/186 16395
/proc/meminfo: memFree=26616084/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=2213500 CPUtime=4481.57 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 500468 0 0 0 445965 2192 0 0 25 0 5 0 521008820 2266624000 499980 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 553375 499980 176 36 0 551280 0
[pid=16356/tid=16358] ppid=16354 vsize=2213500 CPUtime=1118.74 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 88589 0 0 0 110822 1052 0 0 25 0 5 0 521008869 2266624000 499980 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=2213500 CPUtime=1121.4 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 123919 0 0 0 112052 88 0 0 25 0 5 0 521008869 2266624000 499980 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=2213500 CPUtime=1119.1 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 94727 0 0 0 110996 914 0 0 25 0 5 0 521008869 2266624000 499980 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=2213500 CPUtime=1121.81 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 161391 0 0 0 112049 132 0 0 25 0 5 0 521008869 2266624000 499980 33554432000 4194304 4338100 140735546313344 18446744073709551615 4256661 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4481.57
Current children cumulated vsize (KiB) 2213500

[startup+1182.3 s]
/proc/loadavg: 8.08 8.17 8.09 10/186 16397
/proc/meminfo: memFree=26509784/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=2242776 CPUtime=4721.25 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) S 16354 16356 16318 0 -1 4202496 514600 0 0 0 469896 2229 0 0 25 0 5 0 521008820 2296602624 514112 33554432000 4194304 4338100 140735546313344 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 560694 514112 176 36 0 558599 0
[pid=16356/tid=16358] ppid=16354 vsize=2242776 CPUtime=1178.41 cores=1,3,5,7
/proc/16356/task/16358/stat : 16358 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 89819 0 0 0 116776 1065 0 0 25 0 5 0 521008869 2296602624 514112 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=16356/tid=16359] ppid=16354 vsize=2242776 CPUtime=1181.39 cores=1,3,5,7
/proc/16356/task/16359/stat : 16359 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 127306 0 0 0 118048 91 0 0 25 0 5 0 521008869 2296602624 514112 33554432000 4194304 4338100 140735546313344 18446744073709551615 4253933 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=2242776 CPUtime=1179.1 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 96970 0 0 0 116985 925 0 0 25 0 5 0 521008869 2296602624 514112 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=16356/tid=16361] ppid=16354 vsize=2242776 CPUtime=1181.82 cores=1,3,5,7
/proc/16356/task/16361/stat : 16361 (ttsth-5-4) R 16354 16356 16318 0 -1 4202560 168663 0 0 0 118040 142 0 0 25 0 5 0 521008869 2296602624 514112 33554432000 4194304 4338100 140735546313344 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4721.25
Current children cumulated vsize (KiB) 2242776



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.1 s]
/proc/loadavg: 8.05 8.16 8.09 6/183 16397
/proc/meminfo: memFree=28321032/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) Z 16354 16356 16318 0 -1 4203548 519007 0 0 0 477765 2245 0 0 18 0 2 0 521008820 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 0 0 0 0 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=0 CPUtime=1198.91 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4203588 97567 0 0 0 118955 936 0 0 25 0 2 0 521008869 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.1 s]
/proc/loadavg: 8.05 8.16 8.09 6/183 16397
/proc/meminfo: memFree=28321032/32950928 swapFree=67111448/67111528
[pid=16356] ppid=16354 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/16356/stat : 16356 (ttsth-5-4) Z 16354 16356 16318 0 -1 4203548 519007 0 0 0 477765 2245 0 0 18 0 2 0 521008820 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/16356/statm: 0 0 0 0 0 0 0
[pid=16356/tid=16360] ppid=16354 vsize=0 CPUtime=1198.91 cores=1,3,5,7
/proc/16356/task/16360/stat : 16360 (ttsth-5-4) R 16354 16356 16318 0 -1 4203588 97567 0 0 0 118955 936 0 0 25 0 2 0 521008869 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.11
CPU time (s): 4800.12
CPU user time (s): 4777.66
CPU system time (s): 22.4566
CPU usage (%): 399.307
Max. virtual memory (cumulated for all children) (KiB): 2251488

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4777.66
system time used= 22.4566
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 519007
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7710
involuntary context switches= 18417

runsolver used 2.00569 second user time and 4.56031 second system time

The end

Launcher Data 

Begin job on node126 at 2011-04-29 18:21:41
IDJOB=3262750
IDBENCH=83797
IDSOLVER=1654
FILE ID=node126/3262750-1304094101
RUNJOBID= node126-1304094101-16337
PBS_JOBID= 13170590
Free space on /tmp= 71596 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/gomes/bqwh/bqwh.60.1152.shuffled-as.sat03-1650.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3262750-1304094101/watcher-3262750-1304094101 -o /tmp/evaluation-result-3262750-1304094101/solver-3262750-1304094101 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3262750-1304094101.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 98d6317b259165bb4fb7a39e783668e0
RANDOM SEED=527536052

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30925908 kB
Buffers:        245328 kB
Cached:        1397904 kB
SwapCached:         76 kB
Active:         119108 kB
Inactive:      1592716 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30925908 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:            1676 kB
Writeback:           0 kB
AnonPages:       68480 kB
Mapped:          15108 kB
Slab:           246796 kB
PageTables:       4992 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   210968 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71596 MiB
End job on node126 at 2011-04-29 18:41:43