Trace number 3258570

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.19 1202.97

General information on the benchmark

NameCRAFTED/rbsat/random/
forced/rbsat-v2400c266431gyes1.cnf
MD5SUM836d368f7cf78173b8e7f8b131296e2f
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data 

0.09/0.18	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258570-1304051825.cnf 
0.09/0.18	c Using SATTIMEOUT of 4800
0.09/0.18	c Using SATRAM of 15500
0.09/0.18	c Size of subset node = 32
0.09/0.18	c Variables = 2400
0.09/0.18	c Clauses = 266431
0.09/0.18	c Literals = 535112
0.09/0.18	c Reduced size:
0.09/0.18	c Variables = 2400
0.09/0.18	c Clauses = 266431
0.09/0.18	c Literals = 535112
0.09/0.18	c a2
0.29/0.34	c a1
0.69/0.70	c Independent sub-problems = 1
0.69/0.70	c Sub-problem with 2400 variables and 266431 clauses
0.69/0.70	c using permutation method 4
0.69/0.70	c count = 1
0.69/0.70	c Ordering finished 0.660 seconds
0.69/0.70	c mapped 0.690 seconds
0.69/0.70	c permuted 0.690 seconds
0.69/0.70	c about to sort
0.69/0.72	c sorted
1.29/1.38	c clauses now 266431
1.29/1.38	c init finished 0.770 seconds
1.29/1.38	c tree built 1.370 seconds
1.29/1.38	c 
1.29/1.38	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.29/1.38	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.29/1.38	c ................................ | ..........                    |                                  1.4         0.0             
1.29/1.38	c ................................ | ..........                  1 |                                  1.4         0.7             
1.29/1.38	c ................................ | ..........                  2 |                                  1.4         1.4             
1.29/1.38	c ................................ | ..........                  3 |                                  1.4         2.2             
1.29/1.38	c ................................ | ..........                  4 |                                  1.4         2.9             
1.29/1.38	c ................................ | ..........                  5 |                                  1.4         3.6             
1.29/1.39	c ................................ | ..........               1000 |              94                  1.4       719.4         2502
95.70/25.08	c ................................ | ..........            3791000 | 54180        281959             96.0     39481.4     13603773
191.96/49.11	c ................................ | ..........            6799000 | 101207       505559            192.0     35411.5     22540848
287.83/73.15	c ................................ | ..........            9213000 | 143334       685182            288.0     31986.3     30577691
383.70/97.18	c ................................ | ..........           11515000 | 188953       855071            384.0     29984.6     37779710
479.95/121.21	c ................................ | *.........           13766000 | 231390       1002150           480.0     28679.2     42881680
575.83/145.25	c ................................ | *.........           16180000 | 271794       1144203           576.0     28088.8     46953459
671.67/169.28	c ................................ | *.........           18320000 | 307054       1295000           672.0     27261.1     50901783
767.93/193.32	c ................................ | *.........           20270000 | 338531       1445141           768.0     26391.9     55001515
863.79/217.35	c ................................ | *.........           22060000 | 369011       1576629           864.0     25531.8     58172879
959.65/241.39	c ................................ | **........           23789000 | 400363       1706366           960.0     24778.9     61390982
1055.92/265.42	c ................................ | **........           25438000 | 427775       1836173          1056.0     24089.0     63910001
1151.79/289.45	c ................................ | **........           26971000 | 456764       1944570          1152.0     23412.1     67875192
1247.66/313.48	c ................................ | **........           28526000 | 485762       2052730          1248.0     22857.4     70362131
1343.92/337.52	c ................................ | **........           30366000 | 517802       2159702          1344.0     22593.4     74733603
1439.78/361.55	c ................................ | ***.......           32107000 | 550582       2258783          1440.0     22296.4     78171133
1535.64/385.59	c ................................ | ***.......           33740000 | 583786       2364556          1536.0     21965.7     83110931
1631.91/409.62	c ................................ | ***.......           35228000 | 618361       2470847          1632.0     21585.7     89078932
1727.78/433.65	c ................................ | ***.......           36782000 | 651124       2568589          1728.0     21285.9     93577128
1823.68/457.69	c ................................ | ***.......           38519000 | 680517       2659303          1824.0     21117.4     97044556
1919.92/481.73	c ................................ | ****......           40201000 | 710992       2756643          1920.0     20937.6     99540662
2015.77/505.75	c ................................ | ****......           41579000 | 735393       2845456          2016.0     20624.4    101785987
2111.63/529.79	c ................................ | ****......           43006000 | 762207       2931264          2112.0     20362.6    105094699
2207.88/553.83	c ................................ | ****......           44355000 | 790045       3015663          2208.0     20088.3    109784676
2303.75/577.87	c ................................ | ****......           45624000 | 813517       3098438          2304.0     19801.7    110573108
2399.99/601.91	c ................................ | *****.....           46934000 | 839854       3172680          2400.1     19555.3    111716522
2495.87/625.95	c ................................ | *****.....           48204000 | 866785       3252530          2496.1     19312.1    114035532
2591.70/649.98	c ................................ | *****.....           49444000 | 892450       3333978          2592.0     19075.3    117787774
2687.97/674.01	c ................................ | *****.....           50963000 | 920589       3424365          2688.0     18959.3    120958750
2783.81/698.05	c ................................ | *****.....           52535000 | 950635       3509754          2784.0     18870.1    122224159
2880.08/722.14	c ................................ | ******....           54000000 | 977962       3590635          2880.2     18748.5    124443745
2975.93/746.12	c ................................ | ******....           55306000 | 1003275      3662427          2976.0     18583.8    124729890
3071.80/770.15	c ................................ | ******....           56613000 | 1030938      3736086          3072.0     18428.7    126238692
3167.68/794.18	c ................................ | ******....           57878000 | 1058402      3801849          3168.0     18269.6    127616482
3263.94/818.21	c ................................ | ******....           59103000 | 1091190      3863414          3264.0     18107.5    130172079
3359.80/842.26	c ................................ | *******...           60563000 | 1127276      3945586          3360.1     18024.4    133512022
3456.06/866.30	c ................................ | *******...           61981000 | 1155444      4032795          3456.1     17934.0    135765613
3551.89/890.33	c ................................ | *******...           63316000 | 1179498      4115786          3552.1     17825.2    138334450
3647.77/914.35	c ................................ | *******...           64707000 | 1203153      4194189          3648.0     17737.7    140940935
3744.04/938.41	c ................................ | *******...           66038000 | 1227633      4272056          3744.1     17638.0    143623153
3839.92/962.43	c ................................ | ********..           67267000 | 1251568      4350816          3840.1     17517.2    144381866
3935.72/986.48	c ................................ | ********..           68426000 | 1273244      4428165          3936.1     17384.4    145515170
4031.98/1010.50	c ................................ | ********..           69723000 | 1295032      4507806          4032.0     17292.4    145515170
4127.83/1034.55	c ................................ | ********..           70894000 | 1317842      4581777          4128.0     17173.8    146190987
4223.70/1058.57	c ................................ | ********..           72100000 | 1338066      4663540          4224.0     17069.1    147624849
4319.97/1082.61	c ................................ | *********.           73261000 | 1359743      4734469          4320.0     16958.5    148495761
4415.86/1106.65	c ................................ | *********.           74398000 | 1380442      4809231          4416.1     16847.2    151337359
4511.70/1130.69	c ................................ | *********.           75585000 | 1403698      4881528          4512.1     16751.7    154347900
4607.96/1154.71	c ................................ | *********.           76722000 | 1425236      4952629          4608.0     16649.6    157232823
4703.80/1178.74	c ................................ | *********.           77855000 | 1447748      5018916          4704.0     16550.8    158199623
4799.65/1202.78	c CPU limit (4800) reached
4799.65/1202.78	c Time = 4800.00 seconds
4799.65/1202.78	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258570-1304051825/watcher-3258570-1304051825 -o /tmp/evaluation-result-3258570-1304051825/solver-3258570-1304051825 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258570-1304051825.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 5.67 7.49 7.65 2/177 368
/proc/meminfo: memFree=27172632/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=8980 CPUtime=0 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) R 366 368 330 0 -1 4202496 309 0 0 0 0 0 0 0 21 0 1 0 516780982 9195520 251 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204344439 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/368/statm: 2245 251 147 36 0 150 0

[startup+0.0469411 s]
/proc/loadavg: 5.67 7.49 7.65 2/177 368
/proc/meminfo: memFree=27172632/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=17876 CPUtime=0.04 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) R 366 368 330 0 -1 4202496 2520 0 0 0 4 0 0 0 21 0 1 0 516780982 18305024 2462 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204308165 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/368/statm: 4469 2464 148 36 0 2374 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 17876

[startup+0.100939 s]
/proc/loadavg: 5.67 7.49 7.65 2/177 368
/proc/meminfo: memFree=27172632/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=28808 CPUtime=0.09 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) R 366 368 330 0 -1 4202496 5198 0 0 0 9 0 0 0 21 0 1 0 516780982 29499392 5140 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204308165 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/368/statm: 7202 5142 148 36 0 5107 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28808

[startup+0.3009 s]
/proc/loadavg: 5.67 7.49 7.65 2/177 368
/proc/meminfo: memFree=27172632/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=61612 CPUtime=0.29 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) R 366 368 330 0 -1 4202496 13369 0 0 0 28 1 0 0 22 0 1 0 516780982 63090688 13310 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204042430 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/368/statm: 15403 13310 153 36 0 13308 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 61612

[startup+0.700853 s]
/proc/loadavg: 5.67 7.49 7.65 2/177 368
/proc/meminfo: memFree=27172632/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=68260 CPUtime=0.69 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) R 366 368 330 0 -1 4202496 16176 0 0 0 68 1 0 0 24 0 1 0 516780982 69898240 15075 33554432000 4194304 4338100 140736021514832 18446744073709551615 4225012 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/368/statm: 17065 15075 161 36 0 14970 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 68260

[startup+1.50067 s]
/proc/loadavg: 5.37 7.40 7.62 3/178 369
/proc/meminfo: memFree=26950548/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=360032 CPUtime=1.82 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 45504 0 0 0 177 5 0 0 25 0 5 0 516780982 368672768 43361 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 90008 43361 176 36 0 87913 0
Current children cumulated CPU time (s) 1.82
Current children cumulated vsize (KiB) 360032

[startup+3.1004 s]
/proc/loadavg: 5.37 7.40 7.62 10/186 377
/proc/meminfo: memFree=26810488/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=401752 CPUtime=8.22 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 56441 0 0 0 803 19 0 0 25 0 5 0 516780982 411394048 54298 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 100438 54298 176 36 0 98343 0
[pid=368/tid=374] ppid=366 vsize=401752 CPUtime=1.69 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 4495 0 0 0 169 0 0 0 25 0 5 0 516781121 411394048 54298 33554432000 4194304 4338100 140736021514832 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=375] ppid=366 vsize=401752 CPUtime=1.71 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 2238 0 0 0 164 7 0 0 20 0 5 0 516781121 411394048 54298 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=368/tid=376] ppid=366 vsize=401752 CPUtime=1.7 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 4002 0 0 0 170 0 0 0 20 0 5 0 516781121 411394048 54298 33554432000 4194304 4338100 140736021514832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=377] ppid=366 vsize=401752 CPUtime=1.71 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 2200 0 0 0 164 7 0 0 20 0 5 0 516781121 411394048 54298 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 8.22
Current children cumulated vsize (KiB) 401752

[startup+6.30087 s]
/proc/loadavg: 5.58 7.41 7.62 9/186 377
/proc/meminfo: memFree=26642204/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=467836 CPUtime=21 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 73553 0 0 0 2054 46 0 0 25 0 5 0 516780982 479064064 71410 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 116959 71410 176 36 0 114864 0
[pid=368/tid=374] ppid=366 vsize=467836 CPUtime=4.88 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 11009 0 0 0 486 2 0 0 25 0 5 0 516781121 479064064 71410 33554432000 4194304 4338100 140736021514832 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=375] ppid=366 vsize=467836 CPUtime=4.9 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 5028 0 0 0 471 19 0 0 25 0 5 0 516781121 479064064 71410 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=368/tid=376] ppid=366 vsize=467836 CPUtime=4.9 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 9292 0 0 0 489 1 0 0 25 0 5 0 516781121 479064064 71410 33554432000 4194304 4338100 140736021514832 18446744073709551615 4253946 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=377] ppid=366 vsize=467836 CPUtime=4.91 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 4718 0 0 0 473 18 0 0 25 0 5 0 516781121 479064064 71410 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 21
Current children cumulated vsize (KiB) 467836

[startup+12.7008 s]
/proc/loadavg: 5.78 7.42 7.62 10/186 378
/proc/meminfo: memFree=26348356/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=591612 CPUtime=46.57 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 103165 0 0 0 4555 102 0 0 25 0 5 0 516780982 605810688 101022 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 147903 101022 176 36 0 145808 0
[pid=368/tid=374] ppid=366 vsize=591612 CPUtime=11.24 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 21602 0 0 0 1119 5 0 0 25 0 5 0 516781121 605810688 101022 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=375] ppid=366 vsize=591612 CPUtime=11.3 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 10102 0 0 0 1087 43 0 0 25 0 5 0 516781121 605810688 101022 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=368/tid=376] ppid=366 vsize=591612 CPUtime=11.31 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 18659 0 0 0 1128 3 0 0 25 0 5 0 516781121 605810688 101022 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204042261 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=377] ppid=366 vsize=591612 CPUtime=11.31 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 9296 0 0 0 1086 45 0 0 25 0 5 0 516781121 605810688 101022 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 46.57
Current children cumulated vsize (KiB) 591612

[startup+25.5007 s]
/proc/loadavg: 6.19 7.46 7.63 10/186 378
/proc/meminfo: memFree=25869652/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=788940 CPUtime=97.7 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 153825 0 0 0 9567 203 0 0 25 0 5 0 516780982 807874560 151682 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 197235 151683 176 36 0 195140 0
[pid=368/tid=374] ppid=366 vsize=788940 CPUtime=23.97 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 39364 0 0 0 2385 12 0 0 25 0 5 0 516781121 807874560 151683 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=375] ppid=366 vsize=788940 CPUtime=24.1 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 19128 0 0 0 2321 89 0 0 25 0 5 0 516781121 807874560 151683 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=368/tid=376] ppid=366 vsize=788940 CPUtime=24.11 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 34939 0 0 0 2405 6 0 0 25 0 5 0 516781121 807874560 151683 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204042256 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=377] ppid=366 vsize=788940 CPUtime=24.1 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 16889 0 0 0 2320 90 0 0 25 0 5 0 516781121 807874560 151683 33554432000 4194304 4338100 140736021514832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 97.7
Current children cumulated vsize (KiB) 788940

[startup+51.1004 s]
/proc/loadavg: 6.90 7.51 7.64 10/186 379
/proc/meminfo: memFree=25171684/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=1091388 CPUtime=199.95 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 227627 0 0 0 19581 414 0 0 25 0 5 0 516780982 1117581312 225484 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 272847 225484 176 36 0 270752 0
[pid=368/tid=374] ppid=366 vsize=1091388 CPUtime=49.54 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 66711 0 0 0 4932 22 0 0 25 0 5 0 516781121 1117581312 225484 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204042616 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=1091388 CPUtime=49.7 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 29874 0 0 0 4785 185 0 0 25 0 5 0 516781121 1117581312 225484 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=368/tid=376] ppid=366 vsize=1091388 CPUtime=49.59 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 58006 0 0 0 4947 12 0 0 25 0 5 0 516781121 1117581312 225484 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=377] ppid=366 vsize=1091388 CPUtime=49.7 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 29530 0 0 0 4782 188 0 0 25 0 5 0 516781121 1117581312 225484 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 199.95
Current children cumulated vsize (KiB) 1091388

[startup+102.307 s]
/proc/loadavg: 7.52 7.58 7.66 10/186 380
/proc/meminfo: memFree=24055544/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=1673044 CPUtime=404.49 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 356793 0 0 0 39655 794 0 0 25 0 5 0 516780982 1713197056 354650 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 418261 354650 176 36 0 416166 0
[pid=368/tid=374] ppid=366 vsize=1673044 CPUtime=100.74 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 110780 0 0 0 10036 38 0 0 25 0 5 0 516781121 1713197056 354650 33554432000 4194304 4338100 140736021514832 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=1673044 CPUtime=100.85 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 44056 0 0 0 9721 364 0 0 25 0 5 0 516781121 1713197056 354650 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=368/tid=376] ppid=366 vsize=1673044 CPUtime=100.75 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 115341 0 0 0 10050 25 0 0 25 0 5 0 516781121 1713197056 354650 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=1673044 CPUtime=100.74 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 43110 0 0 0 9713 361 0 0 25 0 5 0 516781121 1713197056 354650 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 404.49
Current children cumulated vsize (KiB) 1673044

[startup+162.301 s]
/proc/loadavg: 7.82 7.65 7.67 10/186 382
/proc/meminfo: memFree=23070724/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=2033812 CPUtime=644.11 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 447170 0 0 0 63253 1158 0 0 25 0 5 0 516780982 2082623488 442401 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 508453 442401 176 36 0 506358 0
[pid=368/tid=374] ppid=366 vsize=2033812 CPUtime=160.73 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 153070 0 0 0 16015 58 0 0 25 0 5 0 516781121 2082623488 442401 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204042990 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=2033812 CPUtime=160.62 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 50582 0 0 0 15528 534 0 0 25 0 5 0 516781121 2082623488 442401 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=368/tid=376] ppid=366 vsize=2033812 CPUtime=160.74 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 152565 0 0 0 16042 32 0 0 25 0 5 0 516781121 2082623488 442401 33554432000 4194304 4338100 140736021514832 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=2033812 CPUtime=160.61 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 47447 0 0 0 15532 529 0 0 25 0 5 0 516781121 2082623488 442401 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 644.11
Current children cumulated vsize (KiB) 2033812

[startup+222.301 s]
/proc/loadavg: 8.02 7.74 7.70 10/186 383
/proc/meminfo: memFree=22238168/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=2337624 CPUtime=883.77 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 523303 0 0 0 86835 1542 0 0 25 0 5 0 516780982 2393726976 518534 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 584406 518534 176 36 0 582311 0
[pid=368/tid=374] ppid=366 vsize=2337624 CPUtime=220.73 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 186436 0 0 0 22001 72 0 0 25 0 5 0 516781121 2393726976 518534 33554432000 4194304 4338100 140736021514832 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 1 0 0 0

################
# More data... #
################

/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 90220 0 0 0 68303 1746 0 0 25 0 5 0 516781121 4485013504 1023733 33554432000 4194304 4338100 140736021514832 18446744073709551615 210204042721 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2800.99
Current children cumulated vsize (KiB) 4379896

[startup+762.307 s]
/proc/loadavg: 8.07 8.02 7.85 10/186 398
/proc/meminfo: memFree=18799844/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=4505312 CPUtime=3040.67 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1057281 0 0 0 300056 4011 0 0 25 0 5 0 516780982 4613439488 1052512 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1126328 1052512 176 36 0 1124233 0
[pid=368/tid=374] ppid=366 vsize=4505312 CPUtime=759.89 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 425428 0 0 0 75800 189 0 0 25 0 5 0 516781121 4613439488 1052513 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=4505312 CPUtime=760.36 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 80714 0 0 0 74174 1862 0 0 25 0 5 0 516781121 4613439488 1052513 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=376] ppid=366 vsize=4505312 CPUtime=758.52 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 414871 0 0 0 75749 103 0 0 25 0 5 0 516781121 4613439488 1052513 33554432000 4194304 4338100 140736021514832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=4505312 CPUtime=760.49 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 92763 0 0 0 74197 1852 0 0 25 0 5 0 516781121 4613439488 1052513 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3040.67
Current children cumulated vsize (KiB) 4505312

[startup+822.301 s]
/proc/loadavg: 8.08 8.03 7.87 10/186 399
/proc/meminfo: memFree=18509168/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=4661712 CPUtime=3280.32 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1096478 0 0 0 323807 4225 0 0 25 0 5 0 516780982 4773593088 1091709 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1165428 1091709 176 36 0 1163333 0
[pid=368/tid=374] ppid=366 vsize=4661712 CPUtime=819.89 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 446197 0 0 0 81789 200 0 0 25 0 5 0 516781121 4773593088 1091709 33554432000 4194304 4338100 140736021514832 18446744073709551615 4284214 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=4661712 CPUtime=820.04 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 80714 0 0 0 80046 1958 0 0 25 0 5 0 516781121 4773593088 1091709 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=376] ppid=366 vsize=4661712 CPUtime=818.5 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 433297 0 0 0 81742 108 0 0 25 0 5 0 516781121 4773593088 1091709 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=4661712 CPUtime=820.49 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 92764 0 0 0 80095 1954 0 0 25 0 5 0 516781121 4773593088 1091709 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3280.32
Current children cumulated vsize (KiB) 4661712

[startup+882.301 s]
/proc/loadavg: 8.07 8.04 7.88 10/186 401
/proc/meminfo: memFree=18288812/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=4871288 CPUtime=3519.95 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1148944 0 0 0 347529 4466 0 0 25 0 5 0 516780982 4988198912 1144175 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1217822 1144175 176 36 0 1215727 0
[pid=368/tid=374] ppid=366 vsize=4871288 CPUtime=879.76 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 477463 0 0 0 87763 213 0 0 25 0 5 0 516781121 4988198912 1144175 33554432000 4194304 4338100 140736021514832 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=375] ppid=366 vsize=4871288 CPUtime=879.81 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 80718 0 0 0 85910 2071 0 0 25 0 5 0 516781121 4988198912 1144175 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254840 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=376] ppid=366 vsize=4871288 CPUtime=878.5 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 454491 0 0 0 87736 114 0 0 25 0 5 0 516781121 4988198912 1144175 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=4871288 CPUtime=880.47 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 92766 0 0 0 85985 2062 0 0 25 0 5 0 516781121 4988198912 1144175 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254010 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3519.95
Current children cumulated vsize (KiB) 4871288

[startup+942.301 s]
/proc/loadavg: 8.11 8.06 7.89 10/186 403
/proc/meminfo: memFree=17918020/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=5074740 CPUtime=3759.61 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1199956 0 0 0 371259 4702 0 0 25 0 5 0 516780982 5196533760 1195187 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1268685 1195187 176 36 0 1266590 0
[pid=368/tid=374] ppid=366 vsize=5074740 CPUtime=939.43 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 515142 0 0 0 93715 228 0 0 25 0 5 0 516781121 5196533760 1195187 33554432000 4194304 4338100 140736021514832 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=375] ppid=366 vsize=5074740 CPUtime=939.8 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 80719 0 0 0 91800 2180 0 0 25 0 5 0 516781121 5196533760 1195187 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=376] ppid=366 vsize=5074740 CPUtime=938.5 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 467823 0 0 0 93731 119 0 0 25 0 5 0 516781121 5196533760 1195187 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=5074740 CPUtime=940.47 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 92766 0 0 0 91877 2170 0 0 25 0 5 0 516781121 5196533760 1195187 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3759.61
Current children cumulated vsize (KiB) 5074740

[startup+1002.3 s]
/proc/loadavg: 8.12 8.07 7.91 10/186 404
/proc/meminfo: memFree=17523652/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=5227188 CPUtime=3999.23 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1223516 0 0 0 394985 4938 0 0 25 0 5 0 516780982 5352640512 1218747 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1306797 1218747 176 36 0 1304702 0
[pid=368/tid=374] ppid=366 vsize=5227188 CPUtime=999.2 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 527678 0 0 0 99685 235 0 0 25 0 5 0 516781121 5352640512 1218747 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=5227188 CPUtime=999.69 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 84938 0 0 0 97674 2295 0 0 25 0 5 0 516781121 5352640512 1218747 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=376] ppid=366 vsize=5227188 CPUtime=998.48 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 470285 0 0 0 99728 120 0 0 25 0 5 0 516781121 5352640512 1218747 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=5227188 CPUtime=1000.45 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 97109 0 0 0 97763 2282 0 0 25 0 5 0 516781121 5352640512 1218747 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3999.23
Current children cumulated vsize (KiB) 5227188

[startup+1062.3 s]
/proc/loadavg: 8.08 8.07 7.91 9/186 406
/proc/meminfo: memFree=17120260/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=5302260 CPUtime=4238.88 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1241757 0 0 0 418742 5146 0 0 25 0 5 0 516780982 5429514240 1236988 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1325565 1236988 176 36 0 1323470 0
[pid=368/tid=374] ppid=366 vsize=5302260 CPUtime=1059.19 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 545917 0 0 0 105673 246 0 0 25 0 5 0 516781121 5429514240 1236988 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=5302260 CPUtime=1059.36 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 84940 0 0 0 103544 2392 0 0 25 0 5 0 516781121 5429514240 1236988 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216473269 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=376] ppid=366 vsize=5302260 CPUtime=1058.47 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 470285 0 0 0 105727 120 0 0 25 0 5 0 516781121 5429514240 1236988 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=5302260 CPUtime=1060.45 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 97109 0 0 0 103663 2382 0 0 25 0 5 0 516781121 5429514240 1236988 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216459982 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4238.88
Current children cumulated vsize (KiB) 5302260

[startup+1122.3 s]
/proc/loadavg: 8.06 8.07 7.92 10/186 407
/proc/meminfo: memFree=16765604/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=5468044 CPUtime=4478.55 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1283713 0 0 0 442505 5350 0 0 25 0 5 0 516780982 5599277056 1278944 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1367011 1278945 176 36 0 1364916 0
[pid=368/tid=374] ppid=366 vsize=5468044 CPUtime=1119.19 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 572010 0 0 0 111661 258 0 0 25 0 5 0 516781121 5599277056 1278945 33554432000 4194304 4338100 140736021514832 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=5468044 CPUtime=1119.04 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 84940 0 0 0 109417 2487 0 0 25 0 5 0 516781121 5599277056 1278945 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=376] ppid=366 vsize=5468044 CPUtime=1118.47 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 486149 0 0 0 111722 125 0 0 25 0 5 0 516781121 5599277056 1278945 33554432000 4194304 4338100 140736021514832 18446744073709551615 4256007 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=5468044 CPUtime=1120.45 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 97109 0 0 0 109570 2475 0 0 25 0 5 0 516781121 5599277056 1278945 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4478.55
Current children cumulated vsize (KiB) 5468044

[startup+1182.3 s]
/proc/loadavg: 8.13 8.09 7.93 10/186 409
/proc/meminfo: memFree=16462144/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=5624444 CPUtime=4718.19 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) S 366 368 330 0 -1 4202496 1324229 0 0 0 466253 5566 0 0 25 0 5 0 516780982 5759430656 1319460 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216450021 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 1406111 1319460 176 36 0 1404016 0
[pid=368/tid=374] ppid=366 vsize=5624444 CPUtime=1179.17 cores=1,3,5,7
/proc/368/task/374/stat : 374 (ttsth-5-4) R 366 368 330 0 -1 4202560 597442 0 0 0 117647 270 0 0 25 0 5 0 516781121 5759430656 1319460 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=368/tid=375] ppid=366 vsize=5624444 CPUtime=1178.7 cores=1,3,5,7
/proc/368/task/375/stat : 375 (ttsth-5-4) R 366 368 330 0 -1 4202560 85541 0 0 0 115284 2586 0 0 25 0 5 0 516781121 5759430656 1319460 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216459982 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=368/tid=376] ppid=366 vsize=5624444 CPUtime=1178.46 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4202560 499753 0 0 0 117717 129 0 0 25 0 5 0 516781121 5759430656 1319460 33554432000 4194304 4338100 140736021514832 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=368/tid=377] ppid=366 vsize=5624444 CPUtime=1180.44 cores=1,3,5,7
/proc/368/task/377/stat : 377 (ttsth-5-4) R 366 368 330 0 -1 4202560 97987 0 0 0 115470 2574 0 0 25 0 5 0 516781121 5759430656 1319460 33554432000 4194304 4338100 140736021514832 18446744073709551615 210216459982 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4718.19
Current children cumulated vsize (KiB) 5624444



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.8 s]
/proc/loadavg: 8.09 8.08 7.93 6/183 409
/proc/meminfo: memFree=16761840/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=0 CPUtime=4800.01 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) Z 366 368 330 0 -1 4203548 1328308 0 0 0 474361 5640 0 0 18 0 2 0 516780982 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 0 0 0 0 0 0 0
[pid=368/tid=376] ppid=366 vsize=0 CPUtime=1198.94 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4203588 499753 0 0 0 119763 131 0 0 25 0 2 0 516781121 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.01
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.91 s]
/proc/loadavg: 8.09 8.08 7.93 6/183 409
/proc/meminfo: memFree=16761840/32951124 swapFree=67088964/67111528
[pid=368] ppid=366 vsize=0 CPUtime=4800.12 cores=1,3,5,7
/proc/368/stat : 368 (ttsth-5-4) Z 366 368 330 0 -1 4203548 1328308 0 0 0 474361 5651 0 0 18 0 2 0 516780982 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/368/statm: 0 0 0 0 0 0 0
[pid=368/tid=376] ppid=366 vsize=0 CPUtime=1199.05 cores=1,3,5,7
/proc/368/task/376/stat : 376 (ttsth-5-4) R 366 368 330 0 -1 4203588 499753 0 0 0 119763 142 0 0 25 0 2 0 516781121 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.12
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.97
CPU time (s): 4800.19
CPU user time (s): 4743.62
CPU system time (s): 56.5744
CPU usage (%): 399.028
Max. virtual memory (cumulated for all children) (KiB): 5640084

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4743.62
system time used= 56.5744
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1328308
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 4311
involuntary context switches= 33769

runsolver used 2.08068 second user time and 4.48332 second system time

The end

Launcher Data 

Begin job on node130 at 2011-04-29 06:37:05
IDJOB=3258570
IDBENCH=71385
IDSOLVER=1654
FILE ID=node130/3258570-1304051825
RUNJOBID= node130-1304051825-349
PBS_JOBID= 13170415
Free space on /tmp= 71988 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v2400c266431gyes1.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258570-1304051825/watcher-3258570-1304051825 -o /tmp/evaluation-result-3258570-1304051825/solver-3258570-1304051825 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258570-1304051825.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 836d368f7cf78173b8e7f8b131296e2f
RANDOM SEED=1889897766

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27173408 kB
Buffers:       1155604 kB
Cached:        4096340 kB
SwapCached:      22432 kB
Active:         364308 kB
Inactive:      4980324 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27173408 kB
SwapTotal:    67111528 kB
SwapFree:     67088964 kB
Dirty:            6328 kB
Writeback:           0 kB
AnonPages:       70252 kB
Mapped:          15232 kB
Slab:           368304 kB
PageTables:       4988 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   213040 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71988 MiB
End job on node130 at 2011-04-29 06:57:10