Trace number 3250516

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.47 1202.57

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/
VanDerWaerden_pd_2-3-23_508.cnf
MD5SUMb891f70c80006fb02dca40f266c0cec1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables254
Number of clauses34661
Sum of the clauses size157137
Maximum clause length23
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 285
Number of clauses of size 331908
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52668

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250516-1304091608.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 254
0.00/0.04	c Clauses = 34661
0.00/0.04	c Literals = 157137
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 254
0.00/0.04	c Clauses = 34661
0.00/0.04	c Literals = 157137
0.00/0.04	c a2
0.19/0.23	c a1
0.59/0.60	c Independent sub-problems = 1
0.59/0.60	c Sub-problem with 254 variables and 34661 clauses
0.59/0.60	c using permutation method 4
0.59/0.60	c count = 1
0.59/0.60	c Ordering finished 0.580 seconds
0.59/0.60	c mapped 0.590 seconds
0.59/0.60	c permuted 0.590 seconds
0.59/0.60	c about to sort
0.59/0.61	c sorted
0.69/0.73	c clauses now 34661
0.69/0.73	c init finished 0.600 seconds
0.69/0.73	c tree built 0.730 seconds
0.69/0.73	c 
0.69/0.73	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.69/0.73	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.69/0.73	c ................................ | ..........                    |                                  0.7         0.0             
0.69/0.73	c ................................ | ..........                  1 |                                  0.7         1.4             
0.69/0.73	c ................................ | ..........                  2 |                                  0.7         2.7             
0.69/0.73	c ................................ | ..........                  3 |                                  0.7         4.1             
0.69/0.73	c ................................ | ..........                  4 |                                  0.7         5.5             
0.69/0.73	c ................................ | ..........                  5 |                                  0.7         6.8             
0.69/0.79	c ................................ | ..........               1000 | 24           422                 0.8      1190.5        33904
96.04/24.66	c ................................ | ..........             391000 | 27561        171059             96.3      4060.2     24090628
191.90/48.64	c ................................ | ..........             600000 | 44417        269671            192.1      3123.9     39762305
288.17/72.71	c ................................ | ..........             770000 | 59624        352322            288.2      2671.5     51769248
384.06/96.71	c ................................ | ..........             962000 | 75205        418732            384.1      2504.6     62848156
480.31/120.80	c ................................ | *.........            1120000 | 91227        474095            480.3      2331.7     73662796
576.19/144.83	c ................................ | *.........            1272000 | 104599       535560            576.3      2207.1     83720840
671.66/168.80	c ................................ | *.........            1408000 | 116824       592362            672.0      2095.1     89351107
768.73/193.06	c ................................ | *.........            1521000 | 129791       637815            769.0      1977.9     96903787
864.21/216.99	c ................................ | *.........            1623000 | 141609       681380            864.6      1877.2    103941296
960.47/241.06	c ................................ | **........            1732000 | 154032       721357            960.7      1802.8    111500326
1056.77/265.14	c ................................ | **........            1836000 | 165321       761080           1056.9      1737.1    119055382
1151.83/288.99	c ................................ | **........            1924000 | 174391       800284           1152.2      1669.8    125030887
1248.10/313.08	c ................................ | **........            2007000 | 185198       835698           1248.4      1607.6    131119483
1344.76/337.22	c ................................ | **........            2091000 | 196985       870596           1344.8      1554.8    134831250
1439.84/361.05	c ................................ | ***.......            2167000 | 206145       900770           1440.1      1504.8    138766676
1536.91/385.36	c ................................ | ***.......            2252000 | 216782       929457           1537.2      1465.0    143825540
1631.99/409.19	c ................................ | ***.......            2336000 | 227449       958641           1632.3      1431.1    150209310
1727.84/433.17	c ................................ | ***.......            2418000 | 237546       989058           1728.2      1399.2    156274349
1824.91/457.49	c ................................ | ***.......            2496000 | 245932       1014995          1825.3      1367.5    163204925
1920.76/481.40	c ................................ | ****......            2567000 | 253904       1041627          1920.8      1336.4    165353676
2016.64/505.42	c ................................ | ****......            2636000 | 262045       1066997          2016.8      1307.1    167499138
2112.11/529.36	c ................................ | ****......            2705000 | 269242       1093644          2112.3      1280.6    171984426
2208.38/553.40	c ................................ | ****......            2785000 | 277334       1125242          2208.4      1261.1    177429064
2303.86/577.38	c ................................ | ****......            2857000 | 282843       1155376          2304.2      1239.9    181996000
2400.52/601.55	c ................................ | *****.....            2930000 | 288384       1182332          2400.8      1220.4    188099675
2496.79/625.62	c ................................ | *****.....            2995000 | 293904       1208162          2496.9      1199.5    193230303
2593.05/649.70	c ................................ | *****.....            3057000 | 299293       1232990          2593.1      1178.9    197846252
2689.73/673.91	c ................................ | *****.....            3117000 | 306147       1258271          2689.8      1158.8    202032721
2784.40/697.61	c ................................ | *****.....            3174000 | 314025       1280472          2784.4      1139.9    205884255
2880.29/721.62	c ................................ | ******....            3239000 | 322214       1301100          2880.3      1124.5    208407071
2976.55/745.74	c ................................ | ******....            3312000 | 329719       1326150          2976.7      1112.6    210933441
3072.81/769.81	c ................................ | ******....            3383000 | 338411       1348987          3072.8      1100.9    215185895
3169.07/793.94	c ................................ | ******....            3445000 | 344865       1368776          3169.3      1087.0    219126859
3265.35/818.07	c ................................ | ******....            3503000 | 350518       1385971          3265.6      1072.7    223773416
3360.42/841.85	c ................................ | *******...            3559000 | 356435       1405733          3360.6      1059.0    228063605
3457.09/866.06	c ................................ | *******...            3614000 | 362445       1423762          3457.3      1045.3    232470343
3553.35/890.12	c ................................ | *******...            3658000 | 368319       1441675          3553.4      1029.4    235871502
3648.42/913.96	c ................................ | *******...            3710000 | 374825       1459080          3648.7      1016.8    235997618
3744.28/937.95	c ................................ | *******...            3759000 | 381598       1477128          3744.5      1003.9    235997618
3839.75/961.86	c ................................ | ********..            3813000 | 388659       1493801          3840.0       993.0    236971519
3936.82/986.10	c ................................ | ********..            3863000 | 395232       1508682          3936.8       981.2    238703234
4032.68/1010.11	c ................................ | ********..            3911000 | 401226       1524390          4032.7       969.8    241431918
4128.56/1034.10	c ................................ | ********..            3961000 | 406369       1539543          4128.6       959.4    244902434
4224.03/1058.02	c ................................ | ********..            4009000 | 411940       1558384          4224.1       949.1    247666629
4320.31/1082.14	c ................................ | *********.            4057000 | 417740       1574361          4320.5       939.0    251923907
4416.57/1106.21	c ................................ | *********.            4105000 | 423654       1589350          4416.6       929.4    255795741
4513.64/1130.57	c ................................ | *********.            4145000 | 429413       1605200          4513.9       918.3    259634814
4607.91/1154.16	c ................................ | *********.            4189000 | 435396       1620229          4608.2       909.0    263668519
4703.80/1178.17	c ................................ | *********.            4229000 | 441693       1635384          4704.1       899.0    267513996

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250516-1304091608/watcher-3250516-1304091608 -o /tmp/evaluation-result-3250516-1304091608/solver-3250516-1304091608 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250516-1304091608.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.00 8.03 8.00 1/175 17427
/proc/meminfo: memFree=19978832/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) R 17425 17427 16471 0 -1 4202496 434 0 0 0 0 0 0 0 25 0 1 0 501576464 9633792 377 33554432000 4194304 4338100 140736620242768 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/17427/statm: 2352 377 146 36 0 257 0

[startup+0.060257 s]
/proc/loadavg: 8.00 8.03 8.00 1/175 17427
/proc/meminfo: memFree=19978832/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=28444 CPUtime=0.05 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) R 17425 17427 16471 0 -1 4202496 4291 0 0 0 5 0 0 0 25 0 1 0 501576464 29126656 4151 33554432000 4194304 4338100 140736620242768 18446744073709551615 264726102201 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/17427/statm: 7111 4152 365 36 0 5016 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 28444

[startup+0.101249 s]
/proc/loadavg: 8.00 8.03 8.00 1/175 17427
/proc/meminfo: memFree=19978832/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=39172 CPUtime=0.09 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) R 17425 17427 16471 0 -1 4202496 6222 0 0 0 9 0 0 0 25 0 1 0 501576464 40112128 5983 33554432000 4194304 4338100 140736620242768 18446744073709551615 4206976 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/17427/statm: 9793 5983 655 36 0 7698 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 39172

[startup+0.300215 s]
/proc/loadavg: 8.00 8.03 8.00 1/175 17427
/proc/meminfo: memFree=19978832/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=16532 CPUtime=0.29 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) R 17425 17427 16471 0 -1 4202496 6529 0 0 0 29 0 0 0 25 0 1 0 501576464 16928768 2148 33554432000 4194304 4338100 140736620242768 18446744073709551615 4207456 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/17427/statm: 4133 2148 159 36 0 2038 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 16532

[startup+0.701124 s]
/proc/loadavg: 8.00 8.03 8.00 1/175 17427
/proc/meminfo: memFree=19978832/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=37388 CPUtime=0.69 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) R 17425 17427 16471 0 -1 4202496 11845 0 0 0 68 1 0 0 25 0 1 0 501576464 38285312 7392 33554432000 4194304 4338100 140736620242768 18446744073709551615 264726070808 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/17427/statm: 9380 7400 164 36 0 7285 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 37388

[startup+1.50087 s]
/proc/loadavg: 8.00 8.03 8.00 6/182 17459
/proc/meminfo: memFree=19921464/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=263392 CPUtime=3.77 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 23959 0 0 0 374 3 0 0 25 0 5 0 501576464 269713408 19506 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 65848 19506 176 36 0 63753 0
[pid=17427/tid=17456] ppid=17425 vsize=263392 CPUtime=0.75 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 2715 0 0 0 75 0 0 0 19 0 5 0 501576538 269713408 19506 33554432000 4194304 4338100 140736620242768 18446744073709551615 4229580 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=263392 CPUtime=0.74 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 2369 0 0 0 74 0 0 0 18 0 5 0 501576538 269713408 19507 33554432000 4194304 4338100 140736620242768 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=263392 CPUtime=0.75 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 2494 0 0 0 75 0 0 0 18 0 5 0 501576538 269713408 19507 33554432000 4194304 4338100 140736620242768 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=263392 CPUtime=0.75 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 2841 0 0 0 75 0 0 0 18 0 5 0 501576538 269713408 19507 33554432000 4194304 4338100 140736620242768 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3.77
Current children cumulated vsize (KiB) 263392

[startup+3.10057 s]
/proc/loadavg: 8.08 8.05 8.01 11/186 17463
/proc/meminfo: memFree=19807416/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=335996 CPUtime=10.16 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 42672 0 0 0 1010 6 0 0 25 0 5 0 501576464 344059904 38219 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 83999 38219 176 36 0 81904 0
[pid=17427/tid=17456] ppid=17425 vsize=335996 CPUtime=2.35 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 8422 0 0 0 234 1 0 0 22 0 5 0 501576538 344059904 38219 33554432000 4194304 4338100 140736620242768 18446744073709551615 4287681 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=335996 CPUtime=2.33 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 6398 0 0 0 233 0 0 0 22 0 5 0 501576538 344059904 38219 33554432000 4194304 4338100 140736620242768 18446744073709551615 4242340 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=335996 CPUtime=2.35 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 6294 0 0 0 234 1 0 0 22 0 5 0 501576538 344059904 38219 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=335996 CPUtime=2.35 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 8017 0 0 0 234 1 0 0 22 0 5 0 501576538 344059904 38219 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 10.16
Current children cumulated vsize (KiB) 335996

[startup+6.30097 s]
/proc/loadavg: 8.08 8.05 8.01 9/186 17463
/proc/meminfo: memFree=19509676/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=474420 CPUtime=22.94 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 76582 0 0 0 2282 12 0 0 25 0 5 0 501576464 485806080 72129 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 118605 72129 176 36 0 116510 0
[pid=17427/tid=17456] ppid=17425 vsize=474420 CPUtime=5.55 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 15172 0 0 0 552 3 0 0 25 0 5 0 501576538 485806080 72129 33554432000 4194304 4338100 140736620242768 18446744073709551615 4256420 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=474420 CPUtime=5.52 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 18360 0 0 0 550 2 0 0 25 0 5 0 501576538 485806080 72129 33554432000 4194304 4338100 140736620242768 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=474420 CPUtime=5.55 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 13715 0 0 0 553 2 0 0 25 0 5 0 501576538 485806080 72129 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738217766 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=474420 CPUtime=5.55 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 15794 0 0 0 554 1 0 0 25 0 5 0 501576538 485806080 72129 33554432000 4194304 4338100 140736620242768 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 22.94
Current children cumulated vsize (KiB) 474420

[startup+12.7008 s]
/proc/loadavg: 8.14 8.06 8.01 9/186 17463
/proc/meminfo: memFree=19058568/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=675668 CPUtime=48.5 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 126699 0 0 0 4830 20 0 0 25 0 5 0 501576464 691884032 122246 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 168917 122246 176 36 0 166822 0
[pid=17427/tid=17456] ppid=17425 vsize=675668 CPUtime=11.95 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 25990 0 0 0 1190 5 0 0 25 0 5 0 501576538 691884032 122246 33554432000 4194304 4338100 140736620242768 18446744073709551615 4229573 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=675668 CPUtime=11.88 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 35183 0 0 0 1184 4 0 0 25 0 5 0 501576538 691884032 122246 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=675668 CPUtime=11.95 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 25681 0 0 0 1191 4 0 0 25 0 5 0 501576538 691884032 122246 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=675668 CPUtime=11.95 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 26304 0 0 0 1192 3 0 0 25 0 5 0 501576538 691884032 122246 33554432000 4194304 4338100 140736620242768 18446744073709551615 4287686 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 48.5
Current children cumulated vsize (KiB) 675668

[startup+25.5003 s]
/proc/loadavg: 8.12 8.06 8.01 9/186 17463
/proc/meminfo: memFree=18369640/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=1005164 CPUtime=99.63 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 209339 0 0 0 9927 36 0 0 25 0 5 0 501576464 1029287936 204886 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 251291 204886 176 36 0 249196 0
[pid=17427/tid=17456] ppid=17425 vsize=1005164 CPUtime=24.75 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 47131 0 0 0 2466 9 0 0 25 0 5 0 501576538 1029287936 204886 33554432000 4194304 4338100 140736620242768 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=1005164 CPUtime=24.61 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 56966 0 0 0 2453 8 0 0 25 0 5 0 501576538 1029287936 204886 33554432000 4194304 4338100 140736620242768 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=1005164 CPUtime=24.75 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 44686 0 0 0 2467 8 0 0 25 0 5 0 501576538 1029287936 204886 33554432000 4194304 4338100 140736620242768 18446744073709551615 4256993 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=1005164 CPUtime=24.75 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 47015 0 0 0 2468 7 0 0 25 0 5 0 501576538 1029287936 204886 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 99.63
Current children cumulated vsize (KiB) 1005164

[startup+51.1005 s]
/proc/loadavg: 8.08 8.06 8.01 9/186 17464
/proc/meminfo: memFree=17389996/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=1525996 CPUtime=201.89 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 339780 0 0 0 20130 59 0 0 25 0 5 0 501576464 1562619904 335327 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 381499 335327 176 36 0 379404 0
[pid=17427/tid=17456] ppid=17425 vsize=1525996 CPUtime=50.35 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 70435 0 0 0 5021 14 0 0 25 0 5 0 501576538 1562619904 335327 33554432000 4194304 4338100 140736620242768 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=1525996 CPUtime=50.08 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 98226 0 0 0 4992 16 0 0 25 0 5 0 501576538 1562619904 335328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=1525996 CPUtime=50.35 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 71297 0 0 0 5021 14 0 0 25 0 5 0 501576538 1562619904 335328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=1525996 CPUtime=50.35 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 86281 0 0 0 5023 12 0 0 25 0 5 0 501576538 1562619904 335328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4287786 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 201.89
Current children cumulated vsize (KiB) 1525996

[startup+102.307 s]
/proc/loadavg: 8.13 8.07 8.01 9/186 17465
/proc/meminfo: memFree=15809856/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=2292064 CPUtime=406.44 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 531360 0 0 0 40547 97 0 0 25 0 5 0 501576464 2347073536 526907 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 573016 526907 176 36 0 570921 0
[pid=17427/tid=17456] ppid=17425 vsize=2292064 CPUtime=101.56 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 123180 0 0 0 10130 26 0 0 25 0 5 0 501576538 2347073536 526907 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=2292064 CPUtime=101.23 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 153510 0 0 0 10096 27 0 0 25 0 5 0 501576538 2347073536 526907 33554432000 4194304 4338100 140736620242768 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=2292064 CPUtime=101.44 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 112724 0 0 0 10122 22 0 0 25 0 5 0 501576538 2347073536 526907 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=2292064 CPUtime=101.44 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 128405 0 0 0 10124 20 0 0 25 0 5 0 501576538 2347073536 526907 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 406.44
Current children cumulated vsize (KiB) 2292064

[startup+162.301 s]
/proc/loadavg: 8.10 8.07 8.01 9/186 17467
/proc/meminfo: memFree=14411504/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=3006332 CPUtime=646.09 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 708781 0 0 0 64478 131 0 0 25 0 5 0 501576464 3078483968 704328 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 751583 704328 176 36 0 749488 0
[pid=17427/tid=17456] ppid=17425 vsize=3006332 CPUtime=161.55 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 163422 0 0 0 16121 34 0 0 25 0 5 0 501576538 3078483968 704328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=3006332 CPUtime=161.22 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 204345 0 0 0 16086 36 0 0 25 0 5 0 501576538 3078483968 704328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=3006332 CPUtime=161.13 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 151144 0 0 0 16083 30 0 0 25 0 5 0 501576538 3078483968 704328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=3006332 CPUtime=161.42 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 176329 0 0 0 16114 28 0 0 25 0 5 0 501576538 3078483968 704328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 646.09
Current children cumulated vsize (KiB) 3006332


################
# More data... #
################

/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 432093 0 0 0 75781 79 0 0 25 0 5 0 501576538 7176572928 1704894 33554432000 4194304 4338100 140736620242768 18446744073709551615 4287643 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3042.87
Current children cumulated vsize (KiB) 7008372

[startup+822.301 s]
/proc/loadavg: 8.12 8.10 8.02 9/186 17484
/proc/meminfo: memFree=5930012/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=7346856 CPUtime=3282.53 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 1793051 0 0 0 327892 361 0 0 25 0 5 0 501576464 7523180544 1788598 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 1836714 1788598 176 36 0 1834619 0
[pid=17427/tid=17456] ppid=17425 vsize=7346856 CPUtime=821.42 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 443706 0 0 0 82045 97 0 0 25 0 5 0 501576538 7523180544 1788598 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=7346856 CPUtime=820.61 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 517983 0 0 0 81964 97 0 0 25 0 5 0 501576538 7523180544 1788598 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=7346856 CPUtime=821.14 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 364135 0 0 0 82035 79 0 0 25 0 5 0 501576538 7523180544 1788598 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=7346856 CPUtime=818.59 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 453686 0 0 0 81775 84 0 0 25 0 5 0 501576538 7523180544 1788598 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3282.53
Current children cumulated vsize (KiB) 7346856

[startup+882.301 s]
/proc/loadavg: 8.11 8.09 8.02 9/186 17486
/proc/meminfo: memFree=5281928/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=7660184 CPUtime=3522.19 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 1872508 0 0 0 351839 380 0 0 25 0 5 0 501576464 7844028416 1868055 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 1915046 1868055 176 36 0 1912951 0
[pid=17427/tid=17456] ppid=17425 vsize=7660184 CPUtime=881.41 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 457643 0 0 0 88040 101 0 0 25 0 5 0 501576538 7844028416 1868055 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=7660184 CPUtime=880.29 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 542942 0 0 0 87926 103 0 0 25 0 5 0 501576538 7844028416 1868055 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=7660184 CPUtime=881.14 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 382859 0 0 0 88031 83 0 0 25 0 5 0 501576538 7844028416 1868055 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=7660184 CPUtime=878.59 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 475523 0 0 0 87769 90 0 0 25 0 5 0 501576538 7844028416 1868055 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3522.19
Current children cumulated vsize (KiB) 7660184

[startup+942.3 s]
/proc/loadavg: 8.04 8.07 8.02 9/186 17488
/proc/meminfo: memFree=4937896/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=7745332 CPUtime=3761.86 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 1894461 0 0 0 375799 387 0 0 25 0 5 0 501576464 7931219968 1890008 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 1936333 1890008 176 36 0 1934238 0
[pid=17427/tid=17456] ppid=17425 vsize=7745332 CPUtime=941.4 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 466023 0 0 0 94037 103 0 0 25 0 5 0 501576538 7931219968 1890008 33554432000 4194304 4338100 140736620242768 18446744073709551615 4253901 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=7745332 CPUtime=939.96 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 547732 0 0 0 93892 104 0 0 25 0 5 0 501576538 7931219968 1890008 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=7745332 CPUtime=941.13 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 386010 0 0 0 94029 84 0 0 25 0 5 0 501576538 7931219968 1890008 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=7745332 CPUtime=938.59 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 481155 0 0 0 93768 91 0 0 25 0 5 0 501576538 7931219968 1890008 33554432000 4194304 4338100 140736620242768 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3761.86
Current children cumulated vsize (KiB) 7745332

[startup+1002.3 s]
/proc/loadavg: 8.05 8.07 8.02 9/186 17489
/proc/meminfo: memFree=4577732/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=7874108 CPUtime=4001.54 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 1926614 0 0 0 399758 396 0 0 25 0 5 0 501576464 8063086592 1922161 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 1968527 1922161 176 36 0 1966432 0
[pid=17427/tid=17456] ppid=17425 vsize=7874108 CPUtime=1001.4 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 472733 0 0 0 100035 105 0 0 25 0 5 0 501576538 8063086592 1922161 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=7874108 CPUtime=999.64 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 569472 0 0 0 99854 110 0 0 25 0 5 0 501576538 8063086592 1922161 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=7874108 CPUtime=1001.14 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 386361 0 0 0 100029 85 0 0 25 0 5 0 501576538 8063086592 1922161 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=7874108 CPUtime=998.59 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 484507 0 0 0 99766 93 0 0 25 0 5 0 501576538 8063086592 1922161 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4001.54
Current children cumulated vsize (KiB) 7874108

[startup+1062.3 s]
/proc/loadavg: 8.06 8.07 8.01 9/186 17491
/proc/meminfo: memFree=4048184/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=8137388 CPUtime=4241.21 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 1991781 0 0 0 423709 412 0 0 25 0 5 0 501576464 8332685312 1987328 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 2034347 1987328 176 36 0 2032252 0
[pid=17427/tid=17456] ppid=17425 vsize=8137388 CPUtime=1061.4 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 489305 0 0 0 106031 109 0 0 25 0 5 0 501576538 8332685312 1987328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=8137388 CPUtime=1059.31 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 589204 0 0 0 105817 114 0 0 25 0 5 0 501576538 8332685312 1987328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=8137388 CPUtime=1061.14 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 399094 0 0 0 106026 88 0 0 25 0 5 0 501576538 8332685312 1987328 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=8137388 CPUtime=1058.6 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 500637 0 0 0 105762 98 0 0 25 0 5 0 501576538 8332685312 1987329 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4241.21
Current children cumulated vsize (KiB) 8137388

[startup+1122.3 s]
/proc/loadavg: 8.05 8.07 8.01 9/186 17492
/proc/meminfo: memFree=3472156/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=8438072 CPUtime=4480.88 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 2068102 0 0 0 447658 430 0 0 25 0 5 0 501576464 8640585728 2063649 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 2109518 2063649 176 36 0 2107423 0
[pid=17427/tid=17456] ppid=17425 vsize=8438072 CPUtime=1121.39 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 504851 0 0 0 112027 112 0 0 25 0 5 0 501576538 8640585728 2063649 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=8438072 CPUtime=1118.99 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 610118 0 0 0 111778 121 0 0 25 0 5 0 501576538 8640585728 2063649 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=8438072 CPUtime=1121.13 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 413230 0 0 0 112022 91 0 0 25 0 5 0 501576538 8640585728 2063649 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=8438072 CPUtime=1118.6 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 526362 0 0 0 111757 103 0 0 25 0 5 0 501576538 8640585728 2063649 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4480.88
Current children cumulated vsize (KiB) 8438072

[startup+1182.3 s]
/proc/loadavg: 8.05 8.07 8.01 9/186 17494
/proc/meminfo: memFree=2897692/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=8751136 CPUtime=4720.56 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 2146717 0 0 0 471606 450 0 0 25 0 5 0 501576464 8961163264 2142264 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 2187784 2142264 176 36 0 2185689 0
[pid=17427/tid=17456] ppid=17425 vsize=8751136 CPUtime=1181.39 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 528293 0 0 0 118023 116 0 0 25 0 5 0 501576538 8961163264 2142264 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=8751136 CPUtime=1178.66 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 629538 0 0 0 117740 126 0 0 25 0 5 0 501576538 8961163264 2142264 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=8751136 CPUtime=1181.14 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 425260 0 0 0 118020 94 0 0 25 0 5 0 501576538 8961163264 2142264 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=8751136 CPUtime=1178.6 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 550085 0 0 0 117750 110 0 0 25 0 5 0 501576538 8961163264 2142264 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4720.56
Current children cumulated vsize (KiB) 8751136



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.03 8.06 8.01 9/186 17495
/proc/meminfo: memFree=2651756/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=8863876 CPUtime=4800.09 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) S 17425 17427 16471 0 -1 4202496 2175117 0 0 0 479554 455 0 0 25 0 5 0 501576464 9076609024 2170664 33554432000 4194304 4338100 140736620242768 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 2215969 2170664 176 36 0 2213874 0
[pid=17427/tid=17456] ppid=17425 vsize=8863876 CPUtime=1201.3 cores=0,2,4,6
/proc/17427/task/17456/stat : 17456 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 539490 0 0 0 120012 118 0 0 25 0 5 0 501576538 9076609024 2170664 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=8863876 CPUtime=1198.46 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 636514 0 0 0 119718 128 0 0 25 0 5 0 501576538 9076609024 2170664 33554432000 4194304 4338100 140736620242768 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=17427/tid=17458] ppid=17425 vsize=8863876 CPUtime=1201.06 cores=0,2,4,6
/proc/17427/task/17458/stat : 17458 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 428703 0 0 0 120011 95 0 0 25 0 5 0 501576538 9076609024 2170664 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=17427/tid=17459] ppid=17425 vsize=8863876 CPUtime=1198.52 cores=0,2,4,6
/proc/17427/task/17459/stat : 17459 (ttsth-5-4) R 17425 17427 16471 0 -1 4202560 556869 0 0 0 119741 111 0 0 25 0 5 0 501576538 9076609024 2170664 33554432000 4194304 4338100 140736620242768 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 8863876

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.4 s]
/proc/loadavg: 8.03 8.06 8.01 9/186 17495
/proc/meminfo: memFree=2651756/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=0 CPUtime=4800.29 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) Z 17425 17427 16471 0 -1 4203548 2175117 0 0 0 479556 473 0 0 18 0 2 0 501576464 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 0 0 0 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=0 CPUtime=1198.64 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4203588 636514 0 0 0 119718 146 0 0 25 0 2 0 501576538 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 0

[startup+1202.5 s]
/proc/loadavg: 8.03 8.06 8.01 9/186 17495
/proc/meminfo: memFree=2651756/32951124 swapFree=67111524/67111528
[pid=17427] ppid=17425 vsize=0 CPUtime=4800.39 cores=0,2,4,6
/proc/17427/stat : 17427 (ttsth-5-4) Z 17425 17427 16471 0 -1 4203548 2175117 0 0 0 479556 483 0 0 18 0 2 0 501576464 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/17427/statm: 0 0 0 0 0 0 0
[pid=17427/tid=17457] ppid=17425 vsize=0 CPUtime=1198.74 cores=0,2,4,6
/proc/17427/task/17457/stat : 17457 (ttsth-5-4) R 17425 17427 16471 0 -1 4203588 636514 0 0 0 119718 156 0 0 25 0 2 0 501576538 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.39
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.57
CPU time (s): 4800.47
CPU user time (s): 4795.57
CPU system time (s): 4.89825
CPU usage (%): 399.185
Max. virtual memory (cumulated for all children) (KiB): 8863876

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.57
system time used= 4.89825
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2175117
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1207
involuntary context switches= 21026

runsolver used 2.04869 second user time and 4.48532 second system time

The end

Launcher Data 

Begin job on node141 at 2011-04-29 17:40:09
IDJOB=3250516
IDBENCH=83275
IDSOLVER=1654
FILE ID=node141/3250516-1304091608
RUNJOBID= node141-1304090404-17334
PBS_JOBID= 13170540
Free space on /tmp= 71420 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden_pd_3k/VanDerWaerden_pd_2-3-23_508.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250516-1304091608/watcher-3250516-1304091608 -o /tmp/evaluation-result-3250516-1304091608/solver-3250516-1304091608 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250516-1304091608.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b891f70c80006fb02dca40f266c0cec1
RANDOM SEED=2112449253

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19978996 kB
Buffers:       1543616 kB
Cached:       10782376 kB
SwapCached:          4 kB
Active:        3733796 kB
Inactive:      8656984 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19978996 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             812 kB
Writeback:           0 kB
AnonPages:       64728 kB
Mapped:          15068 kB
Slab:           516252 kB
PageTables:       4836 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   218464 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71424 MiB
End job on node141 at 2011-04-29 18:00:13