Trace number 3249724

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.25 1203.78

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-14x34.cnf
MD5SUM72677fc46303044d3d9e832fa24a37b2
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1904
Number of clauses207536
Sum of the clauses size824432
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22856
Number of clauses of size 30
Number of clauses of size 4204680
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.19/0.22	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249724-1304065096.cnf 
0.19/0.22	c Using SATTIMEOUT of 4800
0.19/0.22	c Using SATRAM of 15500
0.19/0.22	c Size of subset node = 32
0.19/0.22	c Variables = 1904
0.19/0.22	c Clauses = 207536
0.19/0.22	c Literals = 824432
0.19/0.22	c Reduced size:
0.19/0.22	c Variables = 1904
0.19/0.22	c Clauses = 207536
0.19/0.22	c Literals = 824432
0.19/0.22	c a2
0.59/0.66	c a1
1.59/1.67	c Independent sub-problems = 1
1.59/1.67	c Sub-problem with 1904 variables and 207536 clauses
1.59/1.67	c using permutation method 4
1.59/1.67	c count = 1
1.59/1.67	c Ordering finished 1.630 seconds
1.59/1.67	c mapped 1.670 seconds
1.59/1.67	c permuted 1.670 seconds
1.59/1.67	c about to sort
1.59/1.70	c sorted
1.99/2.09	c clauses now 207536
1.99/2.09	c init finished 1.720 seconds
1.99/2.09	c tree built 2.080 seconds
1.99/2.09	c 
1.99/2.09	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.99/2.09	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.99/2.09	c ................................ | ..........                    |                                  2.1         0.0             
1.99/2.09	c ................................ | ..........                  1 |                                  2.1         0.5             
1.99/2.09	c ................................ | ..........                  2 |                                  2.1         1.0             
1.99/2.09	c ................................ | ..........                  3 |                                  2.1         1.4             
1.99/2.09	c ................................ | ..........                  4 |                                  2.1         1.9             
1.99/2.09	c ................................ | ..........                  5 |                                  2.1         2.4             
1.99/2.10	c ................................ | ..........               1000 | 1            183                 2.1       473.9          609
95.94/25.61	c ................................ | ..........            1352000 | 79842        387269             96.0     14081.9      7825371
191.78/49.65	c ................................ | ..........            2431000 | 132100       674947            192.0     12661.5     14296313
287.65/73.70	c ................................ | ..........            3354000 | 182179       921049            288.0     11644.2     19504010
383.86/97.74	c ................................ | ..........            4052000 | 222962       1093314           384.0     10551.3     23210062
479.70/121.79	c ................................ | *.........            4680000 | 262476       1287561           480.1      9748.8     26992778
575.93/145.82	c ................................ | *.........            5272000 | 302952       1465694           576.0      9152.5     31672867
671.78/169.85	c ................................ | *.........            5801000 | 335517       1609207           672.0      8632.3     35070305
768.03/193.94	c ................................ | *.........            6325000 | 367980       1744872           768.2      8233.6     38416565
863.83/217.94	c ................................ | *.........            6903000 | 403003       1887566           864.0      7989.5     41161954
959.67/241.98	c ................................ | **........            7614000 | 451989       2064080           960.0      7931.2     47010625
1055.91/266.05	c ................................ | **........            8152000 | 490206       2199928          1056.1      7718.7     50304578
1151.73/290.07	c ................................ | **........            8688000 | 526777       2352944          1152.0      7541.5     53737073
1247.97/314.12	c ................................ | **........            9138000 | 558303       2469393          1248.1      7321.7     55099088
1344.21/338.21	c ................................ | **........            9614000 | 591043       2585855          1344.2      7152.0     58062276
1440.05/362.22	c ................................ | ***.......            9988000 | 617026       2668193          1440.2      6935.4     59597683
1535.88/386.22	c ................................ | ***.......           10485000 | 652089       2780247          1536.0      6826.2     60111970
1632.14/410.31	c ................................ | ***.......           10978000 | 684781       2896841          1632.2      6726.1     61322438
1727.96/434.32	c ................................ | ***.......           11406000 | 718553       2997168          1728.0      6600.5     63246581
1823.79/458.36	c ................................ | ***.......           12007000 | 757323       3133575          1824.1      6582.6     64119309
1920.04/482.43	c ................................ | ****......           12498000 | 789318       3244582          1920.2      6508.8     64952677
2015.89/506.44	c ................................ | ****......           12946000 | 815705       3341622          2016.1      6421.4     66746294
2111.73/530.47	c ................................ | ****......           13311000 | 841221       3424195          2112.0      6302.5     67836850
2207.96/554.57	c ................................ | ****......           13642000 | 870357       3498055          2208.2      6177.7     69862798
2303.83/578.58	c ................................ | ****......           13977000 | 897711       3571989          2304.2      6066.0     70525202
2399.67/602.58	c ................................ | *****.....           14330000 | 923704       3651610          2400.0      5970.8     72745647
2495.92/626.68	c ................................ | *****.....           14762000 | 957808       3751577          2496.2      5913.7     75587088
2591.78/650.68	c ................................ | *****.....           15205000 | 989000       3867284          2592.1      5865.9     78687318
2688.01/674.70	c ................................ | *****.....           15554000 | 1017785      3953928          2688.0      5786.4     79796245
2783.86/698.76	c ................................ | *****.....           15872000 | 1041366      4022055          2784.1      5700.9     81459819
2880.11/722.82	c ................................ | ******....           16162000 | 1065546      4081120          2880.2      5611.4     82508242
2975.98/746.83	c ................................ | ******....           16495000 | 1091489      4142578          2976.1      5542.5     83692168
3071.77/770.87	c ................................ | ******....           16862000 | 1117131      4226185          3072.1      5488.8     85294417
3168.00/794.92	c ................................ | ******....           17309000 | 1138220      4335729          3168.1      5463.5     85583363
3263.79/818.98	c ................................ | ******....           17705000 | 1160804      4433168          3264.1      5424.1     85583363
3359.64/842.99	c ................................ | *******...           18028000 | 1180465      4522830          3360.0      5365.5     85583363
3455.89/867.07	c ................................ | *******...           18375000 | 1198980      4623154          3456.2      5316.6     85583363
3551.71/891.08	c ................................ | *******...           18714000 | 1216999      4714129          3552.0      5268.5     85695823
3647.95/915.12	c ................................ | *******...           19041000 | 1238697      4796319          3648.0      5219.5     86067028
3743.79/939.17	c ................................ | *******...           19327000 | 1261442      4863752          3744.1      5162.0     86401698
3840.04/963.27	c ................................ | ********..           19659000 | 1286182      4951359          3840.3      5119.1     86696534
3936.29/987.32	c ................................ | ********..           19947000 | 1304037      5019726          3936.4      5067.3     87195732
4032.14/1011.34	c ................................ | ********..           20282000 | 1332149      5093718          4032.3      5029.9     87342686
4127.98/1035.34	c ................................ | ********..           20610000 | 1353671      5174163          4128.2      4992.5     88415235
4224.22/1059.41	c ................................ | ********..           21042000 | 1377714      5270177          4224.3      4981.2     90774548
4320.09/1083.45	c ................................ | *********.           21434000 | 1398301      5357431          4320.3      4961.3     92506052
4415.91/1107.47	c ................................ | *********.           21736000 | 1417616      5435833          4416.2      4921.9     94178451
4512.11/1131.53	c ................................ | *********.           22012000 | 1437578      5503850          4512.2      4878.3     94554816
4607.93/1155.58	c ................................ | *********.           22294000 | 1456411      5569279          4608.2      4837.8     94554816
4704.51/1179.73	c ................................ | *********.           22624000 | 1481276      5641460          4704.6      4808.9     94554816
4799.94/1203.64	c CPU limit (4800) reached
4799.94/1203.64	c Time = 4800.10 seconds
4799.94/1203.64	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249724-1304065096/watcher-3249724-1304065096 -o /tmp/evaluation-result-3249724-1304065096/solver-3249724-1304065096 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249724-1304065096.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.94 8.00 7.91 5/181 29780
/proc/meminfo: memFree=21840564/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) R 29778 29780 28847 0 -1 4202496 425 0 0 0 0 0 0 0 25 0 1 0 518108448 9633792 367 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272688896 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/29780/statm: 2352 368 147 36 0 257 0

[startup+0.079248 s]
/proc/loadavg: 7.94 8.00 7.91 5/181 29780
/proc/meminfo: memFree=21840564/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=18664 CPUtime=0.07 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) R 29778 29780 28847 0 -1 4202496 2705 0 0 0 7 0 0 0 25 0 1 0 518108448 19111936 2648 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272349893 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/29780/statm: 4666 2648 148 36 0 2571 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 18664

[startup+0.100242 s]
/proc/loadavg: 7.94 8.00 7.91 5/181 29780
/proc/meminfo: memFree=21840564/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=21268 CPUtime=0.09 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) R 29778 29780 28847 0 -1 4202496 3331 0 0 0 9 0 0 0 25 0 1 0 518108448 21778432 3273 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272093039 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/29780/statm: 5317 3274 148 36 0 3222 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 21268

[startup+0.300212 s]
/proc/loadavg: 7.94 8.00 7.91 5/181 29780
/proc/meminfo: memFree=21840564/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=87448 CPUtime=0.29 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) R 29778 29780 28847 0 -1 4202496 17449 0 0 0 27 2 0 0 25 0 1 0 518108448 89546752 17032 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272350241 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/29780/statm: 21862 17042 569 36 0 19767 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 87448

[startup+0.700142 s]
/proc/loadavg: 7.94 8.00 7.91 5/181 29780
/proc/meminfo: memFree=21840564/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=98356 CPUtime=0.69 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) R 29778 29780 28847 0 -1 4202496 19793 0 0 0 66 3 0 0 25 0 1 0 518108448 100716544 19203 33554432000 4194304 4338100 140736510539040 18446744073709551615 4228108 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/29780/statm: 24589 19204 723 36 0 22494 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 98356

[startup+1.50099 s]
/proc/loadavg: 7.71 7.95 7.90 6/182 29781
/proc/meminfo: memFree=21798032/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=58244 CPUtime=1.49 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) R 29778 29780 28847 0 -1 4202496 23288 0 0 0 146 3 0 0 25 0 1 0 518108448 59641856 12581 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272349893 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/29780/statm: 14561 12581 156 36 0 12466 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 58244

[startup+3.10071 s]
/proc/loadavg: 7.71 7.95 7.90 9/186 29785
/proc/meminfo: memFree=21758232/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=343416 CPUtime=6.09 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 35454 0 0 0 594 15 0 0 25 0 5 0 518108448 351657984 23935 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 85854 23935 176 36 0 83759 0
[pid=29780/tid=29782] ppid=29778 vsize=343416 CPUtime=0.99 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 1846 0 0 0 98 1 0 0 19 0 5 0 518108658 351657984 23935 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514676 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=343416 CPUtime=0.99 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 986 0 0 0 97 2 0 0 19 0 5 0 518108658 351657984 23935 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=343416 CPUtime=0.98 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 649 0 0 0 96 2 0 0 25 0 5 0 518108658 351657984 23935 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514676 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=343416 CPUtime=1 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 874 0 0 0 97 3 0 0 25 0 5 0 518108658 351657984 23935 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 6.09
Current children cumulated vsize (KiB) 343416

[startup+6.30114 s]
/proc/loadavg: 7.71 7.95 7.90 9/186 29785
/proc/meminfo: memFree=21712848/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=371832 CPUtime=18.87 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 42288 0 0 0 1838 49 0 0 25 0 5 0 518108448 380755968 30769 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 92958 30769 176 36 0 90863 0
[pid=29780/tid=29782] ppid=29778 vsize=371832 CPUtime=4.2 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 3948 0 0 0 411 9 0 0 18 0 5 0 518108658 380755968 30769 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=371832 CPUtime=4.19 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 3428 0 0 0 408 11 0 0 22 0 5 0 518108658 380755968 30769 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=371832 CPUtime=4.16 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 1690 0 0 0 405 11 0 0 25 0 5 0 518108658 380755968 30769 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=371832 CPUtime=4.19 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 2123 0 0 0 408 11 0 0 25 0 5 0 518108658 380755968 30769 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 18.87
Current children cumulated vsize (KiB) 371832

[startup+12.701 s]
/proc/loadavg: 7.91 7.98 7.91 10/186 29785
/proc/meminfo: memFree=21641516/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=422408 CPUtime=44.43 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 54582 0 0 0 4310 133 0 0 25 0 5 0 518108448 432545792 43063 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 105602 43063 176 36 0 103507 0
[pid=29780/tid=29782] ppid=29778 vsize=422408 CPUtime=10.59 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 8747 0 0 0 1032 27 0 0 25 0 5 0 518108658 432545792 43063 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=422408 CPUtime=10.59 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 5980 0 0 0 1025 34 0 0 25 0 5 0 518108658 432545792 43063 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=422408 CPUtime=10.52 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 3839 0 0 0 1020 32 0 0 19 0 5 0 518108658 432545792 43063 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=422408 CPUtime=10.59 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 4917 0 0 0 1026 33 0 0 25 0 5 0 518108658 432545792 43063 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 44.43
Current children cumulated vsize (KiB) 422408

[startup+25.5007 s]
/proc/loadavg: 7.93 7.98 7.91 9/186 29786
/proc/meminfo: memFree=21459124/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=579996 CPUtime=95.53 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 94704 0 0 0 9263 290 0 0 25 0 5 0 518108448 593915904 83185 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 144999 83185 176 36 0 142904 0
[pid=29780/tid=29782] ppid=29778 vsize=579996 CPUtime=23.39 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 24257 0 0 0 2282 57 0 0 25 0 5 0 518108658 593915904 83185 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=579996 CPUtime=23.38 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 13842 0 0 0 2262 76 0 0 25 0 5 0 518108658 593915904 83185 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=579996 CPUtime=23.24 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 11450 0 0 0 2251 73 0 0 18 0 5 0 518108658 593915904 83185 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=579996 CPUtime=23.39 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 14056 0 0 0 2262 77 0 0 25 0 5 0 518108658 593915904 83185 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 95.53
Current children cumulated vsize (KiB) 579996

[startup+51.1012 s]
/proc/loadavg: 8.01 8.00 7.91 9/186 29786
/proc/meminfo: memFree=21197980/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=798140 CPUtime=197.77 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 149802 0 0 0 19143 634 0 0 25 0 5 0 518108448 817295360 138283 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 199535 138283 176 36 0 197440 0
[pid=29780/tid=29782] ppid=29778 vsize=798140 CPUtime=48.98 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 45468 0 0 0 4771 127 0 0 25 0 5 0 518108658 817295360 138283 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=798140 CPUtime=48.98 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 25096 0 0 0 4731 167 0 0 25 0 5 0 518108658 817295360 138283 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=798140 CPUtime=48.7 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 22186 0 0 0 4707 163 0 0 25 0 5 0 518108658 817295360 138283 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=798140 CPUtime=48.97 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 25953 0 0 0 4727 170 0 0 25 0 5 0 518108658 817295360 138283 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 197.77
Current children cumulated vsize (KiB) 798140

[startup+102.307 s]
/proc/loadavg: 8.12 8.03 7.93 9/186 29788
/proc/meminfo: memFree=20833272/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=1088368 CPUtime=402.25 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 223890 0 0 0 38892 1333 0 0 25 0 5 0 518108448 1114488832 212371 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 272092 212371 176 36 0 269997 0
[pid=29780/tid=29782] ppid=29778 vsize=1088368 CPUtime=100.16 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 70707 0 0 0 9738 278 0 0 25 0 5 0 518108658 1114488832 212371 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=1088368 CPUtime=100.18 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 40163 0 0 0 9670 348 0 0 20 0 5 0 518108658 1114488832 212371 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=1088368 CPUtime=99.61 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 38446 0 0 0 9614 347 0 0 25 0 5 0 518108658 1114488832 212371 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=1088368 CPUtime=100.17 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 43475 0 0 0 9664 353 0 0 18 0 5 0 518108658 1114488832 212371 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 402.25
Current children cumulated vsize (KiB) 1088368

[startup+162.3 s]
/proc/loadavg: 8.10 8.04 7.93 9/186 29789
/proc/meminfo: memFree=20436832/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=1419316 CPUtime=641.82 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 307486 0 0 0 62121 2061 0 0 25 0 5 0 518108448 1453379584 295967 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 354829 295967 176 36 0 352734 0
[pid=29780/tid=29782] ppid=29778 vsize=1419316 CPUtime=160.15 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 94014 0 0 0 15569 446 0 0 25 0 5 0 518108658 1453379584 295967 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=1419316 CPUtime=160.16 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 64013 0 0 0 15483 533 0 0 21 0 5 0 518108658 1453379584 295967 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=1419316 CPUtime=159.26 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 54314 0 0 0 15396 530 0 0 25 0 5 0 518108658 1453379584 295967 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=1419316 CPUtime=160.14 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 64046 0 0 0 15468 546 0 0 18 0 5 0 518108658 1453379584 295967 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 641.82
Current children cumulated vsize (KiB) 1419316

[startup+222.301 s]
/proc/loadavg: 8.03 8.03 7.93 9/186 29791
/proc/meminfo: memFree=20137212/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=1671008 CPUtime=881.39 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 371061 0 0 0 85252 2887 0 0 25 0 5 0 518108448 1711112192 359542 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 417752 359542 176 36 0 415657 0
[pid=29780/tid=29782] ppid=29778 vsize=1671008 CPUtime=220.11 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 113877 0 0 0 21381 630 0 0 19 0 5 0 518108658 1711112192 359542 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272084393 0 0 4096 0 0 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 150038 0 0 0 67767 2148 0 0 25 0 5 0 518108658 3024691200 683224 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2798.23
Current children cumulated vsize (KiB) 2953800

[startup+762.307 s]
/proc/loadavg: 8.10 8.13 8.02 9/186 29805
/proc/meminfo: memFree=18472448/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3054688 CPUtime=3037.86 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 720098 0 0 0 295003 8783 0 0 25 0 5 0 518108448 3128000512 708579 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 763672 708579 176 36 0 761577 0
[pid=29780/tid=29782] ppid=29778 vsize=3054688 CPUtime=758.71 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 227002 0 0 0 73959 1912 0 0 25 0 5 0 518108658 3128000512 708579 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3054688 CPUtime=760 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 168880 0 0 0 73719 2281 0 0 18 0 5 0 518108658 3128000512 708579 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3054688 CPUtime=757.88 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 137363 0 0 0 73512 2276 0 0 18 0 5 0 518108658 3128000512 708579 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3054688 CPUtime=759.15 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 155754 0 0 0 73608 2307 0 0 18 0 5 0 518108658 3128000512 708579 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3037.86
Current children cumulated vsize (KiB) 3054688

[startup+822.301 s]
/proc/loadavg: 8.19 8.14 8.03 9/186 29807
/proc/meminfo: memFree=18372860/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3134832 CPUtime=3277.38 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 738802 0 0 0 318165 9573 0 0 25 0 5 0 518108448 3210067968 727283 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 783708 727283 176 36 0 781613 0
[pid=29780/tid=29782] ppid=29778 vsize=3134832 CPUtime=818.4 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 234288 0 0 0 79751 2089 0 0 25 0 5 0 518108658 3210067968 727283 33554432000 4194304 4338100 140736510539040 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3134832 CPUtime=819.97 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 173732 0 0 0 79510 2487 0 0 18 0 5 0 518108658 3210067968 727283 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3134832 CPUtime=817.82 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 139804 0 0 0 79301 2481 0 0 18 0 5 0 518108658 3210067968 727283 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3134832 CPUtime=819.07 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 159879 0 0 0 79397 2510 0 0 25 0 5 0 518108658 3210067968 727283 33554432000 4194304 4338100 140736510539040 18446744073709551615 4284185 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3277.38
Current children cumulated vsize (KiB) 3134832

[startup+882.3 s]
/proc/loadavg: 8.13 8.13 8.03 9/186 29809
/proc/meminfo: memFree=18346564/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3136020 CPUtime=3516.98 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 739931 0 0 0 341433 10265 0 0 25 0 5 0 518108448 3211284480 728412 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 784005 728412 176 36 0 781910 0
[pid=29780/tid=29782] ppid=29778 vsize=3136020 CPUtime=878.37 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 234548 0 0 0 85593 2244 0 0 19 0 5 0 518108658 3211284480 728412 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3136020 CPUtime=879.95 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 174023 0 0 0 85326 2669 0 0 18 0 5 0 518108658 3211284480 728412 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3136020 CPUtime=877.8 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 140094 0 0 0 85120 2660 0 0 18 0 5 0 518108658 3211284480 728412 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3136020 CPUtime=878.73 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 160167 0 0 0 85187 2686 0 0 25 0 5 0 518108658 3211284480 728412 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3516.98
Current children cumulated vsize (KiB) 3136020

[startup+942.301 s]
/proc/loadavg: 8.10 8.12 8.03 9/186 29810
/proc/meminfo: memFree=18296080/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3161836 CPUtime=3756.56 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 747525 0 0 0 364695 10961 0 0 25 0 5 0 518108448 3237720064 736006 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 790459 736006 176 36 0 788364 0
[pid=29780/tid=29782] ppid=29778 vsize=3161836 CPUtime=938.35 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 234730 0 0 0 91433 2402 0 0 22 0 5 0 518108658 3237720064 736006 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3161836 CPUtime=939.94 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 181024 0 0 0 91146 2848 0 0 18 0 5 0 518108658 3237720064 736006 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3161836 CPUtime=937.77 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 140300 0 0 0 90934 2843 0 0 21 0 5 0 518108658 3237720064 736006 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3161836 CPUtime=938.36 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 160372 0 0 0 90975 2861 0 0 18 0 5 0 518108658 3237720064 736006 33554432000 4194304 4338100 140736510539040 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3756.56
Current children cumulated vsize (KiB) 3161836

[startup+1002.3 s]
/proc/loadavg: 8.09 8.11 8.03 9/186 29812
/proc/meminfo: memFree=18246828/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3187784 CPUtime=3996.19 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 754400 0 0 0 387990 11629 0 0 25 0 5 0 518108448 3264290816 742881 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 796946 742881 176 36 0 794851 0
[pid=29780/tid=29782] ppid=29778 vsize=3187784 CPUtime=998.35 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 234936 0 0 0 97284 2551 0 0 25 0 5 0 518108658 3264290816 742881 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3187784 CPUtime=999.92 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 185680 0 0 0 96972 3020 0 0 25 0 5 0 518108658 3264290816 742881 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3187784 CPUtime=997.77 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 142093 0 0 0 96755 3022 0 0 25 0 5 0 518108658 3264290816 742881 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3187784 CPUtime=998.03 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 160592 0 0 0 96773 3030 0 0 25 0 5 0 518108658 3264290816 742881 33554432000 4194304 4338100 140736510539040 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3996.19
Current children cumulated vsize (KiB) 3187784

[startup+1062.3 s]
/proc/loadavg: 8.08 8.10 8.03 9/186 29813
/proc/meminfo: memFree=18097256/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3307308 CPUtime=4235.81 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 784428 0 0 0 411284 12297 0 0 25 0 5 0 518108448 3386683392 772909 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 826827 772909 176 36 0 824732 0
[pid=29780/tid=29782] ppid=29778 vsize=3307308 CPUtime=1058.34 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 242930 0 0 0 103132 2702 0 0 25 0 5 0 518108658 3386683392 772909 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3307308 CPUtime=1059.91 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 192850 0 0 0 102799 3192 0 0 25 0 5 0 518108658 3386683392 772909 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3307308 CPUtime=1057.75 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 150064 0 0 0 102576 3199 0 0 25 0 5 0 518108658 3386683392 772909 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3307308 CPUtime=1057.68 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 167485 0 0 0 102571 3197 0 0 25 0 5 0 518108658 3386683392 772909 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4235.81
Current children cumulated vsize (KiB) 3307308

[startup+1122.3 s]
/proc/loadavg: 8.03 8.08 8.02 9/186 29815
/proc/meminfo: memFree=17949560/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3420708 CPUtime=4475.42 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 812987 0 0 0 434659 12883 0 0 25 0 5 0 518108448 3502804992 801468 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 855177 801468 176 36 0 853082 0
[pid=29780/tid=29782] ppid=29778 vsize=3420708 CPUtime=1118.02 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 251384 0 0 0 108972 2830 0 0 25 0 5 0 518108658 3502804992 801468 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272083994 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3420708 CPUtime=1119.9 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 200898 0 0 0 108645 3345 0 0 25 0 5 0 518108658 3502804992 801468 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272084020 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3420708 CPUtime=1117.74 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 156149 0 0 0 108420 3354 0 0 25 0 5 0 518108658 3502804992 801468 33554432000 4194304 4338100 140736510539040 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3420708 CPUtime=1117.63 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 173457 0 0 0 108415 3348 0 0 25 0 5 0 518108658 3502804992 801468 33554432000 4194304 4338100 140736510539040 18446744073709551615 268272084217 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4475.42
Current children cumulated vsize (KiB) 3420708

[startup+1182.3 s]
/proc/loadavg: 5.64 7.41 7.79 5/176 29835
/proc/meminfo: memFree=18948100/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=3421368 CPUtime=4714.9 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) S 29778 29780 28847 0 -1 4202496 813965 0 0 0 457965 13525 0 0 25 0 5 0 518108448 3503480832 802446 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284491749 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 855342 802446 176 36 0 853247 0
[pid=29780/tid=29782] ppid=29778 vsize=3421368 CPUtime=1177.69 cores=1,3,5,7
/proc/29780/task/29782/stat : 29782 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 251704 0 0 0 114801 2968 0 0 25 0 5 0 518108658 3503480832 802446 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29780/tid=29783] ppid=29778 vsize=3421368 CPUtime=1179.83 cores=1,3,5,7
/proc/29780/task/29783/stat : 29783 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 201209 0 0 0 114475 3508 0 0 20 0 5 0 518108658 3503480832 802446 33554432000 4194304 4338100 140736510539040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=3421368 CPUtime=1177.67 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 156322 0 0 0 114239 3528 0 0 25 0 5 0 518108658 3503480832 802446 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29780/tid=29785] ppid=29778 vsize=3421368 CPUtime=1177.58 cores=1,3,5,7
/proc/29780/task/29785/stat : 29785 (ttsth-5-4) R 29778 29780 28847 0 -1 4202560 173631 0 0 0 114243 3515 0 0 25 0 5 0 518108658 3503480832 802446 33554432000 4194304 4338100 140736510539040 18446744073709551615 268284514997 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4714.9
Current children cumulated vsize (KiB) 3421368



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.7 s]
/proc/loadavg: 5.17 7.18 7.71 2/173 29835
/proc/meminfo: memFree=20242012/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=0 CPUtime=4800.17 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) Z 29778 29780 28847 0 -1 4203548 815683 0 0 0 466233 13784 0 0 18 0 2 0 518108448 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 0 0 0 0 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=0 CPUtime=1199.06 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4203588 156392 0 0 0 116307 3599 0 0 25 0 2 0 518108658 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.7 s]
/proc/loadavg: 5.17 7.18 7.71 2/173 29835
/proc/meminfo: memFree=20242012/32950928 swapFree=67111524/67111528
[pid=29780] ppid=29778 vsize=0 CPUtime=4800.17 cores=1,3,5,7
/proc/29780/stat : 29780 (ttsth-5-4) Z 29778 29780 28847 0 -1 4203548 815683 0 0 0 466233 13784 0 0 18 0 2 0 518108448 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/29780/statm: 0 0 0 0 0 0 0
[pid=29780/tid=29784] ppid=29778 vsize=0 CPUtime=1199.06 cores=1,3,5,7
/proc/29780/task/29784/stat : 29784 (ttsth-5-4) R 29778 29780 28847 0 -1 4203588 156392 0 0 0 116307 3599 0 0 25 0 2 0 518108658 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.17
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.78
CPU time (s): 4800.25
CPU user time (s): 4662.33
CPU system time (s): 137.913
CPU usage (%): 398.765
Max. virtual memory (cumulated for all children) (KiB): 3427888

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4662.33
system time used= 137.913
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 815683
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 857476
involuntary context switches= 32529

runsolver used 1.90771 second user time and 4.6103 second system time

The end

Launcher Data 

Begin job on node122 at 2011-04-29 10:18:17
IDJOB=3249724
IDBENCH=83182
IDSOLVER=1654
FILE ID=node122/3249724-1304065096
RUNJOBID= node122-1304063811-29554
PBS_JOBID= 13170447
Free space on /tmp= 71656 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-14x34.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249724-1304065096/watcher-3249724-1304065096 -o /tmp/evaluation-result-3249724-1304065096/solver-3249724-1304065096 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249724-1304065096.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 72677fc46303044d3d9e832fa24a37b2
RANDOM SEED=501329822

node122.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      21840720 kB
Buffers:       1625616 kB
Cached:        8552272 kB
SwapCached:          4 kB
Active:        1322872 kB
Inactive:      9237656 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      21840720 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            5080 kB
Writeback:           0 kB
AnonPages:      381548 kB
Mapped:          15524 kB
Slab:           486108 kB
PageTables:       5432 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   574800 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71660 MiB
End job on node122 at 2011-04-29 10:38:22