Trace number 3258614

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.24 1202.88

General information on the benchmark

NameCRAFTED/rbsat/random/forced/
rbsat-v1375c111739gyes10.cnf
MD5SUMc904604702db7cfabf92e607e197d409
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark8.17476
Satisfiable
(Un)Satisfiability was proved
Number of variables1375
Number of clauses111739
Sum of the clauses size224743
Maximum clause length25
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2111684
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 555

Solver Data 

0.05/0.07	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258614-1304052149.cnf 
0.05/0.07	c Using SATTIMEOUT of 4800
0.05/0.07	c Using SATRAM of 15500
0.05/0.07	c Size of subset node = 32
0.05/0.07	c Variables = 1375
0.05/0.07	c Clauses = 111739
0.05/0.07	c Literals = 224743
0.05/0.07	c Reduced size:
0.05/0.07	c Variables = 1375
0.05/0.07	c Clauses = 111739
0.05/0.07	c Literals = 224743
0.05/0.07	c a2
0.09/0.14	c a1
0.19/0.27	c Independent sub-problems = 1
0.19/0.27	c Sub-problem with 1375 variables and 111739 clauses
0.19/0.27	c using permutation method 4
0.19/0.27	c count = 1
0.19/0.27	c Ordering finished 0.260 seconds
0.19/0.27	c mapped 0.270 seconds
0.19/0.27	c permuted 0.270 seconds
0.19/0.27	c about to sort
0.19/0.28	c sorted
0.49/0.53	c clauses now 111739
0.49/0.53	c init finished 0.300 seconds
0.49/0.53	c tree built 0.520 seconds
0.49/0.53	c 
0.49/0.53	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.49/0.53	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.49/0.53	c ................................ | ..........                    |                                  0.5         0.0             
0.49/0.53	c ................................ | ..........                  1 |              2                   0.5         1.9             
0.49/0.53	c ................................ | ..........                  2 |              2                   0.5         3.8             
0.49/0.53	c ................................ | ..........                  3 |              2                   0.5         5.7             
0.49/0.53	c ................................ | ..........                  4 |              2                   0.5         7.5             
0.49/0.53	c ................................ | ..........                  5 |              2                   0.5         9.4             
0.49/0.54	c ................................ | ..........               1000 | 2            146                 0.5      1851.9         7412
95.84/24.44	c ................................ | ..........            3867000 | 95866        374541             96.0     40272.9     16668827
191.70/48.48	c ................................ | ..........            6919000 | 169355       669776            192.0     36032.7     29763544
287.93/72.51	c ................................ | ..........            9610000 | 236430       927243            288.0     33368.1     39017315
383.79/96.56	c ................................ | ..........           12050000 | 293360       1151003           384.1     31376.1     47132813
479.66/120.58	c ................................ | *.........           14148000 | 349175       1344427           480.0     29475.0     54588428
575.89/144.63	c ................................ | *.........           16544000 | 414660       1596418           576.0     28721.7     61737555
671.71/168.68	c ................................ | *.........           19094000 | 498431       1813723           672.0     28412.8     64519485
767.91/192.72	c ................................ | *.........           21490000 | 560719       2023497           768.0     27981.8     66656901
863.76/216.76	c ................................ | *.........           23698000 | 616354       2220158           864.0     27427.3     70223335
959.96/240.82	c ................................ | **........           25616000 | 665169       2401476           960.0     26682.2     72427318
1055.80/264.85	c ................................ | **........           27475000 | 716720       2588476          1056.0     26017.3     78821914
1151.66/288.89	c ................................ | **........           29533000 | 770922       2781064          1152.0     25636.3     81138913
1247.87/312.93	c ................................ | **........           31846000 | 835550       2971961          1248.0     25517.4     83186472
1343.71/336.98	c ................................ | **........           34006000 | 900014       3151801          1344.0     25301.5     84973988
1439.94/361.02	c ................................ | ***.......           36088000 | 963488       3309946          1440.0     25060.2     88842603
1535.73/385.07	c ................................ | ***.......           38051000 | 1026620      3490525          1536.0     24772.5     92437178
1631.95/409.11	c ................................ | ***.......           40093000 | 1091446      3679562          1632.0     24566.8     93688367
1727.81/433.15	c ................................ | ***.......           41991000 | 1142246      3858783          1728.0     24299.9     94132379
1823.65/457.19	c ................................ | ***.......           43924000 | 1193908      4022019          1824.0     24081.0     98028782
1919.83/481.24	c ................................ | ****......           45827000 | 1247472      4189035          1920.0     23868.1    102709606
2015.66/505.29	c ................................ | ****......           47502000 | 1291839      4347647          2016.0     23561.9    104076379
2111.90/529.33	c ................................ | ****......           49221000 | 1337162      4518325          2112.0     23305.0    109867305
2207.71/553.38	c ................................ | ****......           50761000 | 1386546      4677370          2208.0     22989.2    115570558
2303.93/577.41	c ................................ | ****......           52220000 | 1430453      4830655          2304.0     22664.9    119992962
2399.69/601.48	c ................................ | *****.....           53571000 | 1473930      4974059          2400.0     22320.9    119992962
2495.94/625.53	c ................................ | *****.....           54865000 | 1517154      5110300          2496.0     21980.8    119992962
2591.72/649.57	c ................................ | *****.....           56453000 | 1561766      5254422          2592.0     21779.5    120557003
2687.96/673.61	c ................................ | *****.....           58000000 | 1601863      5440028          2688.0     21577.2    121637924
2783.78/697.66	c ................................ | *****.....           59513000 | 1642164      5603619          2784.0     21376.6    124557443
2880.01/721.70	c ................................ | ******....           61249000 | 1689243      5770687          2880.0     21266.8    126846822
2975.84/745.75	c ................................ | ******....           63036000 | 1738206      5936360          2976.0     21181.2    129888285
3071.68/769.78	c ................................ | ******....           64619000 | 1782476      6085734          3072.0     21034.8    133811533
3167.89/793.84	c ................................ | ******....           66046000 | 1825283      6223871          3168.0     20847.7    134179229
3263.70/817.87	c ................................ | ******....           67355000 | 1864636      6355538          3264.0     20635.7    134802601
3359.91/841.92	c ................................ | *******...           68722000 | 1905020      6475848          3360.0     20453.0    135356011
3455.70/865.98	c ................................ | *******...           70102000 | 1939350      6612195          3456.1     20283.9    135374549
3551.89/890.03	c ................................ | *******...           71568000 | 1975819      6749997          3552.0     20148.6    135374549
3647.71/914.08	c ................................ | *******...           72959000 | 2011187      6886628          3648.0     19999.6    135990423
3743.96/938.12	c ................................ | *******...           74377000 | 2047417      7029473          3744.1     19865.4    137873163
3839.80/962.16	c ................................ | ********..           75587000 | 2080004      7152711          3840.0     19684.0    138763492
3935.99/986.22	c ................................ | ********..           76767000 | 2111588      7274619          3936.1     19503.5    141460166
4031.78/1010.25	c ................................ | ********..           77913000 | 2143204      7395566          4032.0     19323.7    144411216
4127.64/1034.29	c ................................ | ********..           79148000 | 2177265      7517560          4128.0     19173.4    148734812
4223.86/1058.34	c ................................ | ********..           80831000 | 2229444      7641960          4224.0     19136.0    150931424
4320.02/1082.40	c ................................ | *********.           82347000 | 2277143      7759372          4320.0     19061.7    153083248
4415.78/1106.45	c ................................ | *********.           83800000 | 2317179      7871464          4416.0     18976.4    154987447
4511.98/1130.52	c ................................ | *********.           85181000 | 2357660      7983089          4512.1     18878.5    157746531
4607.77/1154.58	c ................................ | *********.           86364000 | 2391249      8077057          4608.1     18741.9    159742526
4703.89/1178.64	c ................................ | *********.           87560000 | 2426545      8224867          4704.1     18613.7    160128513
4799.67/1202.69	c CPU limit (4800) reached
4799.67/1202.69	c Time = 4800.04 seconds
4799.67/1202.69	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258614-1304052149/watcher-3258614-1304052149 -o /tmp/evaluation-result-3258614-1304052149/solver-3258614-1304052149 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258614-1304052149.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.72 7.99 7.93 6/181 30573
/proc/meminfo: memFree=10001528/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=9760 CPUtime=0 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) R 30571 30573 30368 0 -1 4202496 497 0 0 0 0 0 0 0 25 0 1 0 516814534 9994240 439 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30573/statm: 2440 441 147 36 0 345 0

[startup+0.061214 s]
/proc/loadavg: 7.72 7.99 7.93 6/181 30573
/proc/meminfo: memFree=10001528/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=21008 CPUtime=0.05 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) R 30571 30573 30368 0 -1 4202496 3262 0 0 0 5 0 0 0 25 0 1 0 516814534 21512192 3204 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30573/statm: 5252 3206 148 36 0 3157 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 21008

[startup+0.100208 s]
/proc/loadavg: 7.72 7.99 7.93 6/181 30573
/proc/meminfo: memFree=10001528/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=31268 CPUtime=0.09 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) R 30571 30573 30368 0 -1 4202496 5850 0 0 0 9 0 0 0 25 0 1 0 516814534 32018432 5791 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473664043 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30573/statm: 7817 5791 153 36 0 5722 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 31268

[startup+0.300237 s]
/proc/loadavg: 7.72 7.99 7.93 6/181 30573
/proc/meminfo: memFree=10001528/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=33040 CPUtime=0.29 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) R 30571 30573 30368 0 -1 4202496 8565 0 0 0 28 1 0 0 25 0 1 0 516814534 33832960 6296 33554432000 4194304 4338100 140736028544272 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30573/statm: 8260 6296 163 36 0 6165 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 33040

[startup+1.45497 s]
/proc/loadavg: 7.72 7.99 7.93 6/181 30573
/proc/meminfo: memFree=10001528/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=296148 CPUtime=4.2 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 30294 0 0 0 407 13 0 0 25 0 5 0 516814534 303255552 27805 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 74037 27805 176 36 0 71942 0
Current children cumulated CPU time (s) 4.2
Current children cumulated vsize (KiB) 296148

[startup+1.50096 s]
/proc/loadavg: 7.72 7.99 7.93 6/181 30573
/proc/meminfo: memFree=10001528/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=296148 CPUtime=4.39 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 30685 0 0 0 425 14 0 0 25 0 5 0 516814534 303255552 28196 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 74037 28197 176 36 0 71942 0
Current children cumulated CPU time (s) 4.39
Current children cumulated vsize (KiB) 296148

[startup+3.10068 s]
/proc/loadavg: 7.72 7.99 7.93 6/182 30606
/proc/meminfo: memFree=14948180/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=346592 CPUtime=10.77 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 43306 0 0 0 1050 27 0 0 25 0 5 0 516814534 354910208 40817 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 86648 40818 176 36 0 84553 0
[pid=30573/tid=30576] ppid=30571 vsize=346592 CPUtime=2.55 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 8572 0 0 0 254 1 0 0 22 0 5 0 516814587 354910208 40818 33554432000 4194304 4338100 140736028544272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=346592 CPUtime=2.54 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 3279 0 0 0 244 10 0 0 23 0 5 0 516814587 354910208 40818 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473362564 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=346592 CPUtime=2.55 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 6949 0 0 0 254 1 0 0 22 0 5 0 516814587 354910208 40818 33554432000 4194304 4338100 140736028544272 18446744073709551615 4263517 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=346592 CPUtime=2.55 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 3602 0 0 0 245 10 0 0 23 0 5 0 516814587 354910208 40818 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473362541 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 10.77
Current children cumulated vsize (KiB) 346592

[startup+6.30112 s]
/proc/loadavg: 7.75 7.99 7.93 8/186 30610
/proc/meminfo: memFree=14679200/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=438048 CPUtime=23.55 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 65476 0 0 0 2303 52 0 0 25 0 5 0 516814534 448561152 62987 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 109512 62987 176 36 0 107417 0
[pid=30573/tid=30576] ppid=30571 vsize=438048 CPUtime=5.76 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 17231 0 0 0 572 4 0 0 25 0 5 0 516814587 448561152 62987 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=438048 CPUtime=5.72 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 6527 0 0 0 552 20 0 0 25 0 5 0 516814587 448561152 62987 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=438048 CPUtime=5.76 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 13745 0 0 0 573 3 0 0 25 0 5 0 516814587 448561152 62987 33554432000 4194304 4338100 140736028544272 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=438048 CPUtime=5.74 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 7068 0 0 0 553 21 0 0 25 0 5 0 516814587 448561152 62987 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.55
Current children cumulated vsize (KiB) 438048

[startup+12.701 s]
/proc/loadavg: 7.77 7.99 7.93 9/186 30610
/proc/meminfo: memFree=14415964/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=586096 CPUtime=49.13 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 103150 0 0 0 4810 103 0 0 25 0 5 0 516814534 600162304 100661 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 146524 100661 176 36 0 144429 0
[pid=30573/tid=30576] ppid=30571 vsize=586096 CPUtime=12.15 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 32272 0 0 0 1209 6 0 0 25 0 5 0 516814587 600162304 100661 33554432000 4194304 4338100 140736028544272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=586096 CPUtime=12.14 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 11440 0 0 0 1170 44 0 0 25 0 5 0 516814587 600162304 100661 33554432000 4194304 4338100 140736028544272 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=586096 CPUtime=12.16 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 25721 0 0 0 1211 5 0 0 25 0 5 0 516814587 600162304 100661 33554432000 4194304 4338100 140736028544272 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=586096 CPUtime=12.12 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 12812 0 0 0 1168 44 0 0 25 0 5 0 516814587 600162304 100661 33554432000 4194304 4338100 140736028544272 18446744073709551615 4256369 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 49.13
Current children cumulated vsize (KiB) 586096

[startup+25.5007 s]
/proc/loadavg: 7.82 7.99 7.93 9/186 30611
/proc/meminfo: memFree=14036056/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=816436 CPUtime=100.24 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 160050 0 0 0 9817 207 0 0 25 0 5 0 516814534 836030464 157561 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 204109 157562 176 36 0 202014 0
[pid=30573/tid=30576] ppid=30571 vsize=816436 CPUtime=24.95 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 59752 0 0 0 2481 14 0 0 25 0 5 0 516814587 836030464 157562 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=816436 CPUtime=24.93 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 16151 0 0 0 2403 90 0 0 25 0 5 0 516814587 836030464 157562 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473362591 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=816436 CPUtime=24.96 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 43807 0 0 0 2488 8 0 0 25 0 5 0 516814587 836030464 157562 33554432000 4194304 4338100 140736028544272 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=816436 CPUtime=24.84 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 19436 0 0 0 2393 91 0 0 25 0 5 0 516814587 836030464 157562 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473362512 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.24
Current children cumulated vsize (KiB) 816436

[startup+51.1002 s]
/proc/loadavg: 7.88 7.99 7.93 9/186 30611
/proc/meminfo: memFree=13326336/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=1245640 CPUtime=202.47 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 268016 0 0 0 19824 423 0 0 25 0 5 0 516814534 1275535360 264087 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 311410 264087 176 36 0 309315 0
[pid=30573/tid=30576] ppid=30571 vsize=1245640 CPUtime=50.54 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 104704 0 0 0 5026 28 0 0 25 0 5 0 516814587 1275535360 264087 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473362857 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=1245640 CPUtime=50.52 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 29968 0 0 0 4863 189 0 0 25 0 5 0 516814587 1275535360 264087 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485780174 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=1245640 CPUtime=50.55 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 77474 0 0 0 5041 14 0 0 25 0 5 0 516814587 1275535360 264087 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473362662 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=1245640 CPUtime=50.3 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 34965 0 0 0 4842 188 0 0 25 0 5 0 516814587 1275535360 264087 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 202.47
Current children cumulated vsize (KiB) 1245640

[startup+102.301 s]
/proc/loadavg: 7.95 7.99 7.93 9/186 30613
/proc/meminfo: memFree=12421744/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=1851304 CPUtime=406.95 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 417427 0 0 0 39865 830 0 0 25 0 5 0 516814534 1895735296 413498 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 462826 413498 176 36 0 460731 0
[pid=30573/tid=30576] ppid=30571 vsize=1851304 CPUtime=101.73 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 180204 0 0 0 10121 52 0 0 25 0 5 0 516814587 1895735296 413498 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485780262 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=1851304 CPUtime=101.71 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 39687 0 0 0 9798 373 0 0 25 0 5 0 516814587 1895735296 413498 33554432000 4194304 4338100 140736028544272 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=1851304 CPUtime=101.75 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 126174 0 0 0 10149 26 0 0 25 0 5 0 516814587 1895735296 413498 33554432000 4194304 4338100 140736028544272 18446744073709551615 4284509 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=1851304 CPUtime=101.2 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 50457 0 0 0 9745 375 0 0 25 0 5 0 516814587 1895735296 413498 33554432000 4194304 4338100 140736028544272 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 406.95
Current children cumulated vsize (KiB) 1851304

[startup+162.301 s]
/proc/loadavg: 8.05 8.01 7.93 9/186 30614
/proc/meminfo: memFree=11624788/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=2388136 CPUtime=646.57 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 548300 0 0 0 63374 1283 0 0 25 0 5 0 516814534 2445451264 544371 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 597034 544371 176 36 0 594939 0
[pid=30573/tid=30576] ppid=30571 vsize=2388136 CPUtime=161.72 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 249441 0 0 0 16095 77 0 0 25 0 5 0 516814587 2445451264 544371 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=2388136 CPUtime=161.7 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 52746 0 0 0 15590 580 0 0 25 0 5 0 516814587 2445451264 544371 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=2388136 CPUtime=161.64 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 159416 0 0 0 16130 34 0 0 25 0 5 0 516814587 2445451264 544371 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=2388136 CPUtime=160.96 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 65792 0 0 0 15507 589 0 0 25 0 5 0 516814587 2445451264 544371 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.57
Current children cumulated vsize (KiB) 2388136

[startup+222.301 s]
/proc/loadavg: 8.15 8.03 7.94 9/186 30616
/proc/meminfo: memFree=11176640/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=2592248 CPUtime=886.13 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 600220 0 0 0 86912 1701 0 0 25 0 5 0 516814534 2654461952 596291 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 648062 596291 176 36 0 645967 0
[pid=30573/tid=30576] ppid=30571 vsize=2592248 CPUtime=221.69 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 294303 0 0 0 22073 96 0 0 25 0 5 0 516814587 2654461952 596291 33554432000 4194304 4338100 140736028544272 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 87847 0 0 0 67731 2276 0 0 25 0 5 0 516814587 4441210880 1027351 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2802.55
Current children cumulated vsize (KiB) 4337120

[startup+762.301 s]
/proc/loadavg: 8.00 8.04 7.99 9/186 30630
/proc/meminfo: memFree=7650188/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=4662676 CPUtime=3042.13 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1096665 0 0 0 299024 5189 0 0 25 0 5 0 516814534 4774580224 1092736 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1165669 1092736 176 36 0 1163574 0
[pid=30573/tid=30576] ppid=30571 vsize=4662676 CPUtime=761.4 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 600932 0 0 0 75907 233 0 0 25 0 5 0 516814587 4774580224 1092736 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=4662676 CPUtime=760.86 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 56863 0 0 0 73679 2407 0 0 25 0 5 0 516814587 4774580224 1092736 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=4662676 CPUtime=759.25 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 330075 0 0 0 75843 82 0 0 25 0 5 0 516814587 4774580224 1092736 33554432000 4194304 4338100 140736028544272 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=4662676 CPUtime=760.05 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 87890 0 0 0 73542 2463 0 0 25 0 5 0 516814587 4774580224 1092736 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3042.13
Current children cumulated vsize (KiB) 4662676

[startup+822.3 s]
/proc/loadavg: 8.15 8.07 8.00 9/186 30632
/proc/meminfo: memFree=7421244/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=4722372 CPUtime=3281.69 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1112317 0 0 0 322642 5527 0 0 25 0 5 0 516814534 4835708928 1108388 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1180593 1108388 176 36 0 1178498 0
[pid=30573/tid=30576] ppid=30571 vsize=4722372 CPUtime=821.36 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 613145 0 0 0 81894 242 0 0 25 0 5 0 516814587 4835708928 1108388 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473965627 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=4722372 CPUtime=820.83 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 57268 0 0 0 79515 2568 0 0 25 0 5 0 516814587 4835708928 1108388 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=4722372 CPUtime=818.93 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 333027 0 0 0 81809 84 0 0 25 0 5 0 516814587 4835708928 1108388 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473363194 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=4722372 CPUtime=820.01 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 87972 0 0 0 79372 2629 0 0 25 0 5 0 516814587 4835708928 1108388 33554432000 4194304 4338100 140736028544272 18446744073709551615 274473362681 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3281.69
Current children cumulated vsize (KiB) 4722372

[startup+882.301 s]
/proc/loadavg: 8.24 8.10 8.01 9/186 30633
/proc/meminfo: memFree=7272004/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=4741404 CPUtime=3521.17 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1121572 0 0 0 346271 5846 0 0 25 0 5 0 516814534 4855197696 1117643 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1185351 1117643 176 36 0 1183256 0
[pid=30573/tid=30576] ppid=30571 vsize=4741404 CPUtime=881.32 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 617679 0 0 0 87883 249 0 0 25 0 5 0 516814587 4855197696 1117643 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=4741404 CPUtime=880.76 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 59135 0 0 0 85355 2721 0 0 25 0 5 0 516814587 4855197696 1117643 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=4741404 CPUtime=878.55 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 333093 0 0 0 87771 84 0 0 25 0 5 0 516814587 4855197696 1117643 33554432000 4194304 4338100 140736028544272 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=4741404 CPUtime=879.97 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 90760 0 0 0 85209 2788 0 0 25 0 5 0 516814587 4855197696 1117643 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3521.17
Current children cumulated vsize (KiB) 4741404

[startup+942.306 s]
/proc/loadavg: 8.09 8.08 8.01 9/186 30635
/proc/meminfo: memFree=7008584/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=4873472 CPUtime=3760.75 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1146625 0 0 0 369902 6173 0 0 25 0 5 0 516814534 4990435328 1142696 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1218368 1142696 176 36 0 1216273 0
[pid=30573/tid=30576] ppid=30571 vsize=4873472 CPUtime=941.13 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 619699 0 0 0 93859 254 0 0 25 0 5 0 516814587 4990435328 1142696 33554432000 4194304 4338100 140736028544272 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=4873472 CPUtime=940.74 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 59511 0 0 0 91198 2876 0 0 25 0 5 0 516814587 4990435328 1142696 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=4873472 CPUtime=938.36 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 354616 0 0 0 93745 91 0 0 25 0 5 0 516814587 4990435328 1142696 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=4873472 CPUtime=939.97 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 91894 0 0 0 91049 2948 0 0 25 0 5 0 516814587 4990435328 1142696 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3760.75
Current children cumulated vsize (KiB) 4873472

[startup+1002.3 s]
/proc/loadavg: 8.10 8.08 8.01 9/186 30637
/proc/meminfo: memFree=6673256/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=5055456 CPUtime=4000.22 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1182603 0 0 0 393557 6465 0 0 25 0 5 0 516814534 5176786944 1178674 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1263864 1178674 176 36 0 1261769 0
[pid=30573/tid=30576] ppid=30571 vsize=5055456 CPUtime=1000.76 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 633699 0 0 0 99814 262 0 0 25 0 5 0 516814587 5176786944 1178674 33554432000 4194304 4338100 140736028544272 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=5055456 CPUtime=1000.68 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 60065 0 0 0 97054 3014 0 0 25 0 5 0 516814587 5176786944 1178674 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=5055456 CPUtime=998.31 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 375512 0 0 0 99733 98 0 0 25 0 5 0 516814587 5176786944 1178674 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=5055456 CPUtime=999.92 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 92422 0 0 0 96904 3088 0 0 25 0 5 0 516814587 5176786944 1178674 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4000.22
Current children cumulated vsize (KiB) 5055456

[startup+1062.3 s]
/proc/loadavg: 8.10 8.08 8.01 9/186 30638
/proc/meminfo: memFree=6348236/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=5321600 CPUtime=4239.82 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1248852 0 0 0 417181 6801 0 0 25 0 5 0 516814534 5449318400 1244923 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1330400 1244923 176 36 0 1328305 0
[pid=30573/tid=30576] ppid=30571 vsize=5321600 CPUtime=1060.4 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 673835 0 0 0 105762 278 0 0 25 0 5 0 516814587 5449318400 1244923 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=5321600 CPUtime=1060.65 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 60119 0 0 0 102899 3166 0 0 25 0 5 0 516814587 5449318400 1244923 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=5321600 CPUtime=1058.29 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 401507 0 0 0 105723 106 0 0 25 0 5 0 516814587 5449318400 1244923 33554432000 4194304 4338100 140736028544272 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=5321600 CPUtime=1059.91 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 92486 0 0 0 102745 3246 0 0 25 0 5 0 516814587 5449318400 1244923 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4239.82
Current children cumulated vsize (KiB) 5321600

[startup+1122.3 s]
/proc/loadavg: 8.09 8.08 8.01 9/186 30649
/proc/meminfo: memFree=6100156/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=5471876 CPUtime=4479.23 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1287042 0 0 0 440813 7110 0 0 25 0 5 0 516814534 5603201024 1283113 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1367969 1283113 176 36 0 1365874 0
[pid=30573/tid=30576] ppid=30571 vsize=5471876 CPUtime=1120 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 698546 0 0 0 111710 290 0 0 25 0 5 0 516814587 5603201024 1283113 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=5471876 CPUtime=1120.59 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 60180 0 0 0 108750 3309 0 0 25 0 5 0 516814587 5603201024 1283113 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=5471876 CPUtime=1118.22 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 414861 0 0 0 111711 111 0 0 25 0 5 0 516814587 5603201024 1283113 33554432000 4194304 4338100 140736028544272 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=5471876 CPUtime=1119.86 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 92550 0 0 0 108590 3396 0 0 25 0 5 0 516814587 5603201024 1283113 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4479.23
Current children cumulated vsize (KiB) 5471876

[startup+1182.3 s]
/proc/loadavg: 8.03 8.06 8.00 9/186 30650
/proc/meminfo: memFree=5830824/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=5619156 CPUtime=4718.65 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) S 30571 30573 30368 0 -1 4202496 1324099 0 0 0 464450 7415 0 0 25 0 5 0 516814534 5754015744 1320170 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 1404789 1320170 176 36 0 1402694 0
[pid=30573/tid=30576] ppid=30571 vsize=5619156 CPUtime=1179.71 cores=1,3,5,7
/proc/30573/task/30576/stat : 30576 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 719465 0 0 0 117672 299 0 0 25 0 5 0 516814587 5754015744 1320170 33554432000 4194304 4338100 140736028544272 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30573/tid=30577] ppid=30571 vsize=5619156 CPUtime=1180.52 cores=1,3,5,7
/proc/30573/task/30577/stat : 30577 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 60253 0 0 0 114598 3454 0 0 25 0 5 0 516814587 5754015744 1320170 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=5619156 CPUtime=1178.03 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 430826 0 0 0 117687 116 0 0 25 0 5 0 516814587 5754015744 1320170 33554432000 4194304 4338100 140736028544272 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30573/tid=30579] ppid=30571 vsize=5619156 CPUtime=1179.82 cores=1,3,5,7
/proc/30573/task/30579/stat : 30579 (ttsth-5-4) R 30571 30573 30368 0 -1 4202560 92650 0 0 0 114440 3542 0 0 25 0 5 0 516814587 5754015744 1320170 33554432000 4194304 4338100 140736028544272 18446744073709551615 274485793461 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4718.65
Current children cumulated vsize (KiB) 5619156



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.7 s]
/proc/loadavg: 8.02 8.06 8.00 6/183 30651
/proc/meminfo: memFree=5899380/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=0 CPUtime=4800.05 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) Z 30571 30573 30368 0 -1 4203548 1343482 0 0 0 472470 7535 0 0 18 0 2 0 516814534 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 0 0 0 0 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=0 CPUtime=1198.31 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4203588 441187 0 0 0 119710 121 0 0 25 0 2 0 516814587 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.05
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.81 s]
/proc/loadavg: 8.02 8.06 8.00 6/183 30651
/proc/meminfo: memFree=5899380/32950928 swapFree=67111528/67111528
[pid=30573] ppid=30571 vsize=0 CPUtime=4800.16 cores=1,3,5,7
/proc/30573/stat : 30573 (ttsth-5-4) Z 30571 30573 30368 0 -1 4203548 1343482 0 0 0 472470 7546 0 0 18 0 2 0 516814534 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30573/statm: 0 0 0 0 0 0 0
[pid=30573/tid=30578] ppid=30571 vsize=0 CPUtime=1198.41 cores=1,3,5,7
/proc/30573/task/30578/stat : 30578 (ttsth-5-4) R 30571 30573 30368 0 -1 4203588 441187 0 0 0 119710 131 0 0 25 0 2 0 516814587 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.16
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.88
CPU time (s): 4800.24
CPU user time (s): 4724.7
CPU system time (s): 75.5345
CPU usage (%): 399.061
Max. virtual memory (cumulated for all children) (KiB): 5697488

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4724.7
system time used= 75.5345
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1343482
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7106
involuntary context switches= 25607

runsolver used 2.0027 second user time and 4.57031 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 06:42:29
IDJOB=3258614
IDBENCH=71397
IDSOLVER=1654
FILE ID=node125/3258614-1304052149
RUNJOBID= node125-1304049739-30387
PBS_JOBID= 13170412
Free space on /tmp= 72760 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v1375c111739gyes10.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258614-1304052149/watcher-3258614-1304052149 -o /tmp/evaluation-result-3258614-1304052149/solver-3258614-1304052149 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258614-1304052149.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c904604702db7cfabf92e607e197d409
RANDOM SEED=166370810

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      10001808 kB
Buffers:       2218192 kB
Cached:       14975456 kB
SwapCached:          0 kB
Active:       14196212 kB
Inactive:      8213212 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      10001808 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            1184 kB
Writeback:           0 kB
AnonPages:     5215088 kB
Mapped:          15212 kB
Slab:           468408 kB
PageTables:      14808 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  5544432 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72760 MiB
End job on node125 at 2011-04-29 07:02:33