EXPERIMENTELL: JsonL Datenbank für js-controller

[crunchip]

@apollon77 hab mich grad daran erinnert, hab ja noch ein script laufen, das liefert folgendes

[{"command":"iobroker.js-controller","pid":"9819","cpu":61.9,"mem":10.4,"vmem":1169840,"rss":583488,"start":"20:50","time":"33:43"},{"command":"io.javascript.0","pid":"11149","cpu":12.9,"mem":6.2,"vmem":1167828,"rss":350740,"start":"20:52","time":"6:49"},{"command":"io.admin.0","pid":"9986","cpu":1,"mem":3,"vmem":929792,"rss":168668,"start":"20:50","time":"0:34"},{"command":"io.iogo.0","pid":"10729","cpu":5.4,"mem":2.4,"vmem":1947004,"rss":136788,"start":"20:52","time":"2:53"},{"command":"io.simple-api.0","pid":"12375","cpu":1.2,"mem":2.1,"vmem":732936,"rss":120760,"start":"20:53","time":"0:38"},{"command":"io.influxdb.2","pid":"10695","cpu":4.4,"mem":1.8,"vmem":675636,"rss":103412,"start":"20:51","time":"2:21"},{"command":"io.influxdb.0","pid":"10516","cpu":4.8,"mem":1.8,"vmem":667648,"rss":101056,"start":"20:51","time":"2:36"},{"command":"io.sonoff.0","pid":"12456","cpu":4.9,"mem":1.7,"vmem":664560,"rss":97656,"start":"20:53","time":"2:32"},{"command":"io.backitup.0","pid":"10016","cpu":0.1,"mem":1.6,"vmem":663928,"rss":93176,"start":"20:50","time":"0:05"},{"command":"io.iot.0","pid":"11123","cpu":0.2,"mem":1.5,"vmem":793480,"rss":88832,"start":"20:52","time":"0:07"},{"command":"io.influxdb.1","pid":"10531","cpu":0.4,"mem":1.5,"vmem":654816,"rss":88220,"start":"20:51","time":"0:14"},{"command":"io.shelly.0","pid":"11502","cpu":1,"mem":1.5,"vmem":660668,"rss":88220,"start":"20:52","time":"0:33"},{"command":"io.radar2.0","pid":"11427","cpu":0.4,"mem":1.4,"vmem":911532,"rss":82720,"start":"20:52","time":"0:13"},{"command":"io.wled.0","pid":"13018","cpu":1.6,"mem":1.4,"vmem":634420,"rss":79948,"start":"20:54","time":"0:49"},{"command":"io.sourceanalytix.0","pid":"12471","cpu":1.4,"mem":1.3,"vmem":632440,"rss":78008,"start":"20:53","time":"0:44"},{"command":"io.web.0","pid":"12980","cpu":0.3,"mem":1.3,"vmem":648700,"rss":77616,"start":"20:54","time":"0:11"},{"command":"io.zigbee.0","pid":"13072","cpu":0.4,"mem":1.3,"vmem":780548,"rss":77480,"start":"20:54","time":"0:12"},{"command":"io.tr-064.0","pid":"12692","cpu":0.3,"mem":1.3,"vmem":643116,"rss":76360,"start":"20:53","time":"0:09"},{"command":"io.virtualpowermeter.0","pid":"12743","cpu":0.6,"mem":1.3,"vmem":648892,"rss":76148,"start":"20:53","time":"0:19"},{"command":"io.telegram.0","pid":"12497","cpu":0.4,"mem":1.3,"vmem":783552,"rss":75044,"start":"20:53","time":"0:13"},{"command":"io.info.0","pid":"10706","cpu":0.1,"mem":1.3,"vmem":781448,"rss":74792,"start":"20:51","time":"0:05"},{"command":"io.telegram.1","pid":"12651","cpu":0.4,"mem":1.3,"vmem":847456,"rss":74664,"start":"20:53","time":"0:12"},{"command":"io.shuttercontrol.0","pid":"11674","cpu":0.2,"mem":1.3,"vmem":645352,"rss":74332,"start":"20:52","time":"0:07"},{"command":"io.linux-control.0","pid":"11164","cpu":0.6,"mem":1.2,"vmem":907652,"rss":72120,"start":"20:52","time":"0:19"},{"command":"io.fully-tablet-control.0","pid":"10307","cpu":0.3,"mem":1.2,"vmem":642736,"rss":70696,"start":"20:51","time":"0:12"},{"command":"io.history.0","pid":"10471","cpu":0.4,"mem":1.2,"vmem":706744,"rss":69844,"start":"20:51","time":"0:14"},{"command":"io.socketio.0","pid":"12434","cpu":0.1,"mem":1.1,"vmem":709680,"rss":66284,"start":"20:53","time":"0:05"},{"command":"io.smartcontrol.0","pid":"12408","cpu":0.3,"mem":1.1,"vmem":638636,"rss":65908,"start":"20:53","time":"0:09"},{"command":"io.tankerkoenig.0","pid":"12482","cpu":0.2,"mem":1.1,"vmem":901992,"rss":64736,"start":"20:53","time":"0:06"},{"command":"io.pollenflug.0","pid":"11393","cpu":0.1,"mem":1.1,"vmem":705264,"rss":64552,"start":"20:52","time":"0:05"},{"command":"io.unifi.0","pid":"30663","cpu":5.8,"mem":1.1,"vmem":635608,"rss":63940,"start":"21:43","time":"0:06"},{"command":"io.alexa2.0","pid":"10001","cpu":0.2,"mem":1.1,"vmem":903352,"rss":63756,"start":"20:50","time":"0:08"},{"command":"io.chromecast.0","pid":"10225","cpu":0.1,"mem":1.1,"vmem":1665116,"rss":63296,"start":"20:51","time":"0:06"},{"command":"io.logparser.0","pid":"11179","cpu":0.2,"mem":1.1,"vmem":619548,"rss":63196,"start":"20:52","time":"0:09"},{"command":"io.trashschedule.0","pid":"12708","cpu":0.1,"mem":1,"vmem":634032,"rss":60344,"start":"20:53","time":"0:04"},{"command":"io.jarvis.0","pid":"11134","cpu":0.1,"mem":1,"vmem":633720,"rss":60168,"start":"20:52","time":"0:04"},{"command":"io.ble.0","pid":"10038","cpu":0.3,"mem":1,"vmem":617376,"rss":60064,"start":"20:51","time":"0:10"},{"command":"io.ping.0","pid":"11378","cpu":0.4,"mem":1,"vmem":619148,"rss":60064,"start":"20:52","time":"0:13"},{"command":"io.milight-smart-light.0","pid":"11359","cpu":0.1,"mem":1,"vmem":634572,"rss":59776,"start":"20:52","time":"0:04"},{"command":"io.wiffi-wz.0","pid":"12997","cpu":0.1,"mem":1,"vmem":634844,"rss":59232,"start":"20:54","time":"0:05"},{"command":"io.sayit.0","pid":"11445","cpu":0.1,"mem":1,"vmem":615444,"rss":57984,"start":"20:52","time":"0:04"},{"command":"io.email.0","pid":"10277","cpu":0.1,"mem":1,"vmem":615524,"rss":57080,"start":"20:51","time":"0:03"},{"command":"io.text2command.0","pid":"12672","cpu":0.1,"mem":1,"vmem":614948,"rss":56676,"start":"20:53","time":"0:04"},{"command":"io.mihome.0","pid":"11341","cpu":0.1,"mem":1,"vmem":615708,"rss":56172,"start":"20:52","time":"0:04"},{"command":"io.yeelight-2.0","pid":"13045","cpu":0.1,"mem":1,"vmem":631564,"rss":55944,"start":"20:54","time":"0:04"},{"command":"io.broadlink2.0","pid":"10061","cpu":0.1,"mem":0.9,"vmem":614528,"rss":55520,"start":"20:51","time":"0:04"},{"command":"io.seq.0","pid":"11475","cpu":0.1,"mem":0.9,"vmem":614256,"rss":54000,"start":"20:52","time":"0:04"},{"command":"io.vis-inventwo.0","pid":"12774","cpu":0.1,"mem":0.9,"vmem":613384,"rss":53872,"start":"20:54","time":"0:03"}]

[crunchip]

@alcalzone es wurde ne 8,1mb große Datei angelegt, weiß auch nicht ob das so korrekt ist, da er nach dem Umwandel Befehl ne ganze Reihe
Move: unknown source....
ausgegeben hat,

jedenfalls ist die Datei zu groß um hier hochgeladen zu werden, wohin soll ich sie schicken?

[AlCalzone]

@crunchip sagte in EXPERIMENTELL: JsonL Datenbank für js-controller:

das sind 59 einzelne *-v8.log Dateien, die da angelegt wurden!!

Öh doof, das ist dann je eine pro Adapterprozess. Kannst du irgendwie ausmachen, welche davon zum js-controller gehört hat? Nur die dann bitte umwandeln (oder jede in eine eigene Datei, nicht alle in dieselbe processed.txt).

Die Dateien kannst du zippen und mir per Mail schicken.

[AlCalzone]

@dr-bakterius Es kann auch sein, dass neue Objekte dazu gekommen sind und dass demnach die Bedingung "doppelt so groß wie nötig" erst später erfüllt wird.

[crunchip]

@alcalzone sagte in EXPERIMENTELL: JsonL Datenbank für js-controller:

nicht alle in dieselbe

haha, du bist lustig, hab doch geschrieben, es wurde eine Datei durchs umwandeln angelegt. Die hab ich auch schon verschickt. Guck mal, ob das was brauchbares ist, wenn nicht, mach ichs nochmal, da ich die Dateien schon wieder rausgelöscht hab

Ich mach das jetzt nochmal,
anstatt mit main.js, nehm ich controller.js, das Ergebnis lass ich dir dann zukommen

[AlCalzone]

@crunchip sagte in EXPERIMENTELL: JsonL Datenbank für js-controller:

es wurde eine Datei durchs umwandeln angelegt

Ja, aber alle Befehle haben die gleiche Datei als Ziel gehabt und ich fürchte jeweils überschrieben. Die von dir gesendete Datei enthält die Auswertung für iobroker.shelly.

Probiers mal so:

for file in *-v8.log; do   node --prof-process --preprocess "$file" > "OUT_$file"; done

Das wertet alle aus und schreibt das Ergebnis jeweils in ne eigene Datei, die mit "OUT_" beginnt.

[crunchip]

@alcalzone ok, mach ich gleich, allerdings bin ich am Rätseln, ob der Befehl "node --prof ..." das ständige starten des controllers verursacht. Siehe https://forum.iobroker.net/post/581582
ich habe nämlich, nachdem ich iobroker manuell starte 20 Einträge.
iobroker wurde um 10:34 beendet, siehe log, 10:39 Befehl node --prof ausgeführt und anschließend iobroker manuell gestartet

Edit: ok, das umwandeln hat geklappt, hab nun alle einzeln, wenn du mir noch verrätst wo ich sehen kann, welche die für den controller ist? die größte Datei hat 83.8mb

[AlCalzone]

@crunchip Pack sie am besten alle in eine ZIP, damit sollte sich die Größe massiv reduzieren, sodass du mir ggf. einfach alle per Mail schicken kannst. Ich will nicht ausschließen, dass der Befehl selbst Probleme verursacht, hab aber gerade keinen Ansatz woran das liegt.

[crunchip]

@alcalzone ok, mach ich

[simatec]

Die Umstellung auf meinem Testsystem lief problemlos durch. Werde nun das ganze testen und beobachten.

[crunchip]

@alcalzone bekomms grad nicht gebacken, wie ich die zip Datei versenden könnte, da sie immer noch 32,5mb und zu groß für Mail ist. Muss ich mal schauen oder zur Not teil ich es auf.

[apollon77]

@crunchip Dropbox? Google Drive? und Freigeben?

[AlCalzone]

Wir haben ein potentielles Performance-Problem im JS-Controller identifiziert. Das müsste so 1:1 auch die klassische File DB betreffen.

[crunchip]

@apollon77 hat sich schon geklärt

[crunchip]

@alcalzone ahja, also schon gleich was gefunden? deshalb auch die cpu des js-controllers so hoch?

[apollon77]

@crunchip Jain ... das file hat ein thema gezeigt wo viel Performance drauf geht was man optimieren kann ... aber wie @AlCalzone gesagt hat sollte das identisch die normale File DB betreffen. Also warum es "mit jsonl db plötzlich höher ist" im CPU verbrauch war nichts offensichtliches zu sehen

[crunchip]

wegen js-controller neustarts hab ich nun getestet, Problem lag dann wirklich nur an dem "node --prof" Befehl

Gesamt iobroker cpu bei über 200% kurz nach dem Neustart, pendelt sich so gerade bei 155% ein

[{"command":"iobroker.js-controller","pid":"2181","cpu":82.3,"mem":9.1,"vmem":1095432,"rss":508712,"start":"14:47","time":"6:06"},{"command":"io.javascript.0","pid":"3647","cpu":32.7,"mem":6,"vmem":1165212,"rss":339588,"start":"14:49","time":"1:52"},{"command":"io.admin.0","pid":"2378","cpu":5.1,"mem":3,"vmem":874488,"rss":172184,"start":"14:47","time":"0:22"},{"command":"io.iogo.0","pid":"3212","cpu":10.1,"mem":2.1,"vmem":1933128,"rss":119652,"start":"14:49","time":"0:36"},{"command":"io.simple-api.0","pid":"4952","cpu":2.7,"mem":1.7,"vmem":682268,"rss":96152,"start":"14:50","time":"0:08"},{"command":"io.backitup.0","pid":"2408","cpu":0.7,"mem":1.6,"vmem":663108,"rss":93352,"start":"14:48","time":"0:03"},{"command":"io.influxdb.0","pid":"2969","cpu":8,"mem":1.6,"vmem":663992,"rss":91604,"start":"14:48","time":"0:30"},{"command":"io.iot.0","pid":"3621","cpu":1.3,"mem":1.6,"vmem":794744,"rss":90332,"start":"14:49","time":"0:04"},{"command":"io.socketio.0","pid":"5007","cpu":0.8,"mem":1.5,"vmem":713808,"rss":88456,"start":"14:50","time":"0:02"},{"command":"io.influxdb.2","pid":"2999","cpu":7.5,"mem":1.5,"vmem":655672,"rss":84036,"start":"14:48","time":"0:27"},{"command":"io.influxdb.1","pid":"2984","cpu":2.3,"mem":1.3,"vmem":652268,"rss":78040,"start":"14:48","time":"0:08"},{"command":"io.shelly.0","pid":"4031","cpu":2.1,"mem":1.3,"vmem":648952,"rss":76452,"start":"14:49","time":"0:06"},{"command":"io.shuttercontrol.0","pid":"4060","cpu":1.3,"mem":1.3,"vmem":650464,"rss":74600,"start":"14:50","time":"0:03"},{"command":"io.virtualpowermeter.0","pid":"5521","cpu":3.2,"mem":1.3,"vmem":647368,"rss":73356,"start":"14:50","time":"0:08"},{"command":"io.zigbee.0","pid":"5898","cpu":2.2,"mem":1.3,"vmem":711332,"rss":73116,"start":"14:51","time":"0:04"},{"command":"io.sonoff.0","pid":"5026","cpu":6.7,"mem":1.2,"vmem":642520,"rss":70156,"start":"14:50","time":"0:19"},{"command":"io.linux-control.0","pid":"3662","cpu":4.7,"mem":1.2,"vmem":904212,"rss":69780,"start":"14:49","time":"0:16"},{"command":"io.tr-064.0","pid":"5298","cpu":1.3,"mem":1.2,"vmem":639808,"rss":69224,"start":"14:50","time":"0:03"},{"command":"io.web.0","pid":"5618","cpu":1.5,"mem":1.2,"vmem":644320,"rss":68676,"start":"14:51","time":"0:03"},{"command":"io.unifi.0","pid":"5332","cpu":4.2,"mem":1.1,"vmem":637712,"rss":66624,"start":"14:50","time":"0:10"},{"command":"io.telegram.0","pid":"5074","cpu":2.1,"mem":1.1,"vmem":773524,"rss":66572,"start":"14:50","time":"0:05"},{"command":"io.radar2.0","pid":"3951","cpu":1.3,"mem":1.1,"vmem":902612,"rss":65816,"start":"14:49","time":"0:04"},{"command":"io.telegram.1","pid":"5238","cpu":2,"mem":1.1,"vmem":837916,"rss":65676,"start":"14:50","time":"0:05"},{"command":"io.history.0","pid":"2923","cpu":2,"mem":1.1,"vmem":704220,"rss":63940,"start":"14:48","time":"0:08"},{"command":"io.smartcontrol.0","pid":"4979","cpu":1.2,"mem":1.1,"vmem":637000,"rss":63272,"start":"14:50","time":"0:03"},{"command":"io.mihome.0","pid":"3694","cpu":0.5,"mem":1.1,"vmem":612384,"rss":63212,"start":"14:49","time":"0:01"},{"command":"io.sourceanalytix.0","pid":"5041","cpu":4.3,"mem":1.1,"vmem":628040,"rss":63044,"start":"14:50","time":"0:12"},{"command":"io.fully-tablet-control.0","pid":"2729","cpu":1,"mem":1.1,"vmem":635484,"rss":62720,"start":"14:48","time":"0:04"},{"command":"io.alexa2.0","pid":"2393","cpu":1.1,"mem":1.1,"vmem":883308,"rss":62212,"start":"14:47","time":"0:04"},{"command":"io.pollenflug.0","pid":"3922","cpu":1,"mem":1.1,"vmem":702068,"rss":61896,"start":"14:49","time":"0:03"},{"command":"io.info.0","pid":"3189","cpu":0.7,"mem":1,"vmem":701368,"rss":59348,"start":"14:49","time":"0:02"},{"command":"io.wled.0","pid":"5653","cpu":3.4,"mem":1,"vmem":616556,"rss":59180,"start":"14:51","time":"0:07"},{"command":"io.tankerkoenig.0","pid":"5056","cpu":0.9,"mem":1,"vmem":699740,"rss":58624,"start":"14:50","time":"0:02"},{"command":"io.logparser.0","pid":"3678","cpu":0.9,"mem":1,"vmem":614992,"rss":58512,"start":"14:49","time":"0:03"},{"command":"io.trashschedule.0","pid":"5315","cpu":0.7,"mem":1,"vmem":632632,"rss":57456,"start":"14:50","time":"0:01"},{"command":"io.wiffi-wz.0","pid":"5635","cpu":0.8,"mem":1,"vmem":631548,"rss":57312,"start":"14:51","time":"0:01"},{"command":"io.milight-smart-light.0","pid":"3888","cpu":0.6,"mem":1,"vmem":631256,"rss":56780,"start":"14:49","time":"0:02"},{"command":"io.ble.0","pid":"2430","cpu":0.6,"mem":0.9,"vmem":612420,"rss":55780,"start":"14:48","time":"0:02"},{"command":"io.chromecast.0","pid":"2647","cpu":0.4,"mem":0.9,"vmem":1662116,"rss":55600,"start":"14:48","time":"0:01"},{"command":"io.jarvis.0","pid":"3632","cpu":0.4,"mem":0.9,"vmem":629756,"rss":54608,"start":"14:49","time":"0:01"},{"command":"io.ping.0","pid":"3903","cpu":0.9,"mem":0.9,"vmem":612412,"rss":54352,"start":"14:49","time":"0:02"},{"command":"io.sayit.0","pid":"3969","cpu":0.5,"mem":0.9,"vmem":612308,"rss":53352,"start":"14:49","time":"0:01"},{"command":"io.text2command.0","pid":"5283","cpu":0.5,"mem":0.9,"vmem":612764,"rss":53288,"start":"14:50","time":"0:01"},{"command":"io.vis-inventwo.0","pid":"5554","cpu":0.5,"mem":0.9,"vmem":610072,"rss":53124,"start":"14:51","time":"0:01"},{"command":"io.broadlink2.0","pid":"2453","cpu":0.3,"mem":0.9,"vmem":612204,"rss":53000,"start":"14:48","time":"0:01"},{"command":"io.yeelight-2.0","pid":"5827","cpu":0.7,"mem":0.9,"vmem":628392,"rss":52644,"start":"14:51","time":"0:01"},{"command":"io.seq.0","pid":"3999","cpu":0.7,"mem":0.9,"vmem":611844,"rss":52400,"start":"14:49","time":"0:02"},{"command":"io.email.0","pid":"2699","cpu":0.3,"mem":0.9,"vmem":612620,"rss":51616,"start":"14:48","time":"0:01"}]

[AlCalzone]

@crunchip wie @apollon77 schon geschrieben hat, nichts auffälliges was ich direkt mit jsonl in Verbindung bringen würde. Ich werde später auch noch mal nen test starten wo ich sehr viele Schreibzugriffe generiere, vielleicht sieht man da noch was anderes.

[Dr. Bakterius]

@alcalzone, @apollon77 Also nachdem bis jetzt die objects.jsonl nicht neu geschrieben wurde, habe ich mal den ioBroker beendet und das File vorher und nachher verglichen.

• vorher: ....... 46.209.486 Byte - 21.201 Zeilen
• nachher: .... 15.569.482 Byte - 10.744 Zeilen

Da müsste doch schon lange das File neu geschrieben worden sein, oder?

Und noch etwas ist mir aufgefallen:

user@iobroker:~$ iobroker upgrade self
No connection to databases possible ...

Besteht da ein Zusammenhang mit der neuen Datenbank?

[apollon77]

@dr-bakterius naja doppelte grösse ... 10744*2 = 21.488 ... Vllt wäre es "bald" gegangen :-))

Den Fehler kann ich bei mir nicht nachstellen ... master/slave oder single host?