Skip to content

Dm 1651 filewriter pool

Afonso Mukai requested to merge DM-1651_filewriter_pool into master

Created by: SkyToGround

As requested, here is the worker pool functionality as a single PR. Scanning through the code, I recognised of a bunch of shortcommings, lets see if you can find them! Still, I have been using this branch in "production-testing" for the past few weeks and things seems to be mostly working fine. Enjoy the review!

Changes

  • The command system has been re-done. The code is (currently) the documentation but it is stronly suggested that instead of reading that, you use the (file-writer-control)[https://github.com/ess-dmsc/file-writer-control] Python library to control the file-writer.
  • Some old and now out of date documentation on controlling the file-writer has been deleted.
  • Command and status topics have now been merged into one command + status topic. The command line argument has thus also been changed to --command-status-uri.
  • There is now a job pool functionality. The job pool topic is set with the argument --job-pool-uri.
  • To better work with grafana, you no longer set the service-id. Instead you set the service name which is then used to generate a service id. The service name is set with the --service-name argument.
  • Fixed a bug in the metrics code that prevented more than 10 counters from being pushed to Grafana.
  • Flatbuffer verifiers have been re-enabled for command messages.
  • Kafka timing settings have been changed to reduce the probability of timeouts.
  • Removed a bunch of old (and now unused) Kafka metadata code.
  • Improved log messages slightly.
  • Modified/fixed/cleaned the system test.

Merge request reports