Dm 1651 filewriter pool
Created by: SkyToGround
As requested, here is the worker pool functionality as a single PR. Scanning through the code, I recognised of a bunch of shortcommings, lets see if you can find them! Still, I have been using this branch in "production-testing" for the past few weeks and things seems to be mostly working fine. Enjoy the review!
Changes
- The command system has been re-done. The code is (currently) the documentation but it is stronly suggested that instead of reading that, you use the (file-writer-control)[https://github.com/ess-dmsc/file-writer-control] Python library to control the file-writer.
- Some old and now out of date documentation on controlling the file-writer has been deleted.
- Command and status topics have now been merged into one command + status topic. The command line argument has thus also been changed to
--command-status-uri
. - There is now a job pool functionality. The job pool topic is set with the argument
--job-pool-uri
. - To better work with grafana, you no longer set the service-id. Instead you set the service name which is then used to generate a service id. The service name is set with the
--service-name
argument. - Fixed a bug in the metrics code that prevented more than 10 counters from being pushed to Grafana.
- Flatbuffer verifiers have been re-enabled for command messages.
- Kafka timing settings have been changed to reduce the probability of timeouts.
- Removed a bunch of old (and now unused) Kafka metadata code.
- Improved log messages slightly.
- Modified/fixed/cleaned the system test.