Skip to content

Filter refactoring: Remove old code and start using the new

Afonso Mukai requested to merge filter_refactor_part_7_remove_replace into v3.0.0

Created by: SkyToGround

Issue

  • Hopefully the last PR for version 3.0 of the file-writer

Changes in this PR

  • Deleted old code.
  • Replaced calls to old code with calls to new code.
  • Turns out that I forgot to add logic for determining when to stop a stream thus I had to add it in this PR.
  • Several of the system tests had bugs in them. Those have now been fixed to some extent.
  • Increased the execution speed of some of the system tests.
  • Modified some system tests to better reflect the current implementation.
  • It turns out that empty topics does not result in an end-of-partition message but rather in a timeout. The fix is to treat timeouts as errors.
  • Added some missing comand line arguments.
  • Removed some old/unused/ridiculous command line arguments.
  • Added and modified some code to more easily deal with durations and time points.
  • The way in which I had implemented the functionality for dealing with messages from the same source going to multiple different types of writer modules was buggy. This has been fixed.
  • Improved the documenation a bit.

Additional notes:

  • As I added some functionality in this PR that I had forgot to implement in earlier PRs, I should really write some unit tests for that. I am thinking that I can do that next week and you can get a start on looking at this PR.
  • The system tests are unnessecarily slow and fragile, we should probably fix that at some point.
  • I see no reason not to treat kafka timeouts as errors but I know we have discussed this before though I do not remember what the conclusion of those discussions were.
  • I still really think that we should replace std::chrono::milliseconds with durations and time points throughout (mostly) the whole application. Making the C++ type system help us to avoid mistakes is a good thing in my opinion.

Merge request reports