Filter refactoring: Remove old code and start using the new
Created by: SkyToGround
Issue
- Hopefully the last PR for version 3.0 of the file-writer
Changes in this PR
- Deleted old code.
- Replaced calls to old code with calls to new code.
- Turns out that I forgot to add logic for determining when to stop a stream thus I had to add it in this PR.
- Several of the system tests had bugs in them. Those have now been fixed to some extent.
- Increased the execution speed of some of the system tests.
- Modified some system tests to better reflect the current implementation.
- It turns out that empty topics does not result in an end-of-partition message but rather in a timeout. The fix is to treat timeouts as errors.
- Added some missing comand line arguments.
- Removed some old/unused/ridiculous command line arguments.
- Added and modified some code to more easily deal with durations and time points.
- The way in which I had implemented the functionality for dealing with messages from the same source going to multiple different types of writer modules was buggy. This has been fixed.
- Improved the documenation a bit.
Additional notes:
- As I added some functionality in this PR that I had forgot to implement in earlier PRs, I should really write some unit tests for that. I am thinking that I can do that next week and you can get a start on looking at this PR.
- The system tests are unnessecarily slow and fragile, we should probably fix that at some point.
- I see no reason not to treat kafka timeouts as errors but I know we have discussed this before though I do not remember what the conclusion of those discussions were.
- I still really think that we should replace
std::chrono::milliseconds
with durations and time points throughout (mostly) the whole application. Making the C++ type system help us to avoid mistakes is a good thing in my opinion.