Based upon the Active Project Status achieved in August 2021, the project team reviewed and updated it architectural direction. The following slides give a sense of progress made to date on the Spark Integration Efforts.
Main conclusions continue to be:
- Code conversion of existing GenevaERS to Java for pre-processing increases product consistency
- Spark adds significant capabilities to the tail-end of the GenevaERS Performance Engine, perhaps replacing the GVBMR88 Summarization Engine
- Approaches for this integration require further development:
- File based approaches may be more simple to code, but can Spark be configured to skip its Map phase?
- Streaming integration may allow for reduced IO, but perhaps increase integration complexity.
The continued open Friday R&D Session (typically noon ET) will progress thinking on these points.
The following is a video extract of the Technical Steering Committee review of the slides which follow,
Here are the slides used in the discussion:
