2023 HPCC Systems Community Summit: Parquet Support for ECL

Подробнее о видео

Parquet Support for ECL - Jack Del Vecchio, LexisNexis Risk Solutions

Introducing the Parquet Plugin, an interface between ECL and Apache Arrow that gives the ECL programmer the capability to interact with the parquet file format. This talk will demonstrate how ECL programmers can efficiently read and write parquet files with ease. 

With this interface, ECL programmers can partition datasets, read any partitioned or non-partitioned dataset, and write to a parquet file. In the demo all the functions of the plugin will be shown. One of the key highlights of the plugin is its capability to handle datasets larger than memory. By leveraging streaming techniques, we ensure efficient processing of large-scale datasets without sacrificing performance or data integrity.

Attendees will gain insights into the integration of parquet and how the Apache Arrow library was leveraged to give the ECL programmer efficient access to the parquet file format. A variety of demos to include usage examples will be shown as well as opportunities to ask questions and learn more about the plugin. 

© 2023 LexisNexis Risk Solutions