> For the complete documentation index, see [llms.txt](https://au-dohp.gitbook.io/au-dohp-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://au-dohp.gitbook.io/au-dohp-docs/whats-new/integration-of-dynamic-hop-pipeline-creation-feature.md). # Integration of Dynamic Hop Pipeline Creation Feature The pipeline is generated by selecting the data source (Data-product Parquet file), applying desired transforms, and specifying a predefined schema. *** **Core Functionality and Architecture** The feature is activated through a user interface that allows users to input the necessary details to generate a Hop pipeline. By selecting data sources and defining the transformation processes, users can visualize and preview the final merge pipeline before execution in the Hop GUI. *** **Key Components** * **Helper Functions for Hop Pipeline Generation:** The system includes new helper functions that handle the core logic for dynamically generating Hop pipelines. These functions use the user input to create an appropriate pipeline that merges data sources with specific transformations and schemas. * **Merge Pipeline Creation Method:** A new method has been created to generate a merge pipeline from the selected data-product Parquet inputs, a list of transformations, and a predefined schema. This method ensures the correct combination of data sources and processes, handling all logic in the background to produce a functional pipeline. * **List of Supported Transforms:** A simplified list of supported transforms has been integrated into the system. This list allows users to choose from a set of predefined transformations that can be applied to the input data, enabling flexibility and ease of use without overwhelming the user with complex options. * **User Interface for Pipeline Merge:** The user interface allows users to specify the desired transforms, input files, and schema directly. This interface facilitates a straightforward process for creating and visualizing the pipeline without needing deep technical knowledge of Hop. * **Hop GUI Preview:** The Hop GUI is used to previsualize the generated merge pipeline. This enables users to see the resulting pipeline before applying it, offering a clear understanding of how the input data will be processed. *** **Technical Workflow** 1. **User Input:** The user selects the data sources (Parquet files), transformations, and schema through the UI. 2. **Pipeline Generation:** Based on the input, the system generates the corresponding Hop pipeline using helper functions. The generated pipeline is then ready for preview. 3. **Pre-visualization:** The user can view the pipeline structure in the Hop GUI, ensuring all transformations and schema configurations are correct. 4. **Execution:** After previewing the pipeline, the user can proceed to execute the merge pipeline, combining the selected data products with the specified transformations. *** **Key Features** * **Dynamic Merge Pipeline Creation:** The user can dynamically create a pipeline that merges data from multiple Parquet files with user-defined transformations and schemas. * **Predefined Transform Options:** A simplified list of transforms allows users to quickly apply common transformations, reducing complexity. * **Hop GUI Integration for Previsualization:** Users can preview the final merge pipeline within the Hop GUI, giving them a clear picture of how their data will be merged and transformed before execution. *** **How to Test** To test the feature, ensure that the resulting Hop pipeline merges two Parquet files as per the user's input. This process involves selecting data sources, transformations, and schemas, and then previewing the pipeline through the Hop GUI to confirm it meets expectations. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://au-dohp.gitbook.io/au-dohp-docs/whats-new/integration-of-dynamic-hop-pipeline-creation-feature.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.