> For the complete documentation index, see [llms.txt](https://au-dohp.gitbook.io/au-dohp-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://au-dohp.gitbook.io/au-dohp-docs/get-help/libraries-and-components.md).

# Libraries and components

Data Product:

· **Druid:** As the cornerstone analytical database, Druid facilitates real-time data ingestion, flexible data exploration, and quick data aggregation. With its unique columnar storage, Druid ensures lightning-speed query responses

· **HOP**: It is an open-source tools which allows data mapping and transformation

· **Airflow** : It is orchestration dynamo for data transformation, offering tools to programmatically author, schedule, and monitor complex workflows, ensuring that all data processes are executed seamlessly.

· **SuperSet:** Enables creation, exploration, and sharing of interactive, real-time dashboards. Its integration with Druid is seamless, enabling users to visualize data analytics in real-time.

· **MINIO**: Distributed object storage system which leverage S3 API compatibility for widespread accessibility and scalability.

Data Contract:

· **LakeFS**: An open-source solution on top of the object storage layer. It offers a scalable and format-agnostic version control for Data products and employs a Git-like data versioning support.

LakeFS branches support the creation of isolated snapshots of data. Data files are not copied in case of Branch creation and therefore storage space is optimized. We will use one lakeFS repository for all data products of a domain team. All data files holding the data of a data product are stored in a separate directory tree in lakeFS file system.

· **Data Contract CLI :** an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.

· **Marquez** : It is a tool for metadata support logging of Data products. It implements the Open lineage Standard, for metadata tracking. It provides a visual representation of a data lineage graph to further support inspecting the processed data in a data product.

Self-serve platform:

· **Apache Griffin:** Apache Griffin is an open-source Data Quality solution. It offers a unified process to measure the data quality from different perspectives.

Data Quality checks ´can be then orchestrated using Airflow, ingested in Druid and the result can then be visualized in SuperSet.

· **Keycloak :** Manages user access and identities, ensuring secure logins and appropriate access controls over Data Products within SpeedyMesh

· **Apache SkyWalking:** Application performance monitor tool for distributed systems.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://au-dohp.gitbook.io/au-dohp-docs/get-help/libraries-and-components.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.