What if we look more to the  direct experience of an interviewer

Interview Question 1:

You've been asked to design a large-scale data pipeline that ingests and processes billions of events per day from various sources like mobile apps, web logs, and IoT devices. These events need to be enriched, validated, and transformed before being loaded into a data warehouse for analytical workloads. Additionally, some derived datasets need to be created in real-time for operational monitoring purposes.

Describe the overall architecture you would propose for this system. Talk through the different components, their responsibilities, and how they would interact with each other. Address topics like:

• Data ingestion and buffering mechanisms

• Event parsing, enrichment, and transformation

• Handling late/out-of-order events

• Real-time and batch processing paths

• Error handling and monitoring

• Scaling and performance considerations

Interview Question 2:

Your company wants to build a metadata management platform that can serve as a centralized repository for all data assets across the organization. This platform should enforce data governance policies, capture data lineage, and provide self-service capabilities for data discovery and access.

Design the architecture for this metadata platform, covering aspects like:

• Metadata ingestion from various data sources (databases, data lakes, BI tools)

• Metadata storage and indexing

• Enforcing data quality rules and integrity constraints

• Capturing data lineage and impact analysis

• Role-based access control and security

• Search and browse functionality for data assets

• API design for integrating with other data tools

• Scaling and availability considerations

For both questions, be prepared to discuss the trade-offs of different design choices, potential bottlenecks and failure modes, and how you would address them. You should also highlight any specific experiences or learnings from your past work that are relevant to the problem at hand.