Continuously and incrementally ingesting data as it arrives in cloud storage has become a common workflow in our customers’ ETL pipelines. However, managing this workflow is rife with challenges, such as scalable and efficient file discovery, schema inference and evolution, and fault tolerance with exactly-once guarantees. Auto Loader is a new Structured Streaming source in Databricks as our all-in-one solution to tackle these challenges.
In this talk, we’ll discuss how Auto Loader:
Can discover files efficiently through file notifications or incremental file listing
Can scale to handling billions of files as metadata and still provide exactly once processing guarantees
Can infer the schema of data and detect schema drift over time
Can evolve the schema of the data being processed
Is used within Databricks to ingest millions of files that are being uploaded every hour efficiently
Connect with us:
Website: [ Ссылка ]
Facebook: [ Ссылка ]
Twitter: [ Ссылка ]
LinkedIn: [ Ссылка ]
Instagram: [ Ссылка ] Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. [ Ссылка ]
