![]() ![]() ![]() You're talking five or 10 times more capable by using Iceberg as a table format." "The sheer volume of data you can manage the number of data objects you can manage and the complexity of the partitioning: it's a multiplication factor. The result is a reduction in the cost of moving data, and improved throughput and performance, Royles said. You can also bring whichever tools you choose to bear on that data." You can determine how to manage, secure and own it. It enables us to think in terms of how different clients both within the Cloudera ecosystem, and outside it – the likes of Google or Snowflake – could interact with the same data. "It's a client library: you can integrate it with any number of client applications, and they can become capable of managing Iceberg table format. "It has lots of capability, but it's very simple," he said. Today, Cloudera sees itself as a multi-cloud data lake platform, and in July it announced its adoption of the Iceberg open table format.Ĭhris Royles, Cloudera's Field CTO, told The Register that since it was first developed, Iceberg had seen steady adoption as the contributions grew from a number of different organizations, but vendor interest has begun to ramp up over the last year. He noted Cloudera and Snowflake were now supporting Iceberg while Google has a partnership with Salesforce over the Iceberg table format.Ĭloudera started in 2008 as a data lake company based on Hadoop, which in its early days was run on distributed commodity systems on-premises, with a gradual shift to cloud hosting coming later. He said Iceberg was becoming the "primary format," although Google is committed to supporting Hudi and Delta in the future. Snap is one of our early customers, all their analytics is and they wanted to push us towards Iceberg over other formats." Iceberg, Hudi and Delta Some of our largest customers were basically deciding in the same realm and they wanted to have something that was really open, driven by the community and so on. We had some discussion around whether we are going with Iceberg, Delta or Hudi, and our prioritization was based customer feedback. Speaking to The Register, Sudhir Hasbe, senior director of product management at Google Cloud, said: "If you're doing fine-grained access control, you need to have a real table format, Spark is not enough for that. In October, BigLake, Google Cloud's data lake storage engine, began support for Apache Iceberg, with Databricks format Delta and Hudi streaming set to come soon. Iceberg enables direct data access needed by all of these use cases and, uniquely, does it without compromising the SQL behavior of data warehouses." Instead, many different processes all use the same underlying data and coordinate through the table format along with a very lightweight catalog. "Iceberg was built on the assumption that there is no single query layer. "If you're looking at Iceberg from a data lake background, its features are impressive: queries can time travel, transactions are safe so queries never lie, partitioning (data layout) is automatic and can be updated, schema evolution is reliable – no more zombie data! – and a lot more," Blue explained in a blog.īut it also has implications for data warehouses, he said. Data lakes alone were estimated to be worth $11.7 billion in 2021, forecast to grow to $61.07 billion by 2029. Iceberg sits in the middle of what is a big and growing market. As well as making life tough for query engines, it makes changing schemas and time travel difficult. Iceberg in the data lakeĬloud-based blob storage like AWS S3 does not have a way of showing the relationships between files or between a file and a table. It has also won support from data warehouse and data lake big hitters including Google, Snowflake and Cloudera. The move promises to help organizations bring their analytics engine of choice to their data without going through the expensive and inconvenience of moving it to a new data store. The project was developed at Netflix by Ryan Blue and Dan Weeks, now co-founders of Iceberg company Tabular, and was donated to the Apache Software Foundation as an open source project in November 2018.Īpache Iceberg is an open table format designed for large-scale analytical workloads while supporting query engines including Spark, Trino, Flink, Presto, Hive and Impala. Out of these performance and usability challenges inherent in Apache Hive tables in large and demanding data lake environments, the Netflix data team developed a specification for Iceberg, a table format for slow-moving data or slow-evolving data, as Gooch put it. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |