This is a paper that I have a strong interest in as I’ve seen how the industry evolved in creating file formats (Avro, Parquet, Arrow) and also invested in companies like Tabular and Eto Labs that are also building new systems that include new data formats like Apache Iceberg and Rikai.

Disaggregated Data Services

Like the Data Lakehouse paper, I covered previously, we’re seeing systems are going on step beyond the separation of storage and compute, but also having storage layer leveraging Cloud storage like S3 that’s shared among multiple compute frameworks.

One of the challenges with this architecture was maintaining data consistency, which is what Apache Iceberg and Databricks’s Delta lake aim to solve.

However, one particular challenge that isn’t solved is performance. The example used in the paper was Redshift vs Redshift Spectrum (loads external data on query time), which performs 8x slower due to the slower I/O to S3 and inefficiency of not being able to load and structure the data in an optimal way as the data is serialized into a standard format (e.g: Parquet, Arrow) without any additional data structures (index, etc) that is typically maintained in memory in the database engine itself.

Self-Organizing Data Containers

The core idea of the paper is to augment the data format that’s being stored in the cloud, into what the author calls self-organizing data containers (SDC). Modern high performance data systems usually able to achieve high performance by storing and building additional metadata such as indexes, determine and change efficient compression based on data, also applying physical data layout changes such as clustering and partitioning based on data stored and also materialized views against the data.

Instead of having a single data systems such as Redshift or Snowflake create and maintain these data in a silo, data containers is meant to bundle these data in the data format so all systems accessing the same data can leverage these performance metadata and optimizations.

SDC includes client libraries that directly access storage or in-memory to read SDC files without any intermediate services. The client libraries is able to publish metadata about the SDC and also make optimizations to the SDC based on its read patterns. These optimizations can include creating materialized views, replicating part of the SDC for frequently accessed data, creating and updating indexes or storage layout, etc.

The challenge is to be able allow clients concurrently access and modify SDC without performing conflicting operations.

A SDC besides storing data blocks like Arrow or Parquet in columnar fashion, also includes index blocks and metadata blocks. All SDC must include one Index block that contains a sparse row index that maps row id ranges to blocks.

However, additional index blocks can be created for more efficient access. The paper uses the example of creating a quad-tree index that allows efficient lookup for finding range of two attributes. Clients can further create more splits of a quad-tree quadrant when it finds frequent access in a region, by creating more data blocks and creating a new index block. By evolving the indexes and the data blocks, the SDC becomes self-evolving over time through the clients.

SDC vs Delta Lake

Comparing SDC and Delta Lake both stored on disk, SDC can perform up to 10x speedup through optimal range partitioning using the mentioned quad-tree index, while Delta Lake only optimizes 1.25x using z-ordering. The biggest reason Delta Lake isn’t able to perform efficiently in this use case is that the z-order isn’t optimized to the workload access that is observed, while SDC can based on the access pattern evolve to have more efficient indexes.

Last Thoughts

The paper has left a lot more research directions and questions to be answered, such as how to coordinate optimizations being done among different clients to perform optimal outcomes with the same goal, or how to create incremental changes to the data layout and what policy to apply this for.

Nonetheless, I do find this research direction really interesting as I think we are going to see more efficient data access and performance optimizations into the data format layers that is designed for particular workload patterns.

This creates a new interesting paradigm where we see the decoupled database systems continue to evolve where query, compute, storage are being unbundled and shared among systems. We’re already seeing projects and companies moving towards this direction and excited to see how this unfolds.

This is an interesting paper as it is a great example of how a specifically designed storage engine for a use case such as monitoring can yield quite a big difference in performance compared to existing time-series databases.

What’s in Observability Data?

Today all production systems store monitoring metrics from services either sent to external products like Datadog or stored in monitoring systems that are usually backed by time-series databases such as Influx or metrics storage systems like Prometheus.

Peeking into what monitoring data looks like, each service or instance (aka source) usually emits a fixed set of labels with a series of values with a timestamp.

The challenge of storing this data is that it can be huge. The paper specifically talked about Slack, which collects metrics from 4 billion unique sources per day at a rate of 12 million samples per second, generating up to 12 TB of compressed data every day.

Also, each source emits data with variable time delay and can start and stop at various times.

Challenge of using existing systems

Current systems that are commonly being used such as time-series databases (Influx, Timescale, Clickhouse, etc) are built for good support for analytical queries but don’t have great ingestion performance.

Monitoring storage systems such as Prometheus do have better ingestion performances. However, it still has scalability problems when hitting a certain scale threshold as it’s designed to ensure multiple writers to the same data source are in order with a mutex per source, which contributes to 25% of the write performance overhead.

More scalable versions of Prometheus such as M3 and Thanos alleviate these problems by running a sharded Prometheus with a coordination layer on top. This helps scale the existing Prometheus design but each shard still performs the same way.

How MACH works

Mach is designed to be able to handle ingesting large amounts of sources and metrics data with a variable rate (multivariate) while having low latency reads in real-time.

The key design to improve ingestion is to have an architecture with minimal locking and global state and to have loosely coupled components with minimum locking and coordination. This allows Mach to scale to a larger scale of writers and readers with better performance.

Time-series databases allow updates and out-of-order writes which requires them to store and organize data with B+ trees or similar, which slows down the ingestion performance. Mach assumes data is mostly in order and no updates are allowed, which allows writers to directly be appended and have optionally an out-of-order in-memory buffer that gets merged later.

Each Mach writer thread holds its own state and writes to a single segment in memory, whereas the global hash table only holds which source write should be routed to which writer thread.

Each writer has three levels of data per writer:

1. Active Segment: It first appends to the active segment in memory up until a certain configured size

2. Active Block: Once the segment is full, it moves to compress the full segment in memory in batch, instead of compressing on each write in TSDBs. This becomes part of an active block in memory

3. Persistent Block: Once the active block is full, the full block is appended to a dedicated file for the writer thread on disk. Once the file size exceeds the limit, the writer acquires a new file to write.

On the read side, Mach serves read requests by creating snapshots based on the source and time of the request, which records the block metadata written in all three levels of writer data without blocking any writes. The requestor can then request to read blocks that are within the time range requested, which is also efficient given most already appended data are immutable and only requires fast locking.

How fast is Mach?

Using data coming from 700 data sources each with 18 metrics and 32 samples each, Mach outperforms the 2nd fastest system in terms of ingestion by 2-4x (depending on the rate of data), which also outperforms InfluxDB by 20x and RocksDB by 60x. Running with 1 million data sources, Mach consistently performs 4x over Prometheus, while InfluxDB and RocksDB both timed out.

On read performance, while reading data that are within the recent 1 hour, Mach and Prometheus both perform 5x better than InfluxDB as they’re both designed to serve recent data faster. However, when reading data within the last 24 hours, InfluxDB is on par with Mach, but cannot scale once concurrent reads start to increase.

Final thoughts

It’s interesting to see the tradeoff when it comes to designing a storage system, between balancing both write/read performance and also the range of queries and the temporal characteristics.

It’s also another case where a share-nothing design leads to much scalable performance on a single node, which is similar to how newer systems like Scyalladb and Redpanda (leveraging Seastar) also help reduce bottlenecks when it comes to performance.

Zoned Namespaces in SSD

Zoned namespaces are a new NVM SSD interface that was introduced around 2020, instead of every application writing blocks interleaved in SSDs, ZNS can divide an SSD into fixed-sized zones, and allow the application to decide which SSD zone to write data from.

Having fixed zones in SSDs, allows each zone to be garbage collected together and helps reduce a lot of the write amplification issues with SSDs.

This also creates more opportunities for data applications to utilize the new commands that are introduced with ZNS, such as Zone Append which directly appends data into each zone without keeping track of which block / tail the zone has written to. Using Zone Append, data applications no longer have to keep track of pointers and perform locking, which also improves performance as SSD can write data on better hardware alignment as well.

(For more details can also refer to this material from Vault 2020)

Group Append

The paper proposed a new command to be introduced into ZNS, which is an even simpler version of Zone Append. Zone Append requires each writer to write out a block size of data, which essentially means the application has to keep a buffer internally and do the bookkeeping necessary to flush. Group Append allows the SSD controller to take any arbitrary size of writes and can buffer data on the device level.

How software / hardware codesign improves data management

The paper went on to discuss how various data management systems (log-based file systems, RocksDB, Databases, and Event logs) can leverage ZNS SSD and new commands like Group Append can change how the software is designed, also further possible research directions for each as well.

I won’t cover every improvement proposal, so I'm just picking a few to highlight.

RocksDB is a key-value store that uses an LSM (Log-structured merge) tree to store data, which internally keeps an append-only key-value pair structure called SSTables (Sorted String Tables).

With normal SSDs, RocksDB will be writing out SSTables to disk which blocks will be interleaved with other IO writes. Since the design of LSM is to store multiple levels of SSTable cache, each time an SSTable with a particular level is merged to write out to the upper level, data needs to be both removed and written to the new level. This causes write-amplification on the SSD that reduces the lifetime of the disk, and also performance wise isn’t the most optimal.

Using ZNS SSD, we can now map the SSTable data blocks that have the same lifetime into each zone. This allows the LSM tree to instruct the SSD to remove the entire zone in one command, which greatly reduces write amplification and improves efficiency.

However, this also brings more research directions and questions with this approach, listing a few here:

• What is the most optimal way to distribute the data blocks with zones given the data blocks will vary in size?

• How do we store the metadata associated with the SSTables, as it could involve a lot of random writes?

Relational Databases such as MySQL currently write all the transactions into a write-ahead log (WAL) which allows the databases to recover their state in restarts. With normal SSDs, it’s up to the DBMS to understand how and when to flush the WAL files among multiple log writer threads into different files concurrently.

With ZNS SSD and Group Append, we can move each file write to each zone, and each writer can send Group Append with data to the SSD device controller without buffering on the DBMS side. This allows a lot more frequent commits and more efficient implementations, trading off more activities and buffering on the controller side. This also allows the DBMS to efficiently remove WAL by erasing zones directly.

Similarly, this brings more research directions. One example is how can we redesign the DBMS implementation even further, assuming all the incoming changes can be stored effectively to WAL with Group Append, DBMS should be able to just track these appends commit IDs with additional metadata and simplify what it needs to record in its own database persistent logs.

Last words

It’s exciting to see the continuous improvements that happen on the hardware side, and how software can be redesigned that brings a lot more efficiency.

Seeing changes like this also is a reminder that improvements to the disk storage are not just about the cost and performance of IO, but also additional software commands that controllers can bring that can perform operations a lot more efficiently.

Covering this paper is quite interesting to me for two reasons:

• I look at quite a few databases or related startups from an investor POV, and quite often you see mmap being used in some part of the systems. It’s usually part of the discussion I have with founders, especially that who haven’t built datastores in production at scale, who usually underestimate by a few orders of magnitude how difficult it is to implement a new one.

• One of my typical system design interview questions, when I was hiring engineers, was also discussing how to design a key-value store with specific constraints, and it’s quite often you see mmap being part of the solution as well.

Quoting directly from the paper, “mmap and DBMSs are like coffee and spicy food: an unfortunate combination that becomes obvious after the fact.”

What is MMAP?

For those that are unfamiliar with mmap, it’s a system call in Linux systems that allows you to map file content on disk into memory without having to spend I/O up front to load them. This allows programmers to very simply write data in memory, and let OS handles the I/O as needed in the background. There is also some potential performance benefits since having the OS accessing content can leverage the OS page cache directly without a user-space copy.

Many databases has started their v1 implementation using mmap for it’s main storage implementation, like Influx, Singlestore, QuestDB, etc, while a lot of them has moved away after hitting various problems.

(Source: CIDR 2022 video)

What’s wrong with mmap?

If you choose to use mmap for your database, you will hit the following four problems:

• Transactional safety

Because the OS can choose to flush data from memory mapped from mmap to disk any time based on memory pressure, means the database could be writing dirty or in-progress data to disk before it’s fully committed.

Developers hitting this problem with mmap has to implement a WAL (write ahead log) with multiple copies of data either in user space or another mmaped file, which causes duplicated data with a lot more complex logic.

• I/O stalls

Another related problem from the OS evicting data from memory, is that the database no longer knows which part of the memory mapped file is actually in memory. This causes unpredictable stalls when reading from a mmaped memory, and none of the solutions on top (mlock, madvise) is an easy fix.

• Error handling

Databases typically do checksum matching to ensure data integrity, and this becomes hard with a mmaped file as the OS might have evict and sync to disk at any point. This essentially means the database need to validate the checksum on each access.

It’s also harder to do error handling with mmaped memory calls as instead of having generic I/O errors when reading from disk, the OS can produce signals that is harder to manage.

• Performance issues

The most significant problem is actually performance. Even though calling a mmaped file can avoid system calls like open/close and use the page cache, typically OS like Linux isn’t designed with multiple threads/cores attempting to access the same mmap region which causes issues like TLB shootdowns. In evaluation we can see that performance gets worse (2-20x) especially at the point when memory is full.

Closing thoughts

The following is the advice the paper parted with →

Which in my simpler laymen terms, use mmap if you want to hack a v0/v1 product or a small in-memory db, but otherwise stay away from it.

This is also a good illustration of why building a production ready database is hard, there are lots of details in hand to make a database not lose data but also have the performance or features to be able to stand out.

Hi all!

Just recorded our 2nd OSS System Breakdown, where I interviewed Simon Shillaker who is a researcher at Imperial College of London working in the Large Scale Data Systems group.

We talked about his latest publication and project Faasm, that is a new Wasm runtime that is designed for more flexibility in isolation and resource sharing, so it can provide much higher performance for stateful applications.

There is a whole realm of research that is happening on Wasm / Serverless runtime and I learned a lot through this interview about some of the interesting exploration and results Simon was able to get.

Please check it out and will appreciate sharing with others interested in serverless and wasm runtimes as well!

Tim

Today as a developer, developing applications to run in the cloud in a highly available and performant manner isn’t straightforward at all. There are many factors that can

cause downtimes (VM crashing or slowing down, underlying infra degrades, outages, etc). This has led to where we are today which is what we all know as the “cloud-native” application way. Even though we see the rise of Docker, Kubernetes making some level complexity easier, the whole toolchain is really hard to learn and a normal application developer needs to know a lot of knowledge to be able to leverage all of these frameworks well.

What if, there can be a new programming model and language, that can understand the intent of the application, and create a compiler that can generate the full application internals that understands how to be “cloud-native”? We’re seeing commercial languages that have been going down this path like Dark or Unison.

What the “New Directions in Cloud Programming” paper is attempting is not to specify a complete solution in detail, but to propose a high-level programming stack and further work needed to complete the puzzle.

Moving towards a declarative model with a gradual approach

Today’s programming languages are most imperative, which leads to a lot more control to the developers but also means each developer needs to handle a lot of the details of the service themselves.

Just like how we moved database querying to a declarative language (SQL), we should also move as much as possible to a declarative fashion when it comes to programming the cloud.

The paper described a new compiler/language stack called the Hydro, that is designed to bring on a staged approach of generating Hydroflow programs that is ultimately able to orchestrate various cloud resources to deploy the application.

At a high level, Hydro first accepts various kinds of existing DSLs such as Spark, Actor frameworks like Orleans, or Futures like Ray, which are existing programming frameworks and languages. This creates a Hydralogic IR output that is human readable like the following:

This allows developers to fine-tune or configure different aspects of a cloud application, which we will touch on the various areas of configurations that Hydrologic supports in the next section.

After the IR program is generated and configured, then it gets compiled into actual implementation.

PACT: Program semantics, Availability, Consistency, Targets For Optimization

Diving a bit deeper into the design spaces, there are four facets that it allows developers to configure in isolation. This is a pretty interesting paradigm as typically in a program a developer needs to consider multiple facets when writing each line of code.

In the Hydrologic model, it breaks down the declarative sections into these different facets so developers can declare what is level of optimization or configuration it needs for its application. Each facet has its own syntax and code generator, that performs a search-based approach when generating optimized code and not a typical rule-based approach.

The first facet which is the Program Semantics is basically moving the actual business logic of your application into declarative syntax, so the compiler can automatically optimize a lot of distributed / parallel aspects. However, most programs aren’t implemented in a declarative way, and learning a declarative distributed programming language (e.g: CALM) isn’t as simple for developers. The paper proposes to use Verified Lifting, that in summary is a way to translate a sequential language (Java, etc) or framework (Spark, etc) into a target language (which is Hydrologic in this case), by searching over possible candidate code instead of the traditional rule-based approach.

The second facet is Availability, which allows each API to describe the availability factor, such as “This API should tolerate 1 availability zone being unavailable”. One example of how the compiler uses this is it can translate this into deploying the backend into two availability zones and create a load-balancer in front that can automatically failover. It also can replicate the backend state automatically to be stored across two AZs as well.

The third facet is Consistency, which touches the consistency level (ACID isolation, invariances on a property, etc) of your application state that’s configured also on a per API basis. By lifting the consistency specification on the API level, allows code generator to be able to holistically optimize both client and server to optimally avoid coordination and improve performance. This also allows code generators to explore the various possible implementation to enforce consistency based on the data analysis, for example, exploring either leveraging CRDTs or choosing a more consensus-based approach.

The last facet is Target For Optimization, which simply states what are the cost and performance targets that the application should optimize for. This allows the code generator to consider how to target instance types choices, number of machines, and other factors that are able to both optimize for cost and performance.

The output of the code generators from these facets is another IR that can be generate a full implementation / deployments that leverages various existing Cloud resources or managed services to fulfill this goal.

Conclusion

While this paper has proposed a lot more future research areas and questions, it is really interesting to see what a possible high-level path can look like when we can move the developer’s concern into a much more high-level language.

I think moving away from all the proposed research areas, it’s also interesting to ask what is a declarative abstraction that developers will like to leverage in today’s development and infra stack. The rise of IaaC (Terraform, Pulumi, etc) has brought in a higher declarative language to be able to repeatable deploy and maintain infrastructure state.

I believe there will also be innovation that happens on the application level that also allows you to declare application logic and entities in a more declarative way. Chiselstrike (Essence VC portfolio) is one step in that direction, that allows front-end developers to declare the entities / schema and logic around the entities in a typescript library, which in turn gets compiled into a backend API and more.

It’s exciting to see how research and also today’s developer stack will continue to evolve, and also how the move to serverless will continue to happen as well.

Running a high availability online service is still challenging today, as there are many factors that can cause a service to fail, especially when trying to keep your code shipping velocity at a decent pace (aka “Move fast and break things”). And it’s extremely painful to debug and find root causes in online services.

Most infra teams today rely on monitoring metrics and logs, and when there isn’t enough information then making code changes to add logs or instrumentation and redeploy the online services to get more clues.

The problem with this approach is that redeploying and restarting services is usually not a straightforward process, depending on how much infrastructure is set up to do blue/green deployments, traffic steering, etc. Therefore, it’s usually very hard to make changes to acquire more data to debug in a production environment.

Another problem is when a service is restarted, all the in-memory state is lost which makes warming up the service and debugging extra hard especially when the issue only occurs in particular load conditions.

Pylive is looking to change this by allowing engineers to make code changes on the fly without the need to restart a service, and we will go over some of the important elements of how this is done.

What is Pylive?

Pylive is a library that developers can import into their code and runs a separate server that allows requests to be made to the current running python program.

Developers can then add more instrumentation or profiling to existing code by adding callbacks wrapped around functions or patching whole python functions without restarting the service.

Challenges of updating code live

There are several challenges when it comes to implementing Pylive:

1. How do we modify the running code?

2. When can we modify the code?

3. How do we handle multi-threaded / processes updates?

To address the first problem, Pylive leverages the dynamic typing and object model in Python, since each variable type in Python can be changed dynamically, changing an existing argument or variable’s type is just a local change and doesn’t require program-wide modification. Comparing this to statically typed languages like Java or C++, modifying an existing function type often means modifying the underlying VM or binary which is very tricky. Therefore, leveraging the ability to replace code (obj.__code__) and functions dynamically without any underlying changes is a very convenient way of fully leveraging the power of dynamic languages like Python.

The more tricky problem is when it is safe to modify code, as changing code in the middle of execution can be really tricky. Using the following as an example, modifying code in a critical section in between locks can lead to undefined outcomes.

There is no guaranteed automated way to detect what section is safe to modify, so this relies on developers specifically stating which section is safe to change and when which is also referred to as a safe point.

One safepoint that Pylive supports is to replace a function when there is no active caller, which Pylive supports by scanning all thread stacks for any active reference of that section.

Another safepoint is to call a user-provided function to check if it’s safe to modify the code, which in the previous example allows the developer to write code to check if the previous lock is still being held.

The last problem to handle is multi-threaded and processes update. Multi-threading is easier to support in Python given there is a single global lock (GIL), therefore if the Pylive thread is able to hold on to the GIL then it is safe to evaluate safepoints and modify. In a multi-process situation, each subprocess is spawned with a Pylive listener that is able to receive requests, and a single controller dispatches code change requests to all subprocesses.

Why this is useful & Implications

The most obvious value this brings is the ability to add logging and instrumentation, or even add patches to live running production code without waiting for this loop:

code → build → verify → deploy → wait until conditions hit

Although in practice, in larger organizations allowing developers to change production on the fly without any verification is a dangerous step as it can make existing problems even worse.

However, I do think that there is a new paradigm of development that is changing how developers develop, build, test and deploy and debug code.

The current state of the art of ensuring production systems’ quality is still mostly about storing and reacting to information from monitoring systems. This includes watching what metrics and logs have been produced (e.g: Datadog), piecing together how services/requests work (e.g: Lightstep), managing and triaging incidents together (e.g: Jelli), etc.

One new wave of startups is moving towards ensuring software quality by shifting left quality checking, either by faster integration tests (e.g: Signadot), replaying API traffic (e.g: Speedscale), or automated quality, and security scanning (e.g: Levo.ai, Optic).

Now there are also a new set of companies tackling the idea of capturing more detailed information at a code level (e.g: Polar Signals) and also replay code executions (e.g: Replay.io) also debug in production (e.g: Rookout, Thundra).

Pylive or similar research is helping to move to build this new development and debug experiences in a more feasible way, as tools get deeper into a language level, it provides a richer set of interactions that developers can be able to interact with services.

Getting into the language object model not just allows you to change code on the fly, but also captures a lot richer information that helps potentially replay production incidents on your laptops, or capture live state capturing in prod from your IDE, to collaboratively modify code in production with a sign off process, etc.

With more powerful tools certainly comes with greater responsibility, but I believe the flexibility and agility that teams can unlock will continue to push the tools towards this direction overall.

Hope you all have a great start of the year :)

This year I’m looking to experiment with different kinds of content, and the first content series I’m trying out is called the OSS System Breakdown.

As the name suggests, we’re looking to breakdown interesting open source projects on the technical design and implementation side, walk through some difficult problems and design choices that people need to navigate in these projects or features.

Our very first guest is Paul Khuong, who is an engineer at Backtrace Labs and released an open source project called Verneuil, that is a S3-backed async data replication layer for SQLite. (Demo here)

There are interesting tidbits about how to extend the virtual filesystem layer in SQLite, how to design for backoff in replication, and tradeoffs of cost and performance.

Would appreciate any feedback if this is useful content, and thanks again for Paul willing to do this!

Tim

I was intrigued to check out this paper and decided to cover the major points of this paper with my own color.

Driving towards a commodity service

We have seen technologies that started with limited and proprietary access slowly become a commodity. The technologies or communication products we use today such as the phone, internet, PC hardware and more have once was only manufactured by a very few vendors with proprietary interfaces and no open standard.

We have now evolved into having all of these becoming a commodity, where we can use any phone from any service, access a website from any part of the world, because of open standards and shared agreements between vendors and government to be able to enforce a level of uniformity.

As all enterprises and software are all moving to the Cloud, the Cloud infrastructure layer becomes the defacto platform where all applications and developers run on. Some companies are spending 80% of their revenue on the Cloud alone, which for many the Cloud is as electricity.

Now looking at Cloud providers, each cloud IaaS provider even though provides a large number of similar services, has opted to implement completely different API interfaces, identity systems, pricing models, and data patterns which leads to porting from one cloud to another very difficult.

This means each developer needs to carefully choose the specific Cloud provider it wants to deploy on, as it gets increasingly difficult to port to another Cloud when desired.

Why should we abstract away from a single Cloud?

The paper argues that this inherent difficulty to move simply from one Cloud to another is inhibiting the ability for users to choose the best services and prices for their specific needs.

Taking training an ML model on Imagenet, the price and performance tradeoff differs depending on the type of instance you run in different clouds on a factor of sometimes 2-10x.

(credit from MosaicML)

Some large companies are also setting up multi-cloud deployments to also reduce catastrophic failure from a single provider (though not everyone thinks this is worthwhile).

Another interesting argument in the paper is that having each “monolith cloud” being not interoperable with each other stiffens innovation outside of those clouds. For example, for hardware accelerators (GPU, FPGA, AI accelerators) to be able to offer their devices as a cloud service, would either need to partner and convince a large cloud provider to offer them, or create their own managed cloud solution.

If there is a way to create an open standard that provides an abstraction layer for Cloud computing, then it could create a “Big Clouds” vs “The Rest”, where the Big Clouds given their size continue to offer their proprietary interfaces and all other long-tail Cloud players can become the open interconnected cloud and offer lower margins given its ability to interoperable with each other.

Moving from the Clouds, into the Sky

To move away from applications written and deployed into a single cloud, can we come up with a “Sky Computing” abstraction that allows apps to move from any cloud to another seamlessly? Or in classic Marc Benioff marketing, bringing on “No Cloud” that your software never feels like it’s locked into a Cloud…

Similar to how the internet is a fully interoperable layer between several networks, the paper proposes two major components to enable “Sky Computing” to have applications fully interoperable between Clouds (write-once, run-anywhere).

The components are a Compatibility Layer and an Intercloud Layer.

Cloud Compatibility Layer

The Cloud Compatibility Layer is an abstraction layer that abstracts away the interfaces and APIs the cloud provider provides, so applications can be built in a way that’s not directly dependent on the Cloud APIs.

Most open-source data and infrastructure and libraries projects today are already abstracted away from the Cloud specifics, so users can run these projects on any of the Cloud providers. However, each of these bespoke data applications each undergone its own significant effort to make its library portable, and there isn’t a standard stack underneath given the huge amount of possible services and APIs to integrate with. Cloud storage is a more common library that OSS libraries integrate, that understands how to read/write data from each Cloud provided blob storage.

There are certainly vendors looking to abstract away Cloud differences and expose general SDKs or APIs. Backend frameworks like Encore, Appwrite allow users to write their backend that’s Cloud agnostic when they deploy. Upbound abstracts the multi-Cloud operation and development into a single unified interface. Localstack abstracts the Cloud integrations from a developer perspective and allows users to test Cloud integrations from their laptops.

All of these are different takes on moving the Cloud specifics away from the user so they can focus on their applications instead. However, all of these are early efforts and not a well-recognized standard, where there are still many missing pieces to make a full compatibility layer work.

Intercloud Layer

Once we have a compatibility layer that can allow the application to move to any Cloud, the user still need to decide which Cloud to run on themselves. The Intercloud Layer automatically unlocks this choice, by transparently moving the applications to the best provider to run a particular workload, just like how internet users aren’t tracking how the packets are routed among network zones through BGP.

The Intercloud layer should allow the user to specify how they should run the workload in terms of performance/cost/availability, but not the specific implementation details. For example, “this is a Tensorflow job, it involves data that cannot leave Germany, and must be finished within the next two hours for under a certain cost.”

The intercloud layer can be composed of three services: 1) Service Naming Scheme 2) Directory Service 3) Accounting and Charging.

The Service Naming Scheme is a similar concept to DNS, but each identity is a particular Cloud Service that can be invoked, with metadata about pricing, availability, provider name, etc.

The Directory Service is an aggregated list of Services provided by each Cloud provider that’s dynamically updated. A new application request specifying its performance, price, and availability requirements will be submitted to the Directory Service, where the Service will automatically resolve and return the best option to launch and run.

The Accounting & Charging layer aggregates the billing from each Cloud provider and understands how to collect and distribute payment for each service used.

Although there is no technical limitation around building this Intercloud abstraction layer, this hasn’t been any meaningful motivation to drive the major Cloud providers to come up with an intercloud standard, given the strong drive to differentiate themselves.

More Speculation about the Sky

Now we understand what Sky computing is, which essentially is the ability for a developer to deploy an app or workload into the “Sky”, where it decides which cloud(s) to run this on and the services and configurations it should consume.

There are a few questions that are worth asking about:

• Will the Sky exists in the next few years?

It’s very hard to believe that we will have a full abstraction layer like this available in the next few years, as there are so much coverage to cover and ongoing changes on each provider. For the Sky to exist, we need everyone to be able to agree and adopt a set of standards across many layers and services, which the major cloud providers at this point have no incentive to.

IMO the closest standard we have for the Sky today is Kubernetes, which most companies today when building a multi-cloud product or service builds on, as it’s the fastest-growing infrastructure piece that all cloud providers have their own distribution. In theory, a Kubernetes pod that runs on AWS could just run on Google without any changes. Though in reality, there are many leaky abstractions that prevent Applications to be truly portable, from different kinds of authentication, Cloud config plugins, storage configurations, and more. Therefore, there is still a significant amount of work on top of Kubernetes to allow complex applications to be ported from one cloud to another.

• What are further implications once the Sky exists?

The paper mentioned that one future effect if the Sky computing layers exist, is that smaller providers and vendors will choose to embrace this commodity layer to compete as a whole with the larger “stand-alone” providers. These large providers will be priced higher and innovate across the board to retain an advantage, while the Sky computing providers will offer a lower margin but differentiated and focused offerings in the catalog of services.

Final thoughts

The interesting dynamics we see today is that most infra companies are offering multi-cloud solutions, but none of the solutions are truly embraced as the standard.

What will be the new standards in the infra stack that can continue to bring more consolidation? Docker and Kubernetes certainly brought the standard up the stack from VMs and machines, and Wasm has an opportunity to bring a new standard coming from the language and beyond.

There is an opportunity to continue to explore this new design space as we see more paradigms appear like Lakehouse that we covered, and see how this space continue to evolve.

I’ve been wanting to continue to write about papers that I think is interesting and relevant to infrastructure in general, and I want to start off with this paper from Databricks that has been creating a new buzz / marketing around how data management of the future looks like. It’s even more fun given there is a war between Snowflake and Databricks fighting back and forth on who’s faster and cheaper, looking to reiterate the Lakehouse shift.

In this post I’ll walk through some major points about the paper and talk about some my own observations about this space as well.

The world of lakes + warehouses

Let’s talk about what’s being described in the paper as three generations of data warehouses.

The first generation is what we knew as the run your own data warehouse software era, which means enterprises buying appliances that coupled storage and compute and leads to hard to manage cost and inability to handle various data formats.

The second generation is the Hadoop era, which we all knew inspired by the Google Bigtable and led to the great Hortonworks vs Cloudera times, which essentially leads to what we all know as the “Data Lake”, that compute and storage is decoupled and data is mostly stored in HDFS (and now cloud storage), and ETL pipelines will be writing data from storage into specific BI solutions and warehouses to serve specific needs. This also leads to the spawn (and big debate) around Lambda architecture, and the rise of Kafka.

So, what’s wrong with the current generation then?

Here’s what’s the main problems the paper states about the current architecture:

Reliability → The ETL layer that needs to run to keep the Data Lake in sync with the warehouse is introducing more potential quality issues and also engineering cost to maintain.

Data Staleness → Another effect of requiring ETL is essential your data is out of sync of what’s the source of truth that’s stored in your Data lake.

Limited Support for advanced analytics → With the rise of ML, data science teams increasingly want to run arbitrary frameworks on top of their data to answer more complex questions or power user facing features. However, data in data warehouse after being ETL is usually locked in a proprietary format and also data scientists don’t have direct access to the source, which requires another ETL from the warehouse into files that can be loaded into frameworks like Pytorch.

Total cost of ownership → Besides paying for more engineering costs and ETL operation cost, the amount of storage requires given the duplication data also increases.

In summary, having to copy data with a ETL system creates operational burden with additional data staleness and cost challenges.

Data lake + warehouse = Lakehouse

So is there a way to avoid having a extra ETL to serve all the warehouse + ML + reporting need?

This is where the Lakehouse architecture comes in, which in the paper is defined as providing a data management system directly on top of low cost and directly-accessible storage, that combines the benefits of a Data lake and a Data Warehouse.

The first problem to solve without having a “closed-world” data warehouse, is to have the ability to have various warehouse and ingestion systems able to write to Data Lake safely. To solve this problem, the first layer at the bottom of the stack is to introduce a shared Metadata layer that sits below all the writers, which provides additional metadata on next to the raw data stored in the Data Lake. This additional metadata file helps provide a consistent state that allows arbitrary and multiple data frameworks to concurrently write with strong transaction guarantees (aka ACID), also governance and auditing, etc. There are several implementations of this layer that designed a new metadata file format, such as Apache Iceberg (commercially Tabular) which was born out of Netflix, Delta Lake built by Databricks

The second problem to solve is to solve the SQL performance problem. Data warehouse systems today like Redshift, Snowflake and others requires ETL to ingest data because it transforms the data into a internal format that optimizes for its own engine execution for faster and better performance, and also builds up internal indexes for faster random access and filtering. Now without an ETL layer, we essentially need to decouple this process and have a open standard format and index that also provides similar optimizations that is format independent and other query engines can understand and leverage.

Here’s the list of optimizations mentioned in the paper:

Caching:

One optimization that data warehouses typically do serving queries from cloud object storage is to cache data in local SSD or memory. Creating a caching layer with data format that can be natively understood by open ecosystem warehouses can potential improve the performance of the processing nodes, and also potentially benefit multiple frameworks. The paper specifically listed Delta Engine’s example of how it can store its cached data with partially decompressed parquet data, so it can have a faster loading time without extra work.

There is an opportunity here to also see how a general in-memory format like Apache Arrow can become a universal caching data format for multiple query engines. Similar projects has exist in the past such as Alluxio which focuses more on supporting heterogeneous storage systems but also provides a caching layer to speed up performance.

Auxiliary Data:

Auxiliary data such as statistics (e.g: column1 has min value X and max value Y) stored in the open data formats like Parquet, or metadata files like Iceberg or Delta Lake Transaction log can be leveraged to help query optimizers know how to skip over data that isn’t necessary to load and process.

Additionally, indexes can be created that’s stored in object storage to facilitate more efficient data skipping such as Bloom filter in Delta engine.

Data layout:

Another optimization is to decide how to group rows that’s stored in object storage, since each parquet file can contain multiple rows. To improve performance, the query engine ideally wants to skip as much files without needing to read them from storage. Delta engine implemented using z-order on specific columns to help provide less value overlap and more evenly distribute the grouping, so the query engine can more effectively skip files. More future work can be determining compression strategies based on data stored / accessed at runtime, etc.

The last problem is to provide efficient access to other analytic frameworks like ML.

Frameworks like scikit-learn or tensorflow doesn’t run SQL, yet can potentially access large amounts of data, especially using scalable data science frameworks like Dask.

The approach that the paper specifies is to expose a Dataframe API / SDK that translates Dataframe operations into Spark SQL executions so it leverages all the optimizations mentioned before like caching and data skipping. Some examples are Koalas from Databricks or Modin or Dask are all wrappers around existing Dataframe APIs but implements a different backend.

The biggest downsides of this approach is 1) ML users need to adopt the library 2) Only supports dataframes, which limits the support to projects like Pytorch or Tensorflow 3) Each implementation around the Dataframe API is engine and framework specific.

There is a lot more room to grow and explore in this side, as ML / AI isn’t all bottlenecked from data loading and execution, but more data transfer and general “AI tax” work. Also optimizations like pushing ML logic into SQL query optimization (aka factorized ML) are still continuing to eliminate redundant work on the ML side.

Closing thoughts

As mentioned in the paper, the Lakehouse architecture or direction will have different implementations and still many open questions and opportunities to continue to innovate.

It’s also an open question how would enterprise adopt a Lakehouse architecture which uses framework beyond one single vendor, as a lot of the trend today is moving towards a open data ecosystem, which starts from the file formats but also moving upwards into the compute stack and frameworks.

There are also open source projects like Apache Arrow that isn’t mentioned in the paper, that could also provide another level of standardization for cross tools optimization and access.

We’re also seeing standards being created from the dataframe level, SQL execution that further facilitates some newer cross stack optimizations opportunities.

It’s an interesting time in the data infrastructure space to watch how the ecosystem evolves.