computer (1943) - a person who carried out calculations.

computer (2023) - an electronic machine that can store, organize and process information.

intelligence (2023) - the ability to learn, understand and think in a logical way about things.

intelligence (2103) - ???

No matter how irreplaceable you feel at work, if there's a way to automate and improve efficiency, it will be done.

Stop with the "AI could never..."

Instead, reshape how you work and your tools. If done right, you'll work less for more.

Large software companies leverage streaming infrastructure like @apachekafka to build new data applications. 

As a result, data-centric software developers are faced with the problem of moving data in and out of Kafka.

Is Kafka Connect the best option?

🧵

Kafka Connect is a JVM-based tool to stream data between Kafka.

Kafka Connect is now within the critical path of our applications and this leads to software engineers facing the problem of writing "connectors". 

But, are Java and the JVM the right way?

The Conduit SDK uses @Golang. Go uses very little resources, compiles to a small deployable binary, and is very performant.

It allows you to painlessly create and implement streaming data connectors and operate them within your infrastructure.

https://github.com/ConduitIO/conduit-connector-sdk

Is Kafka Connect the best option?

The next generation of tools will be code-first.

Why? Solving problems in a UI is not repeatable or scalable. 

Code that solves problems can run on its own, be tested, and scale. 

Code is more accountable.

🧵

A code-first tool is a tool that enables developers to use code to manipulate resources or perform tasks. 

Ex. @PulumiCorp - Infastructure as Code

A tool that isn't code-first would provide the user with a UI to control system resources. 

Ex. Using the AWS UI.

A code-first tool exposes an SDK that enables developers to leverage the power of code.

This takes the power of the software development lifecycle and applies it to the problems at hand.

Ex. my infrastructure can now be managed like my applications.

The beauty of code is that code has no vertical or market. Code is a general solution to all problems. 

Any tasks that you perform on a computer could be code-first.  

My examples used infrastructure, but here is the same idea but with video:

@remotion_dev - Videos as Code

The downside here is that code-first tools only enable those who can write code. 

True.

However, developers write code to solve problems. Code-first tools increase the capacity for developers to create because they expand tasks that can be performed with code.


User Interfaces will be much more capable and hopefully more accountable with this future. 



The next generation of tools will be code-first.

Most software needs only code and a database. But applications at scale are much more complex. 

Engineers need to step back and consider their systems in that generate or store data as its own problem.

Here is the 🗺 of data infrastructure I'm working on. 

These systems are critical to software engineers developing data intensive applications. 

Let’s start with the obvious. 

• Applications - Software applications are programs that are designed to perform specific tasks or functions. They may be web-based, mobile, desktop, or server-side applications.

• Operational Datastores - An operational datastore is a database that stores data used by operational systems. This type of database is typically optimized for fast reads and writes, and supports features such as transactions and rollbacks.

• Caches - A cache is a temporary storage location for data that is accessed frequently. Caches can improve performance by reducing the time it takes to access data from slower storage devices.

• Analytical Datastores - An analytical datastore is a database used for analytical processing, such as data warehousing or business intelligence. These databases are typically optimized for complex queries and reporting.

• Metric Datastores - A metric datastore is a database used to store metrics, which are measurements of system performance or activity. Metrics can be used to track trends over time, identify areas of improvement, and diagnose problems.

• Object Datastores - An object datastore is a database that stores objects instead of traditional relational data. Objects can include things like files, images, videos, and audio clips. These databases are often used to store large amounts of unstructured data.

• Logging Datastores - A logging datastore is a database used to store log files generated by operational systems. Log files contain information about system activity, errors, and other events that can be useful for debugging or troubleshooting issues.”

• Event brokers - An event broker is a software system that manages the flow of events between applications. Event brokers can be used to connect applications that use different event formats or protocols.

• Search Engines - A search engine is a software system that enables users to search for information stored in a database. Search engines typically provide features such as keyword search, Boolean search, and wildcard search.

• SASS Products (External Database) - SASS products are software applications that are hosted by an external provider. These products are typically subscription-based and may include features such as storage, backup, and security.

The map of data infrastructure

Since @conduitio is written in @golang, it ships as a tiny single binary that can move data from system a to system b. 

a binary that can be embedded, ran locally, scaled in the cloud, etc.

this allows us to think differently about designing data integration software.

Conduit is not just a tool that allows you to build one-off data pipelines. 

It allows you to build tools that orchestrate data movement. 

Conduit does ship with a UI. But, it also ships with an HTTP and GRPC server that allows you to manage the pipeline lifecycle from creation to teardown fully.

The UI is for testing. The APIs are for production.

Building with an API gives us a code-first way to think about data movement. 

It gives us the ability to review, commit, deploy this pipeline like the other critical components of our infrastructure. 

With Conduit, you can stop writing one-off scripts to move data.

This is the 🍖  of my latest article. I included an example and an experiment if you are interested:

https://medium.com/meroxa/writing-data-integration-software-with-the-conduit-rest-api-a17c563e854

You can read the unrolled version of this thread here: https://typefully.com/anaptfox/uObsNOP

Writing Data Integration Software with the Conduit REST API

In 100 years, when we look back on the technology with the most impact, my vote is YouTube.

Nothing will beat the value of good ole education.

As a software engineer, I don't think I'll build software again without Change Data Capture. 

As a data engineer, real-time (vs. batch) CDC changes the game.

Change data capture has been around a long time, I was shocked at how much of a swiss army knife it is. 🧵

If you're unfamiliar, CDC helps enable event-driven applications. It allows applications to listen for changes to a database, data warehouse, etc., and act upon those changes.

With tools like @debezium, @supabase, and @meroxadata CDC is growing and becoming accessible.

CDC is used by databases to keep standby instances up-to-date for failover because it’s effective and scalable. When tapping into these events in a wider architecture, your data warehouse (or others) can be as up-to-date as a standby database instance used for disaster recovery. 

The most powerful destination for a CDC event is a cloud function.

You can pretty easily abstract out a process that relies on data changes into a simple microservice that can be built, maintained, improved, and deployed independently from a monolith.

My favorite is @vercel. 

Storing the raw changes of your data is beyond useful, you not only have the current state of the data, but you also have all the previous states. This can support disaster recovery efforts or it also allows you to answer retroactive questions about your data. 

If you're interested, I expand on these a bit more here. I plan to do more in-depth articles on each: https://medium.com/meroxa/stream-your-database-changes-with-change-data-capture-part-two-8fa4daee6afc