Most software needs only code and a database. But applications at scale are much more complex.
Engineers need to step back and consider their systems in that generate or store data as its own problem.
Here is the 🗺 of data infrastructure I'm working on.
These systems are critical to software engineers developing data intensive applications.
Let’s start with the obvious.
• Applications - Software applications are programs that are designed to perform specific tasks or functions. They may be web-based, mobile, desktop, or server-side applications.
• Operational Datastores - An operational datastore is a database that stores data used by operational systems. This type of database is typically optimized for fast reads and writes, and supports features such as transactions and rollbacks.
• Caches - A cache is a temporary storage location for data that is accessed frequently. Caches can improve performance by reducing the time it takes to access data from slower storage devices.
• Analytical Datastores - An analytical datastore is a database used for analytical processing, such as data warehousing or business intelligence. These databases are typically optimized for complex queries and reporting.
• Metric Datastores - A metric datastore is a database used to store metrics, which are measurements of system performance or activity. Metrics can be used to track trends over time, identify areas of improvement, and diagnose problems.
• Object Datastores - An object datastore is a database that stores objects instead of traditional relational data. Objects can include things like files, images, videos, and audio clips. These databases are often used to store large amounts of unstructured data.
• Logging Datastores - A logging datastore is a database used to store log files generated by operational systems. Log files contain information about system activity, errors, and other events that can be useful for debugging or troubleshooting issues.”
• Event brokers - An event broker is a software system that manages the flow of events between applications. Event brokers can be used to connect applications that use different event formats or protocols.
• Search Engines - A search engine is a software system that enables users to search for information stored in a database. Search engines typically provide features such as keyword search, Boolean search, and wildcard search.
• SASS Products (External Database) - SASS products are software applications that are hosted by an external provider. These products are typically subscription-based and may include features such as storage, backup, and security.