Estuary Flow seems like a powerful and versatile tool for real-time data management. The fact that it offers a free account with up to 10gb/mo in data movement is a great way to get people interested in trying it out. How does Estuary Flow compare to other real-time data platforms, such as Kafka?
@cata is built on top of an open-source streaming spine (gazette.dev) that, from a distance, has a lot of overlap with Kafka. A killer feature of Gazette, that Flow uses extensively, is that cloud storage is _the_ native persistence layer for all data in the system. When you capture data with Flow, it's quite literally building out a real-time data lake. Later -- potentially much later -- when you use that data, it backfills from your cloud storage and thereafter stays up to date.
Flow builds atop this to take your stated desired outcomes -- I want data from here, transformed in this way, and materialized to there -- and make it happen on your behalf.
tl;dr You get the operational capabilities of Kafka, without having to actually run Kafka.
@cata Kafka is low-level infrastructure which can be tough to manage. Confluent makes it easier, but even with that there are some big advantages with Flow:
1. Data history -- you get the ability to retain all historical data and use it very easily with Flow. Kafka and kinesis can be viewed as more of buffers which retain recent data.
2. Transactional connectors -- where possible, our connectors are fully transactional meaning that they employ exactly once semantics.
3. Integration with batch systems -- you can also pull data from batch systems with no problem.
4. Ease and cost -- you can create a live data flow without being super technical for about 1/10th the cost of something like Kafka
Happy to help if I can provide more info!
Awesome product! 🔥 I can see how it can be really useful for businesses that want to keep their data updated in real-time. How easy is it to set up and integrate with existing databases and APIs? 😎
@vera_ratovska We strive to make each integration as easy as possible. If there's anything that seems unnecessarily difficult with one of your systems, please let us know so we can address it. Most of the integrations we support can be setup just by filling in a form, and we have the relevant documentation for each system shown right alongside the forms. Certain integrations, like doing change data capture from a database, will require a bit of technical setup. But we try to make that as easy and painless as possible, too. Our goal is to set the industry standard in ease of setup and use for data integrations!
@vera_ratovska as Phil said, we try to make this as easy as possible. One of our biggest recent projects is to make it so that users don't even have to read documentation when setting up technical systems.
We actually are making it so that we tell you the exact queries that you need to run to get something working if you haven't read our docs by looking into your DB and figuring out which permissions we don't have.
@dzianis_yatsenka thanks for the support and yes! That is something that's natively supported. We can capture from firebase, optionally automatically schematize the data, potentially transform it and push it where you'd like.
@dzianis_yatsenka yes, it can be used to transfer it to Google Sheets -- we actually have a bunch of different connectors that you can use to transfer it to. Ex. Snowflake, Mongo, etc as well!
Is it a tool for data visualization? I don't know well, but it seems like a lot of time and effort. Congratulations on the launch and keep up the good work!
@frida_lee Hey Frida, thank you for your kind words! Flow is for moving and transforming data. We can capture your data from various sources, allow you to transform them on the fly if you want, and finally materialize them to a destination.
The destination might be a data visualisation technology, but it might also be a warehouse / data lake. If you have such a use case, we suggest you try out the app and if you have questions we are more than happy to answer!
Estuary Flow
DesignRevision
Estuary Flow
Estuary Flow
Twixie
Estuary Flow
Estuary Flow
Govar – English speaking app
Estuary Flow
Govar – English speaking app
Estuary Flow
Govar – English speaking app
SlashPage
Estuary Flow
Estuary Flow
Unispend
Estuary Flow
Estuary Flow
Estuary Flow
Estuary Flow