Estuary Flow

Name: Estuary Flow
Rating: 5.0 (1 reviews)

Your data, where you want it, in milliseconds

5.0•1 review•

427 followers

Your data, where you want it, in milliseconds

5.0•1 review•

427 followers

Visit website

Data analysis tools

•

Databases and backend frameworks

•

Cloud Computing Platforms

Managed change data capture and ETL pipelines with streaming SQL transforms.

Free

Launch tags:Data & Analytics•Data•Database

Launch Team

Unblocked — Giving your agents the context to save time and tokens.

Giving your agents the context to save time and tokens.

Promoted

Estuary Flow

Maker

📌

The Pitch: 🚰 Estuary Flow is a Real-Time Data Platform that enables building no-code reliable pipes that don’t require scheduling, and support batch/streaming and materialized views in milliseconds. 📒 A free account with up to 10gb/mo in data movement can be had here: www.estuary.dev The Details: Estuary Flow is built on top of an open-source streaming framework (Gazette) that combines millisecond-latency pub/sub with native persistence to cloud storage. Basically, it’s a real-time data lake. Beyond being able to sync data continuously between sources/destinations without configuring, say, Kafka, there are a few benefits to a UI built on top of this streaming framework, specifically: 🗄️ Managed CDC. Simple, efficient change data capture from databases with minimal impact and latency. Seamless backfills – even over your very large tables that Debezium tends to choke on – and real-time streaming out of the box. 🧑‍💻 Streaming SQL transformations. We have a quite powerful transformation product that allows for Streaming SQL transforms without a requirement for windowing. Join historical data with real-time without having to think about it. Flow also offers schema validation and first-class support for testing transformations, with continuous integration whenever you make changes. 💽 Collections instead of Buffers. When a data source is captured – like Postgres CDC, or Kinesis, or streaming Salesforce – the data is stored in your cloud storage as regular JSON files. Later, you can materialize all of that juicy history and ongoing updates into a variety of different data systems. Create identical, up-to-date views of your data in multiple places, now or in the future. 📈 Continuous Views instead of Sinks. Materialized views update in-place. Go beyond append-only sinks to build real-time fact tables that update with your captured data – even in systems not designed for it, like PostgreSQL or Google Sheets. Make any database a “real time” database. ✅ Completely Incremental, Exactly-Once. Flow uses a continuous processing model, which propagates transactional data changes through your processing graph. This helps keep costs low while maintaining exact copies across different systems. ⏩ Turnkey batch and streaming connectors. Both real-time data as well as historical data supported through one tool and access to pre-built connectors to ~50 endpoints. For example, you can capture from the batch Stripe API, join it with data from Kafka and push that all to Google Sheets – all without building a custom integration. Or if you want, plug in your own connector through Flow’s open protocol.

Report

3yr ago

@dyaffe Congrats on the launch, Real-time data gathering and Synchronising data between multiple sources are impressive features that could save us a lot fo time

Report

3yr ago

Estuary Flow

Maker

@dineshan_sithamparanathan thanks! Appreciate the support and excited to see your product!

Report

3yr ago

Collato

Congrats guys on the launch. The idea looks good. Just a curious question: what is the main difference between Segment and Estuary Flow?

Report

3yr ago

Estuary Flow

Maker

@huy_doan_quang thanks! One difference is that Segment is a Customer Data Platform, whereas Flow is much more generic. We've tended to focus a bit more on integrations with things like databases, and Segment focuses more on things like capturing data directly from your website. We've actually had users push data from Segment into Flow (using webhooks) in order to materialize their data to an OLAP database like Bigquery or Snowflake. One other difference is that Segment is more focused on point-to-point integrations, while Flow focuses more on "data products". When you "capture" data in Flow, you don't just have the ability to materialize it somewhere else. Flow captures data into "collections", which have all the features of data products: - Discoverable: Users can come to Flow to find the data they need - Governance: Flow's authorization system allows you to safely share data across users, teams, and even organizations - Quality: Flow does schema validation at every step, and built-in transforms let you clean up your data and provide meaningful guarantees about the output data - Observable: Flow automatically instruments every task in a pipeline so you can stay on top of what's happening. (note that we're still working on exposing some of these observability features in the UI)

Report

3yr ago

Estuary Flow

Maker

@huy_doan_quang Just to piggyback on Phil's answer: a fun thing you can do with Flow is capture WebHooks coming from Segment to then use for your own purposes. - Flow them into an analytics warehouse like Snowflake or BigQuery. - Roll them up into a live dashboard kept in a Google Sheets. - Transform and aggregate your Segment events into a customer 360 profile.

Report

3yr ago

Collato

@johnny_graettinger1 wow, it sounds super cool. Thank you for the explanation guys! Keep it up! Also, is Estuary Flow on Twitter? Would love to see and catch up with your product updates in the next few weeks/months.

Report

3yr ago

Estuary Flow

Maker

@johnny_graettinger1 @huy_doan_quang yes we are but we haven't been super active yet. Our handle is @estuarydev!

Report

3yr ago

Wonderful real time data platform tool to build no-code reliable pipelines. Can you please enlighten me about few use cases of Estuary Flow?

Report

3yr ago

Estuary Flow

Maker

@gurudeep_shrotriya Hello Gurudeep, thank you for your kind words! Estuary Flow serves the purpose when you want to move data from APIs, databases, or different technologies, to certain destinations. On the way, you can define real-time transformations. All of this happens in real-time so you get milliseconds latency between changes in your source database and your destination. If you have a certain problem that you are trying to solve, we would be more than happy to discuss it as well. A few use cases that come to mind: 1. Gathering your analytics from different SaaS that you use (e.g. HubSpot, Salesforce, Mailchimp, etc.) and consolidating them in a single destination for analysis (e.g. Snowflake) 2. Synchronising data between a database and an Elasticsearch cluster, to allow for your data to be searched on 3. Synchronising data between SQL and/or NoSQL databases, e.g. if you are migrating from one database to another and want to keep data synchronised between the two, or for any reason you have specific use cases for different databases. e.g. Move data from Firestore to Postgres or MySQL to MongoDB, etc. 4. Sending events to a queue technology (e.g. Google PubSub) when certain changes are made to your data in any source 5. [... The list goes on!]

Report

3yr ago

Estuary Flow

Maker

@gurudeep_shrotriya @mahdi_dibaiee and another very common one is just doing change data capture from databases to warehouses! If you use streaming sql to create a materialized view, it can save a good amount of money when you query that warehouse.

Report

3yr ago

@dyaffe fantastic! thanks .

Report

3yr ago

@mahdi_dibaiee that is really awesome. Thank you very much. I'll really use Estuary Flow.

Report

3yr ago

Twitter Spaces Downloader

Congrats David on creating Estuary Flow! 🎉 It's an awesome real-time data platform with great features. Have you considered adding more capture options? That would be really useful. Thanks for providing a free plan, looking forward to exploring more! 🙏

Report

3yr ago

Estuary Flow

Maker

@alosefer Thank you Yaser! Yes we are continuously adding more capture and materialization options. Is there a specific capture option you are looking for? We are more than happy to chat and find a solution for your needs. Our process allows us to add most new capture connectors pretty quickly, so don't hesitate to message us if something is missing!

Report

3yr ago

Twitter Spaces Downloader

@mahdi_dibaiee Hi Mahdi, thank you for your reply. It would be nice if you could add Twitter and MailerLite, as I believe they would be helpful for us and others 👍

Report

3yr ago

Estuary Flow

Maker

@alosefer We can definitely add new capture souces for Twitter and MailerLite, it would take only a couple of days. Would you like to discuss your use case further with our team so we can get a better idea of your needs and hopefully set something up that suits you?

Report

3yr ago

Collato

Congrats on the launch! This looks fantastic! In your opinion, what types of businesses or industries can benefit most from it and how can it improve their operations?

Report

3yr ago

Estuary Flow

Maker

@irina_haupt I appreciate this question because we see SO MANY opportunities in such a wide variety of businesses to improve their operations and get faster access to higher quality data. One of our goals in building Flow was to make realtime streaming data pipelines so easy that they're accessible to anyone. And anyone can benefit from making their existing data pipelines faster, cheaper, and more robust! That said, here's some more specific examples: - Sales and marketing organizations can use Flow to get realtime insights, where previously they had to wait for the next report to be published - Developers can use Flow to sync data between systems in realtime to populate search indices, migrate to new systems, etc. - Analytics orgs can use Flow collections as data products, for a realtime "data mesh" that makes (sane) self-service analytics a reality Perhaps one of the most important benefits can be realized by any business, though: cost savings! Flow's incremental streaming approach is super efficient, which is why our pricing is so low. It's not a gimmick, and we're not planning on raising the prices. You really can just save a ton of money by switching your data pipelines over to Flow, and we anticipate that being one of the most important benefits to most users.

Report

3yr ago

Estuary Flow

Maker

@irina_haupt Thanks!!! So far we've seen somewhat broad appeal, but there are a few specific use cases that we've seen the most appeal with: 1. Customer 360 -- lots of businesses can benefit from this but essentially it's important to have recent data when engaging with customers. The primary type of business that employs this would be marketing but we see other industries too. 2. ML -- If you're doing ML, having historical and real-time data is very improtant for training purposes. 3. Monitoring & Analytics -- Another broad use case, but monitoring requires access to low latency data. 4. Financial and Insurance -- Both finance and insurance need to replicate data from dbs. Finance cares a lot about doing that in real-time. 5. Fraud -- real-time is required to detect fraud 6. CDC -- A generalized one is just impact on systems when doing change data capture (replicating data from databases). From our experience this really works across a lot of industries.

Report

3yr ago

MegaVote

Congrats David & team on your launch! Estuary Flow seems like an impressive real-time data platform that simplifies streaming data products. With 23y in Tech & currently Our team & I are still handle 1000s of data flows (Streaming & Batch), so I'm excited to see how Estuary Flow can benefit my data and analytics work. Will send to my SRE team to check, good luck!

Report

3yr ago

Estuary Flow

Maker

@maxtpham Thank you Max for your motivating words! We are always happy to learn about new use cases that people have for data pipelines, and we are eager to help you get started at any time! Look forward to see how you and your team get on with the app and if at any point you would like to discuss things (specific connectors / requirements, etc.), reach out to us!

Report

3yr ago

MegaVote

@mahdi_dibaiee sure, thank you, Our team will contact you if we need any help!

Report

3yr ago

Estuary Flow

Maker

@maxtpham please feel free to reach out directly if you have any questions at all!

Report

3yr ago

Hi, congrats I love your product, wanna use it in my system, but have some questions: 1. What databases do you support? I’m using cocroachdb and wanna get data from this db

Report

3yr ago

@alexey_yurkevich 2. How fast synchronisation works? Is it real-time or delayed?

Report

3yr ago

@alexey_yurkevich 3. I have very heavy db with a loooot of new data coming. Could this solution be scaled?

Report

3yr ago

1 2 3

•••

5.0

Based on 1 review

Review Estuary Flow?

Reviews

Most Informative