Filament by Galaxy - Open-source data movement engine written in Go
by•
Filament is a data replication engine written in Go. It supports full loads, incremental replication, and change data capture (CDC). As data moves through the engine, Filament verifies Arrow batches at the sink handoff and, where the sink supports it, checks the exact serialized payload. You can run Filament as a standalone binary or embed it in a Go application.
Replies
?makers, if you ranked #1 on @Product Hunt today, would you take on the Subway Challenge and navigate the entire NYC Subway system in the shortest time possible? @mitchbregs @leonkozlowski
@leonkozlowski @fmerian yes, without a doubt. I ran a half marathon once because I lost fantasy football... game for this challenge.
PS - a sneak peak of our new website (not released yet!) for the PH community :)
@leonkozlowski @fmerian I still gotta clean up the copy tho, lol
@mitchbregs @leonkozlowski @fmerian let's go!
Hey, I’m one of the builders and on the founding team at Galaxy. I led a lot of the architecture and design of Filament and recently wrote a blog post diving into how the system comes together to move data efficiently and safely. If you’re interested in some deeper knowledge of whats going on check it out here: https://www.getgalaxy.io/blog/a-look-inside-of-filament
@atterpac legend..... The people should know that the explainer videos were also built in Go!
More to come on that tho 😉
Could not be more pumped for this! LETS GOOO
@chandler_evans * thinking about which meme is ideal in this reply scenario *
Hi all - I'm Leon, one of the co-founders of @Filament by Galaxy. We are super excited about what filament can bring to your data stack, we've seen it reduce latency by up to 15x against some other tools in production.
After working in the data space for nearly the last decade we've seen countless tools (and pricing models... ha) for moving your own enterprise data into warehouses/lakes. We wanted to bring a fresh take on how to do this in the AI era with speed and trust (filament also converted me to a gopher, ✌️ python lol).
Major shoutout to @atterpac on the brunt of the ideation, design, and implementation of filament. If you struggle with cost, scalability, or disaster recovery in your current data replication tool we would love to help you test this out!
Source code: https://github.com/galaxy-io/filament
Docs: https://filament.getgalaxy.io/pages/guides/get-started/introduction
Benchmarks: https://github.com/galaxy-io/benchmarks/tree/main/data-movement
As always we would love feedback, stars, and for you to genuinely battle test filament on your own workloads and we're here to help!
@leonkozlowski doing this with you has already been the journey of a lifetime and i can't imagine doing it with anyone else.. just getting started brother
Hey Product Hunt, it’s amazing to be launching today! I’m Mitch, one of the co-founders of @Filament by Galaxy. Filament is fully open source and lives at github.com/galaxy-io/filament.
For years, data platforms promised less fragmentation and faster answers, and the tooling ended up creating more of both problems. Our view is that data platforms now exist to serve context, and that context should be built like infrastructure that is interoperable, sovereign, and owned by you and your team.
Filament is the first piece. It’s an open source engine that moves data from wherever it lives to wherever it needs to go, fast, correctly and verifiably. It’s built to hold up at enormous scale and in the most sensitive environments. It’s Apache 2.0, it runs anywhere, and it’s yours.
We built it because for years we’ve designed data platforms on homegrown and vendored tooling, and everything we wanted to build started with the same unglamorous step of getting the data in. The tools were slow, expensive, and never felt great to use. This is the tooling we always wished we had, and we want everyone to have it.
If you’d be so kind, star the repo, install it, run it, break it. Any feedback means the absolute world to us, and we’re here all day, so ask us anything!
All the very best,
Mitch
@mitchbregs yesss boi!!
Been lucky to play a small part in this journey with Mitch and Leon. Watching Filament go from repo to launch has been a buzz. Massive day lads, let’s goooo 🚀
Hey everyone! I’m Saloni and I work on Filament’s open source side: building connectors and helping community contributors land their first PRs.
What’s surprised me most is how quickly people can contribute. Most SaaS connectors are just a YAML manifest plus a docs page. I shipped the Asana source this week and community contributors have already picked up Airtable and Jira within days of us opening them up!!
Is there a tool you’d want Filament to connect to or an idea for what we should build next? Drop it in the comments here or open an issue on GitHub!
And if you’d like to build a connector yourself, we’d love to help you get started: https://github.com/galaxy-io/filament
If you try Filament on your own data, we’d also love to hear how it goes!!