Data observability that drives action. Detect and resolve data quality incidents faster by viewing data flowing through your systems and acting on them in real-time with Streamdal, the open-source data observability and governance tool.
@quatermain32 Haha both me and Ustin have a history with elixir - we worked at an elixir shop (Community) before Streamdal and *just* talked to some elixir folks about writing an SDK. So, yeah, very likely to happen sooner rather than later :)
Report
@daniel_selans perfect, ping me and I will infiltrate it to some apps where I can play with it.
Hey @ustin, I've just had a look at Streamdal and I must say, I'm intrigued! 😃 The divide between data handling systems and incident resolution is a real challenge, and seeing your product attempt to bridge this gap is so refreshing. I love the concept of "The Data Graph" – dynamic, real-time visualization sounds like a game changer. 🚀 Just a quick question, when will the data quality management features you mentioned be rolled out? I can't wait for it. Great job and best of luck with your beta phase! 👏
@ustin@paul_gorval Hey Paul! Some of the data quality management features are already available in beta within the console (we haven't put them through their paces yet) in the Pipelines section of the UI.
We've outlined more of our features that are coming soon in our roadmap!
https://github.com/orgs/streamda...
Report
Congrats on your launch Y'all.
How much overhead does this add to something like a NodeJS script?
This is all that's needed?
import { OperationType, Streamdal } from "@streamdal/node-sdk/streamdal";
export const example = async () => {
const streamdal = new Streamdal({
streamdalUrl: "localhost:8082",
streamdalToken: "1234",
serviceName: "test-service-name",
pipelineTimeout: "100",
stepTimeout: "10",
});
const result = await streamdal.processPipeline({
audience: {
serviceName: "test-service",
componentName: "kafka",
operationType: OperationType.PRODUCER,
operationName: "kafka-producer",
},
data: new TextEncoder().encode(JSON.stringify({ key: "value" })),
});
};
Hi Ken, I wrote the node sdk! There is some minimal overhead in that example for stringifying and encoding the data that can be optimized.
As far as the pipeline rules are concerned, we implemented the pipeline rules in wasm and the overhead is very low, about 1% or less. I haven't benchmarked the node sdk yet, but we have for the go sdk and you can see it here: https://github.com/streamdal/go-...
Hello guys, congrats on your impressive launch!🎉
I‘m planning to create an entrepreneurial support group, because I've noticed that many “PH support groups” on #LinkedIn aren't actually helping each other👎, but at the same time, early-stage project teams all need:
1⃣️ help with marketing
2⃣️ Product testing & feedback
3⃣️ finding initial users
So, why not #makershelpmakers ?
Plz let me know if you are interested!
Streamdal
Streamdal
Streamdal
Streamdal
Scade.pro
Streamdal
Streamdal
Sortbird
Streamdal