Deep Lake AI Knowledge Agent conducts Deep Research on your data, no matter its modality, location, or size. Deep Lake supports multi-modal retrieval from the ground up. It uses vision language models for data ingestion and retrieval so that you can connect any data (PDFs, images, videos, structured data, etc.) stored anywhere, to AI. Over time, it learns from your queries, tailoring the results to your work! Deep Lake is used by Fortune 500 companies like Bayer, Matterport, and others.
Interactive
Free Options
Launch Team / Built With
TRDATA
Congrats on the launch! 🚀
I like that the interface of your agent is different than ChatGPT. It’s light in all aspects of the word. That said, I mentioned potential improvements in user flows that might make the overall experience even smoother than it is now. https://stripe-rosehip-b9c.notion.site/Deep-Lake-interface-improvement-suggestions-2e06ce487db34d4cb866a67e3b2f8f01
Deep Lake - AI Knowledge Agent
@anton_osipov oh my, if we had a second Pixel Perfection award we'd give it out. Thank you so so much for the thoughtful feedback! Will pass over to the team for the improvement. :)
TRDATA
A-ha-ha, thank you! I'm just glad that my comments are useful
Deep Lake - AI Knowledge Agent
Hi Product Hunt!
I'm Sasun, Activeloop's (YC S18) Director of Engineering. I've previously co-founded Pixomatic, one of the early successful photo-editing apps. Naturally, one of the things that excites me is how to visualize (and query) unstructured data, like images.
Except… back in the day, there was no SQL for images.
Then I met @david_buniatyan, who started Activeloop with that mission - make the way complex data - like images, videos, text, etc. stored in a more organized way - and make it easily connectible to AI (for training, and asking questions!).
This comes with a number of exciting technical challenges.
1. Unstructured data is… well… unstructured. It's hard to search across such data (imagine saying I want all the images that contain bicycles larger than 200x350 pixels, and two people in them).
Retrieval systems until Deep Lake weren't fit for that.
2. Vector Search is inaccurate.
Achieving accuracy in AI-generated insights is challenging, especially in sectors like legal and healthcare where accuracy is paramount. The issue magnifies with scale—for instance, when searching through the world’s entire scientific research corpus.
Most data lives in data lakes (S3, AWS, GCP)
3. Limited Memory
Bolting a vector index onto traditional database architectures does not provide the scalability required by AI workloads. As the scale of your dataset increases, the memory and compute requirements scale linearly. For datasets that grew past 100M, the cost becomes prohibitive to maintain the index in memory.
My team and I focused on building this as Deep Lake's ‘unfair advantage’, since we're geared more towards analytical cases where the users need to ask questions across complex, large data. As a result, we're up to 10x more efficient than in-memory approaches.
4. AI Agents can fail… spectacularly
Not claiming we've totally solved this issue, but if there's even 1% probability of failing or responding inaccurately every time, in a complex, multi-step system there will be this ‘butterfly’ effect where with every additional step, the probability to fail will increase.
So increasing retrieval accuracy is important - in critical verticals (autonomous driving, life sciences, healthcare, finance) it can be either a matter of life and death, or incalculable losses.
More on this in detail (with benchmarks here).
Feel free to ask me any technical questions on Deep Lake's capabilities, I'd be happy to answer.
Thanks for having us.
Deep Lake - AI Knowledge Agent
@khustup thanks for being a part of our journey and shipping an amazing product!
Deep Lake - AI Knowledge Agent
Deep Lake - AI Knowledge Agent
@mikayel_harut I'd say the most challenging part is to index the large scale data on object storage, keeping the balance between latency and scale.
Deep Lake - AI Knowledge Agent
This is exciting! Compelling demo!
I am curious how effective Deep Lake's integrated knowledge retrieval approach is for avoiding hallucinations and finding relevant articles not found by other tools in the same space?
Deep Lake - AI Knowledge Agent
@ngalstyan4 good question!
I wouldn't say it's possible to completely avoid hallucinations. Hallucinations happen for two reasons: wrong context, wrong answer by model, and right context, but still a wrong answer by a model. In the latter case, we can't do much. But we focus on making the former case obsolete!
How we do this:
Query planning and gathering context from various datasets.
Querying flexibility (choose to do hybrid, vector, keyword search, etc.)
Multi-modality (on ingestion, gaining more depth of insight into what data is about - what is contained in figures, for instance), which helps pass more imoprtant context to the model.
We also learn over time what queries you consider correct, which helps further improve search experience and increase retrieval accuracy. No other vendor can handle this, as well as #3 as well as we do!
OCR-free retrieval of documents, images, and videos? This truly feels like the next era of AI-driven data utilization! Huge congratulations on your launch! 🎉
Deep Lake - AI Knowledge Agent
@kay_arkain thank you so much, Kay! You're absolutely right.
Deep Lake - AI Knowledge Agent
Deep Lake - AI Knowledge Agent
@avag_simonyan hi there! Actually yes - we have a bit technical blogpost talking about diagram-related use case. This doesn't use the AI Knowledge Agent but if it's possible with just code, it would be even easier through the AI Knowledge Agent UI. Handwritten comments wouldn't be an issue (no more than for an OCR). Specifically for architecture, we haven't tested it but should be extensible from other blueprints without an issue!
Would love to chat about the use case with you.
Here are few things that I really liked in my first experiments.
1) Being able to use lots of public data sets before porting my own. Gave use good idea what the usability would be like so it was really helpful.
2) Found the onboard for data access quiet helpful, lots of things were present that were really on point, e.g. CORS configs.
3) Really liked step by step presentation of things before final output would be generated. Would want to see more control there.
Example, when doing a search it allows to see "Generating TQL", which is great, now I want to see what would happen if I were to change TQL itself. Could I see side by side data? Could I see performance metrics. Not all queries are equal in backend side so may be I want to speed things up or help things get to better results.
I can see that the platform does have the ingredients so looking to explore further and give more updates here.
This looks like an incredibly useful tool for tackling complex, multi-source research questions. The ability to search across diverse data types and extract well-researched answers is definitely something the community will benefit from. Looking forward to seeing how people adopt it in different use cases.
Congrats on the launch, @david_buniatyan and team!
Deep Lake - AI Knowledge Agent
@samvelyanmany thanks! What would be your preferred use case?