A Big Thank You! and a Big Ask
Thank you everyone for the support. I have received nearly 40 signups and 1 paid user which is massive for me as I am still on the early stages of validating the product. So thank you everyone.
Next steps: Even though the signups are coming in I am tracking the usage of the app and I dont see many users running the evaluations and I need help. How should I get you to try and test the product.
My current thoughts are:
- User using their API keys means friction and would go away from the platform as can't test it immediately so perhaps allow free trial with my API key which involves unlimited runs with 3 test cases.
- Create a onboarding guide? Like the ones of enterprise softwares that says "Click here" "Next Steps": What tool can I use for this?
For both the options above will still need to inform them about the updates and hope they will signin again.
- Reach out to all 40 users for a 1-1 15 mins call and show them the product. Assuming 30% respond that 12 calls booked.
Do you have any suggestions? Keen on feedback. This is critical as I need to solve these issues before building next features i.e (Adding more models and multi model runs).
Please help me build a product that solves your problems
Im new to the game and dont have a huge marketing or product strategy team hence each one of you is my strategy team :)
Replies
If you don't see many evaluation runs in your analytics, one thing I'd suggest is following your launch link from Product Hunt in a browser where you're not signed in to replicate the flow new users from PH will have. Confirm your evaluations are successfully running for new users. It might be that people are trying to test the product, but it is failing before their evaluation attempt hits your analytics.
I just tried an evaluation, and I got 30+ console errors (lots of supabase 401 errors plus other errors). Under your "Model Responses" section the models I selected (OpenAI: GPT-4o, Anthropic: Claude 3.5 Sonnet, Meta: Llama 3.3 70B Instruct free) all display error: "Error: OpenRouter API Error: OpenRouter proxy error: Edge Function returned a non-2xx status code"
[I'm using MacOS Safari browser in a private tab with a VPN running while not signed in to promptperf.dev.]
// EDIT: I just realized this post was for your first PromptPerf launch (not the new one). My above test experience is from today (Dec 1st) on your 2.0 launch.
@kilpatrick Thanks for this. I did miss the step. I had initially solved the issue on local deployment but definitely forgot about it when moving to PROD had to run the proxy for guest users.
It should have been fixed now. Appreciate you calling this out.