Pheebs
Measure how engineers and teams actually work with AI
Start new thread

Pheebs - Measure how engineers and teams actually work with AI

by
•
Measure how you and your team actually work with AI coding agents. Pheebs is an open-source AI telemetry tool created by Eversynced. It sits quietly inside Claude Code, Cursor, and Codex via hooks, capturing lightweight interaction signals: the shape of the session, not its contents.

Add a comment

Replies

Best
Maker
📌
I've spoken to several engineering leaders who want to better understand their team's AI usage, but don't know where to start. I've also spoken to several engineers who are stuck and don't know how to level up. There's a lot of FOMO building up and being vulnerable about your AI skills is not easy. Adoption metrics like DAU/WAU and token counts tell you who has access and who uses AI tools. While an ok proxy for AI proficiency and productivity, they say very little about what happens inside a session and how engineers and teams actually use AI. Which models get used and were they the right fit for the work? Are we spending consciously? Do tests run before a commit? Do outputs get questioned or accepted as is? Do outputs need a lot of repair or do they just get refined? Are repos missing out on scaffolding opportunities (skills, context files)? How's my team doing in AI competencies like context management, evals, and orchestration? Pheebs captures that and more. It records the shape of sessions and gives engineers a private view of their own work, separate from the team reports. To make sense of the data, Pheebs proposes an AI Proficiency Model, which looks at two things: 1- Repertoire: which harness capabilities show up in an engineer's work across six competencies: models, artifacts, MCP, evals, context management, and orchestration. 2- Judgement (inspired by the Discernment competency from Anthropic's AI Fluency Index): what happens to AI output before a PR is created and merged. Does it get verified, challenged, refined? And was the model used appropriate for the task? Most teams run the largest model for everything. Pheebs shows how much they could save by running smaller models where they fit. Pheebs is open source so give it a try and send us some feedback!

Hi Pheebs team! I really liked the idea of seeing how often outputs get modified and whether those lead to cost-savings. And also running tests before commits. A few comments / suggestions. 1) Are those metrics shown in the images you have for the Overview? And also, for a given company, is there a way to generate a baseline for a developer's activity that the other developers could follow? for example, maybe you don't need the AI to run and analyze tests after every commit, and only do it before pushing the code? (just thinking out loud for a counter example). Or maybe for a given project, it is better NOT to push back on the code generated by the agent?

2) What do you think of a summary page that shows the recommended steps? You might already have it. I'm only going by the images in the overview.

Maker

 Hi Sara, thank you so much for your comment!

1) Yes, those are the metrics Pheebs provides in the Insights feature. There's currently no way for a company or manager to set a baseline - that could be an interesting feature! What we currently do is show team median as a way to establish a baseline. In the future, we'll also show industry benchmarks based on real data. Specifically on pushback, the goal is not to encourage teams to pushback on all outputs, but to encourage a healthy rate that results in a high quality AI practice. Same with other rates, we'll want to establish a ratio that's healthy and results in more quality.

2) We actually offer the recommended steps as part of our AI Enablement Assessment service (powered by Pheebs, of course)! As we get more feedback from companies on Pheebs as a product we'll continue to expand it and possibly create a managed offering.

 Thanks for the details! I figured there was some sort of recommended next steps feature. Congrats on the launch!

I enjoy the overall idea. I think a lot of organizations are struggling to figure out how to track this information and ensure that they are getting the value that they thought they would for AI. Does it simply create a report or does it help in creating restrictions, identifying patterns, etc.?

Maker

 Hi Jay, thanks a lot for your comment - and we completely agree!

Currently Pheebs captures the data and provides analytics using the proposed (repertoire + judgement). To your question, it currently doesn't help creating restrictions specifically, however all reports (engineer and team views) help with identifying patterns and where teams should act with more intention e.g. the team is lacking on artifacts usage let's stand up a skills repo and evangelize it, or the team is lacking on evals let's invest in verification harness, etc.

Congrats Davi! that's awesome

Maker
Moving between subs daily makes it challenging to accurately track use of the models. This seems like it can give a reality check to help encourage more efficient use and model selection.