By Max Sherman ยท Nov 5, 2024
We ran an A/A test with our FB ad to test if doubling ad spend doubles conversions. The test ran for one week, and the results were 25% worse on link clicks and cost per conversion (sign ups not purchases).
This is a pretty large difference but is potentially explained by the "control" A group having been running for months, so maybe more optimized by the FB system.
My takeaway is that when we run an A/B test, we need to beat the control on these metrics by 25% or more. Otherwise it may just be noise.
An SEO course from an entrepreneur I follow on Twitter released this last week. I want to take a look at it this month and see if I can scope out some work to invest in SEO, as this is a high ROI channel for us.
We shipped a huge upgrade to our diarization (speaker segmentation) ML model. There's a lot more work to do, but I think we can realize enough perf improvement to show the fast preview meeting minutes almost instantly. This may involve some tweaks to the open source whisper model as well.
I think we can get a lot of wins by using ffmpeg in process to hold a buffer of the decoded (as WAV) file. Then only serving ML models with this buffer. There's a lot of room for improvement.