by Marty Thomas
Jun 13 '16
It’s hard to know what the next best step should be when starting a company. Last month Ryan and I needed to make a decision with Tend – should we spend a couple weeks making the back-end infrastructure stronger, or spend the time to ship some new features customers have been asking for?
So I gave myself 3 hours to experiment with what what exactly needed to be done if we wen’t the infrastructure route. Specifically, we needed to upgrade our servers to a cluster that could scale more efficiently. And since we were going to need to shut everything down for the server upgrade, I also thought this may be a good opportunity to update the code framework to it’s newest version. After a couple hours I figured that it would take about 2 weeks to complete the infrastructure updates.
We decided to move forward with the infrastructure changes. One of our customers (and good friend of mine), pushed back challenging that we are way to early of a product to be worrying about infrastructure, when there are customer-facing things with the product that need to be fixed/updated. He has a good point. I nearly decided to scratch the infrastructure route all together. But instead, Ryan and I discussed what the minimum amount of customer-facing updates we needed to do, while still pushing the infrastructure forward.
So we continued forward, unsure if we made the right decision. Since we were also focused on doing some customer-facing updates, two weeks turned into three, and we finally launched the infrastructure changes. It felt good. And then shit hit the fan.
With Tend, one of the most important things we can get right is making sure that the visitor data is as accurate as possible. Before creating Tend, not being confident with the analytics data was the most frustrating thing for Ryan and I. It’s one of the reasons that we decided to start Tend in the first place. Getting the data accuracy as close to accurate as humanly as possible, will be a huge part of Tend’s success. With that being said, after we launched the infrastructure changes, we stopped collecting new visitor data.
We launched the infrastructure updates on Sat, Sept 26th at 2pm CST. After the updates, we did our basic testing, and everything looked to be tracking as expected. I gave myself a pat on the back, and got ready for a family wedding that night. Then, on Monday Sept 29th, at 10am CST, I get word that some of the visitor data wasn’t tracking. I jump on the computer and quickly discover the bug, and push up the fix.
We discovered that the tracking code was not working for any new visitors to the sites. All of visitors that previously visited the site were tracked as expected (which made it look like everything was working just fine). But Tend was ignoring all new visitors.
But the bleeding didn’t stop there. On Oct 8th at 1pm CST the old server was deleted. We left this server up for about a week after the infrastructure updates to make sure everything was as it should be. Then, I felt confident deleting it. However, our old tendinc.com domain was still pointed to that old server. So, tracking once again stopped for users using the tendinc.com domain. We discovered and fixed that bug at 9am the next day.
Hopefully the bleeding has stopped now. Moving forward, we will start to integrate more thorough testing into our updates to make sure Tend is always tracking as it should. Testing was one of the first things to go out the window when trying to quickly launch Tend. But now it’s time to make sure it has a solid place in our development cycle.
Loosing data is one of the worst thing we can do at Tend. If our customers can’t have confidence in the data, we won’t have a product. It’s as simple as that. So thanks to all of our users for putting up with us through our growing pains! Our early users are an integral part in making the product better and stronger. I’m more excited than ever that we’re on to something really cool here with Tend. I think it has the potential to fundamentally change how people approach website analytics. And I’m glad we have our early users to join us on that journey.
Subscribe to our Newsletter