At a glance
- Client: a startup in San Francisco building not just a fitness application but a whole fitness system, still unique in its field
- The product: a tablet carrying videos and specialized software that builds an individual training schedule and tracks intensity, plus full-size training equipment aimed at different muscle groups
- When PFLB joined: at the beta phase of functional testing, with just under a year left before market release and over half the code already written
- The obstacle: the tablet, called Hercules, was not available for sale - a custom device, with no comparable substitute, and large enough that it looked like a TV and had to be plugged into a wall outlet
- What made testing possible: the customer shipped two Hercules tablets to the team
- Stack worked with: Realm database, SQLite, Mockito and Robolectric frameworks, alongside manual testing on the physical device
- Two fixes that shipped: ExoPlayer adopted to solve the video download blocker, and auth0 introduced to fix the authorization problems, with e-mail and telephone number added as credentials
- Working model: a deliberately engineered 24-hour shift across time zones, which significantly reduced release time and time to market
The problem was the device, not the code
The application was being developed for one specific device, and the first problem the team hit was that device's uniqueness. The tablet called Hercules was not on sale anywhere. It was a custom build, and no similar device could stand in for it.
This is a harder constraint than it sounds. Testing on similar devices or emulators does not guarantee the same results on the native device - a rendering path, a media codec or a storage behaviour can differ in ways that only surface on the real hardware. The customer solved it the only way it can be solved: they sent two Hercules tablets, which at first sight looked like televisions, since each needed connecting to a wall outlet. If you cannot equip a whole team with the native device, the working compromise is to find at least one, so the QA lead can be confident the team has not missed anything.
The second constraint was distance. PFLB worked alongside the client's designers, development team and software testers, all thousands of kilometres away. The project was a startup, so communication and working processes were less settled than they first appeared, and the customer had no prior experience of working with team members in different time zones. Integrating into the process was itself a challenge - and it was solved procedurally rather than by trying harder.
How the time-zone gap was turned into a shift
- The sprint calendar moved. The customer postponed sprint planning to Thursday, so PFLB received the next sprint's tickets on Friday and had a full day in hand to review them and ask clarifying questions before work started.
- The working day moved. PFLB started a couple of hours later, so a daily meeting could be held in the evening with the customer to update status and settle urgent questions.
- The result: most tasks were finished by the time the customer got back to work the next day, and sprint tasks were ready for testing and merging. Release time was significantly reduced.
Because the customer performed the testing, PFLB's job was to prevent major bugs from reaching them and to deliver high-quality code. The fitness system included the power equipment, but only the tablet was in PFLB's hands - so only the visual software part was available for testing. To cut testing time inside that boundary, a PFLB QA engineer was engaged for smoke testing of the features.
The two improvements that mattered most
- Easy and fast downloading. The program contains a customized exercise program - videos that show the user how to perform each workout correctly. Video downloads became one of the major blockers. The team proposed and implemented ExoPlayer, an application-level media player for Android.
- Security of personal information. Training programs are individual, so users needed secure and convenient personal accounts - but the system had major problems with authorization. The team introduced the auth0 service to simplify the authorization process, and added credentials including e-mail and telephone number.
Results
- A large number of bugs were fixed and architectural solutions were improved.
- The application became more stable and more user-friendly.
- Time-to-market metrics were significantly reduced, thanks to the 24-hour work shifts created by the sprint and working-day adjustments.
- The video download blocker was cleared with ExoPlayer; authorization was rebuilt on auth0.
Questions this engagement answers
How do you test software built for a device that is not on sale?
You get the real device, or you accept a known gap. Testing on similar devices or emulators does not guarantee the same results on the native hardware, which is why native app testing on the actual unit matters here. The customer shipped two Hercules tablets; if outfitting the whole team is impossible, one unit in the QA lead's hands is the minimum that keeps confidence honest.
How long does an embedded QA engagement run?
Longer than a bounded load test. This one started at the beta phase with just under a year to market release and over half the code already written, and ran on the client's sprint cadence to launch. Bounded diagnostic engagements are measured in days or weeks; joining a product team through to release is measured in the release calendar.
Can a distributed team across time zones ship faster, not slower?
Yes, if the calendar is designed rather than endured. Moving sprint planning to Thursday and the working day a couple of hours later turned a gap into a handover, so tasks were finished and ready for merging when the customer came back online.
What can you not test when only part of the product is in the room?
Say it plainly: the fitness system included power equipment, and PFLB had only the tablet, so only the visual software part was in scope. Naming the untested part is more useful to a buyer than implying full coverage.
Building for hardware your testers do not have?
Emulators will tell you the app runs. They will not tell you it runs on your device. If the product only exists on custom hardware, the testing plan has to start with getting at least one real unit into the team's hands - and, if the team is spread across time zones, with a sprint calendar built to hand work over rather than to collide.
