Hardware Test Engineer I/II

  In early 2022, I decided I was going to move in with my long-distance partner down in Oregon. SpaceX is not normally known for allowing remote work, but I floated the idea a few months before my August deadline for leaving, just in case. My good friend, and head of the build reliability engineering team at the time, also proposed a project for me to work on that would enable fast triage and disposition of hardware failures coming out of test. I'd previously taught him some basic web development to get a prototype version of the app up and running, but it was fast approaching its usability limits and needed professional support. They approved me to work on this for three months, with the stipulation that I came in person for one week a month. Since it was going to be short term, I agreed and began working on this website!

  This level of web development was rather new to me, at the time, as I'd always said I'd never become a full time web developer since before even starting college. Still, that wasn't going to stop me, and I quickly began making improvements. One thing SpaceX had already taught me was that development velocity was paramount, and that there should be as little friction as possible between changes getting made, becoming validated, and then deployed. So, I started by creating a fully automated development/staging/production devops environment, and moved the application to docker with a RESTful flask backend and Bootstrap frontend. After three months were up, they decided to let me keep working remote so long as I still came in a week a month. For the next year and a half, this tool grew and improved, gaining hundreds of users a day both in and out of the reliability team. It turned out that many features of my website ended up providing a better experience than the ones developed by SpaceX's core applications engineering team, at least for Starlink's use-case. This did not go unnoticed, and that team ended up reaching out to create an equivalent, and officially supported, first-party tool in the SpaceX application ecosystem. They even created a small team dedicated to its creation and maintenance. This also came at a perfect time, as I was losing my ability to keep up with my custom website as a solo developer. Oh, and did I mentioned I'd been supporting satellite hardware test remotely for the latter nine months at this time, as well?

  In March of 2024, the team lead for components test software (the components team being the parent group of satellite hardware test by that point) reached out and asked me if I wanted to join a small two person infrastructure team. Considering my remote nature, this made a lot more sense than trying to continue supporting satellite hardware tests, and I was going to be able to provide a lot of experience with devops infrastructure and docker that could greatly benefit the team. I agreed, and began working on dockerization and unification of the team's tools into a monorepo. I also began supporting a relatively new tool for running test software that the team was developing, along with a small slew of miscellaneous tools such as those to image and provision servers over PXE.

  After dockerizing all the infrastructure and moving them into the monorepo, my next push was to improve velocity and reliability by adding automated build, test, and deploy tasks for new development, staging, and production environments across them all. I did this by combining a unified set of Makefile interfaces, with automated runs via ansible from pull requests and main branch merges. Application and server monitoring was also added via tools such as Sentry and Grafana, which were piped into OpsGenie so the infra team could be pinged during outages. Backups were also made of production databases during deployments, with restore tooling that could be used for emulating production data during staging, development, and local deploys. Overall, these changes greatly improved the speed at which the team could develop, and helped ensure that what we were deploying was validated as thoroughly as possible in advance.

  At the end of January 2025, I was notified by the company that my remote work exception would no longer be valid as of April 4th. I was now living in northern Washington state and close to a two hour drive, each way, from the closest SpaceX facility. I therefore opted to quit after roughly six years with them in total. Considering the average is two to three years, I'd say I did quite well, and am very proud of what I managed to achieve with the company in that time! If you'd like to know more, feel free to contact me, and I'm happy to share what I'm allowed! This summary was ultimately just the tip of the iceberg for an experience that made six years feel like twelve.