An In-Depth Look at the Software Development Engineer in Test Role at Microsoft

When I contact candidates about the SDET (Software Development Engineer in Test) role at Microsoft, there is initial hesitation about being in a “test” role. In order to help provide clarification on what it means to be an SDET and dispel misconceptions about testing at Microsoft, I enlisted the help of a current Principal Test Lead (John Rodrigues) with the Hotmail Team.

In this interview with John, he helps explain how testing is different at Microsoft versus other companies, what some of the technical challenges are that SDETs face on a daily basis, and what types of candidates Microsoft looks for when hiring SDETs.

Tell me about your background. How did you end up in the test discipline at MSFT? What is your current role?

Since I am a Bay Area native, I decided to stay close to Silicon Valley when I was searching for colleges because I wanted to be close to the hub of technology. I ended up attending the University of California at Santa Cruz and focused my efforts on acquiring an engineering degree. Several years, and many pizza boxes later, I graduated with my Bachelor of Science degree. In early 2003, I joined Microsoft working as an SDET with the MSN Calendar team. The position wasn’t familiar to me at first. I’d spent several nights and weekends developing programs for classes and figured I would eventually move into a Software Development role. When I delved into the SDET role however, I learned how technically challenging testing is at Microsoft.

The technical demands and requirements for the SDET role were the same as an SDE role. What I really liked about the discipline, and continue to love to this day, was the breadth that came with it. I wasn’t just coding an API in my office all night long. Rather, I was helping to design the API, write code to validate it, work with partners to integrate with it, verify that it would work around the world for multiple markets and languages, and ensure the scalability of it in the future. It has helped me grow into a well-rounded development engineer today.

Speaking of today, I’ve had the opportunity to work at Microsoft for about 9 years. During that time, I moved into several different teams and have had lots of different responsibilities. Starting in 2005, I moved into a leadership role, managing a small team of 3 engineers. Since then, I’ve built up experience in not only building large scale services, but also in managing teams. I’m currently a Principal Engineer managing five teams in the Windows Live Hotmail Team where I work on the storage and anti-abuse platforms.

How is testing different at Microsoft versus other companies?

When someone typically thinks of a test position, thoughts of manual testing or button pushing often come to mind. Upon closer inspection however, one may notice a few interesting things about the position at Microsoft. First is the title. In the early 2000’s, Microsoft went away from hiring test-only engineers, commonly referred to as Software Test Engineers. We are now focused on hiring Software Development Engineers in Test. We’ll talk more about the differences in just a bit. Second, are the requirements of the role, which have changed over the years. The expectations are that folks have a development background and are able to meet the same type of technical hiring bar as SDE candidates. Third, SDET’s are a first class citizen in the engineering triad. They are responsible and involved from Day 1 of a project until that project is terminated. Microsoft is committed to building high quality products that scale the globe as rapidly as possible; SDET’s are core to this commitment.

So, what are some of the key differences when it comes to the SDET and SDE discipline? To put it simply, we hire developers who have a mindset bent on understanding how something truly works and then breaking it. SDET’s are involved in the end to end design of how a product comes together and how the integration points will work in the eco-system. Once a design is flushed out, SDET’s work alongside their counterparts to build instrumentation and automation into the product. Often times, this code will ship and be used for validation of a deployment or for debugging and triangulation purposes.

Here’s an example: Let’s say that we need to add an API to Hotmail to retrieve a message for a user from a data store. We’ll call that API ‘GetMessage’. The SDE and SDET would work together to draft up how this API will work, ensuring the requirements for it are met. Requirements may entail such things as what data it will return, the format it will be returned in, the performance of the API, etc.. Once the requirements have been flushed out, the SDE and SDET go to work on writing and validating this new API. The SDE will code up the API and use the in-house test framework to ensure that the right data is being returned in the right way. The SDET, in tandem, is building up the necessary collateral to validate the API. It isn’t as simple, however, as just validating input and output. An SDET needs to have in-depth knowledge of the eco-system and how things will work end to end. For example, how many requests per second will be made and from what locations? Will there be cross data-center calls and, if there are, what type of latency will we be dealing with? If the latency is too high, will we run out of threads waiting for responses? What happens if a particular disk is responding slowly or not at all and you can’t retrieve a message; how do you make sure it is retrieved from another location without user impact? Validating the input and output is the easy part and most tests can either be auto-generated or quickly added with ease. It doesn’t stop at the API level either. Our products often have an advanced user interface that customers use.

Continuing on this topic, think about how one would automate Hotmail UI such that the same test case works across multiple browsers and operating systems. On top of that, what would you do when the UI changes and elements are moved or modified? How do you ensure validation of the product and quality are taken into account without slowing down the product? SDET’s are responsible for these types of questions – both asking them and checking what happens. They will spend significant energy on ensuring we can ship a reliable product that our customers love. When it does come time to ship a product, the SDET team makes the call on if the product is ready or not. They are the gatekeepers to releasing bits into our customer’s hands.

As a final point, it is worth noting that people often move between disciplines at Microsoft. Due in part to the requirements we have for each role, people can move between PM, SDET, and SDE with relative ease. We have people who make the jump from SDET to SDE, SDE to SDET, etc.. Movement like this helps to broaden the perspective one can bring to the team.

What are some of the technical challenges faced by SDETs?

Microsoft creates products for the masses. Be it cloud or client software, we expect people from around the world to interact with our products daily. Working with this in mind, you can quickly start to see some of the challenging aspects of the SDET role: scale, permutations, performance, globalization, integration, security, privacy and more.  Each of these is a complete topic in and of itself, so I’ll pick one and give some examples of the technical challenges an SDET works with daily.

Let’s talk about one of my favorite topics, scale built into Microsoft’s DNA. We know what it takes to build software that can be used around the world by millions to billions of people. This holds true for our client and cloud offerings. In the Hotmail team, our focus is geared primarily towards the latter. With over 1.1 billion Hotmail accounts provisioned around the world, we have a host of challenges that SDET’s face in order to deliver a quality product in a timely fashion. We have different hardware permutations to think through, different user types from small to large to heavily customized to multi-locale based users, numerous hardware failures from different architectures and regions of the world, and different browsers and clients that access Hotmail on different OS’ and device types. This is the tip of the iceberg, but paints a picture of what our engineering team works on. SDET’s work heavily with their SDE peers to design new features, debug existing problems, and plan for better efficiency in this ever changing world. They work on writing instrumentation into the product and using that instrumentation to build a set of highly effective automated suites that ensure quality on every check-in and user transaction. This translates heavily into ensuring data integrity is kept even when we are struck by the most pathological situations possible. On top of this, the team has produced significant collateral in scalability, capacity, and performance analysis that uses instrumentation from the real world to assess changes to code and hardware with minimal to no human work required. Our ability to ship and scale our product up quickly would not be possible if it wasn’t for the SDET’s we have in place today and the strong and diverse backgrounds they bring to the table.

As stated earlier, the requirements for the role are similar for SDE and SDET. We expect SDET’s to come in with a strong computer science fundamentals, the ability to quickly solve challenging problems, great communication skills, and a focus on understanding how things can and will break. Having these skill sets and a mindset focused on the ins and outs of how the product works, enables an SDET to solve some of the most difficult problems we face when building software. It is a rewarding and inspiring role.

How do SDETs interact with fellow PMs and SDEs?

SDET’s are a part of our core engineering team which, in most parts of Microsoft, are comprised of three disciplines: SDET, SDE, and PM. They are a first class citizen within this engineering triad and work daily with their counterparts to design, develop, qualify, ship, and maintain software. From conception to termination, an SDET is involved in all aspects of the software development lifecycle. You will be expected to engage in product and feature decisions, architectural road map, product designs, prototype development and testing, automation development, bug analysis and fixing, and debugging.

Often, a small team of SDET’s, SDE’s, and PM’s are created to tackle a particular component or feature. This small triad of engineers has a close working relationship and will typically meet daily to discuss the work completed, complications that arose, and what is left to ship. The SDET is the gatekeeper for when a component, feature and overarching product is ready to go live. Simply put, these three disciplines are at the core of Microsoft’s engineering DNA and work together to build some of the most widely used products in the world.

If a candidate has been in more of a development versus testing role in the past, will the SDET role be a good fit?

Some of the best candidates we’ve hired into the SDET role have come from a strong development background. We often look for developers who are proactive about understanding how something really ticks, how it might be used from start to finish, and how customers will interact with the product. Folks that are heavily engaged in debugging activities, proactively writing unit tests, and understanding and coding for pathological scenarios will often do quite well in an SDET role. They get to combine their knack for improving the quality of the code and experience for the customer with a strong foundation in computer science.

One will typically notice that the interview questions, focus areas, experience, and knowledge requirements are similar between the SDE and SDET role. The key differentiation, which was touched on above, is really finding that person who has a mindset bent on understanding how things work, how they break, how things can be made more resilient. The hiring bar for each role is the same and we often see folks moving between the two disciplines. Both have the potential to excel in their career with no caps.

What do you look for when hiring strong SDET candidates? How much emphasis do you put on prior testing experience?

By now, folks should have the impression that the SDET role is different at Microsoft when compared to similar roles out in the industry. We are looking for people that have a strong foundation in computer science and top notch problem solving skills. SDET’s code and apply engineering principles to problems daily. They focus on algorithmic problems, have a passion for quality, and have the ability to keep the user in focus at all times. SDE, SDET’s and PM’s are involved from birth to termination, and together, form the core of our engineering group at Microsoft. SDET’s are significantly involved in shipping major products to potentially billions of users around the globe.

When scanning for potential candidates, we have a similar approach for SDE and SDET. We look for development work done in the past, a degree in computer science (or similar field), good problem solving skills, passion for technology, passion for quality, an ability to think about end-to-end product usage, and excellent communication skills. Once we have a strong set of resumes that meet our basic criteria, we drill in to spot differentiations between potential SDE’s and SDET’s. Prior testing experience is one possibility, but more often than not, we look for folks who speak to reducing the number of defects per line of code, those who work across multiple components or features to see the product through to completion, or who have spent significant time debugging and improving a product once live. By now, you can probably see we put a heavy emphasis on hiring for strong development skills coupled with a passion for ensuring the quality of our product is top notch when it goes into the hands of our customers.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: