In standup the other day, my team's DBA was talking about running smoke tests for his most recent project. I've heard people talk about smoke tests...
For further actions, you may consider blocking this person and/or reporting abuse
I'm so happy that you included acceptance testing! As a front end engineer and accessibility specialist who works in consulting, that's the main kind of testing I worry about.
I had no idea that the umbrella term "acceptance testing" covered so much from spec validation to accessibility and security. Do you mind if I update the post and credit you for providing additional domain knowledge here?
Yeah, no prob!
I know that there are automated tools for accessibility testing but I really don't know much about them. Do you have any that you go to in particular, or is most of your accessibility acceptance testing a manual process?
I use Google Lighthouse to do accessibility auditing on the frontend.
Accessibility acceptance testing something I'm working on at UI-licious. Right now the test engine has a bias to use ARIA descriptors as labels for buttons and input fields. Working on refining it further and allowing people to configure a strict mode to strictly evaluate ARIA descriptors and semantic HTML elements only when performing acceptance tests.
So accessibility testing is about 60% automated, 40% manual. The are some things that fundamentally can't be checked automatically, or rather, there are some errors that won't be found with automated tools. (Some of the testing we think about in this space is 'will assistive tech recognize this HTML correctly', and if assistive tech can't parse it correctly, that usually means the automated tools can't parse it correctly either.)
Hi Jess,
thank you for this overview. I try to continuously improve my test skills since the tests improve my confidence in the code.
I'd like to add one aspect. I found it on a german website. Besides the questions "What is tested" and "When are the tests executed" a third question appears: "Who is the tester?".
There are some kinds of tests the developer himself is responsible for: Unit tests, integration tests, smoke tests, regression tests, ... but some tests should be executed for example by the QA, the stakeholder or the customer: Alpha and beta tests, Usability tests, Accessability tests and more.
It's really important to test your own code but it's also important to be supported by other people, the users, to be sure that your app is working.
Thank you very much for this great resource.
I actually dislike the distinction of “who is the tester?”
Software lives and dies by the team, not by the individual. I definitely agree there. As such, testing code, both in an automated and manual fashion, falls collectively to the team and not to an individual. Carving out by role is not super useful, imo.
And I definitely missed a few things here, especially with regard to accessibility and usability testing. Which makes sense - I’ve never touched production front end code. It’s very much a personal blind spot, but there are a number of great resources and points elsewhere in the comments!
Hi Jess,
I think I understand your point and honestly I agree with it. I don't really use the Who-question to distinct the kind of test, I mean, obviously it's not the fact that a developer executes a test which makes it a unit test.
I just thought of it as a nice extra information. In my experience it's usually not a customer or an end user who executes unit tests. These people test by clicking through the app or at least it's what I think they do ;-). So the "Who" is more like a weak evidence when you ask how to distinct between kinds of tests.
(I have to admit that I maybe missed the topic a little bit but I like the discussion.)
Oh My God! What a big bundle of knowledge this article and these comment-discussions are! I'm elated that I drop-by here.
The article in itself is self-sufficient but these comments are the cherry on the cake.
Thank you so much , the entire community is giving back selflessly!
This is a good distinction, but it doesn't necessarily sit at the top level with the other questions, because the "who" is often embedded via personas in all the testing levels from unit on up. So from a certain perspective it doesn't matter who is testing since the who can be embedded. HOWEVER..I think your statement was going more towards User acceptance testing, which doesn't mean much to developers, nicely proving your point. Developers consider success as "I fulfilled the written requirements" whereas everyone else considers success as "the customer got what they wanted." So there is necessarily a gap here in conversation on the value of User acceptance testing. But good UAT is necessary for success, business-wise.
Nice article, I read through it and through the comments. But it should be called "Devs view of QA terminology" or something like that. I am scared for junior QAs reading this and thinking this is it.
I would add one more thing for all the starting in QA. As QA is the only engineering field I know with so many buzzwords, it is important for each team/company/project to create a glossary at the beginning of working. This is in my experience the only steady solution for misunderstandings.
Hmm. Why are you scared for junior QAs reading this article?
This article provides a general overview very useful for devs, but is it not the same for QAs?
In case the concepts are wrong, it would be great if you can share some links with starting points for QAs.
I'm hoping that a junior QA reading this can read in my bio that I'm not a QA person, and has the wherewithal to read critically and understand that this is not a comprehensive overview of the entire field! I think I make it pretty clear up top in the article that this is a broad overview from a dev perspective, but if you disagree let me know where I could try to highlight that this is nowhere near the end of testing education :)
Thank you for this article Jess! Testing vocabulary is indeed unsettled and sometimes very confusing. I found this article on Guru 99 has been helpful in the past, and now I will be adding yours to my resource list!
A couple points I would like to share.
"...Acceptance testing is usually a set of manual tests...". Acceptance does not have to be a manual process. Ideally the entire test/s process should be automated as to facilitate rapid deployment. Tools like uilicious.com/ , cypress.io/, and selenium to name a few.
"...There isn't necessarily pass/fail criteria associated with a performance test...". Failing would be 'application did not load' :) . Honestly though when application reaches the point of implementing performance testing a 'known good' level of operation is know. That becomes the baseline performance. Change should decrease hardware usage per request, decrease wait time to respond, or both. Rarely should changes increase the error rate.
Thank you again for the article. I really enjoy seeing and reading other people care about testing and knowing more about it as it applies to software.
Ah. I think I was using a slightly more literal definition of Acceptance Testing - literally checking that the specs are met on a quick pass through - but those resources for more robust acceptance testing could be very helpful to someone setting up a new environment. Thanks!
Also, for performance testing, I'm assuming if you get to this point, then the application will load. I'd hope you have failing tests well before this stage if things are that broken! :) If you have any good resources on "known good" performance levels I'd love to add that to my personal resource list! It can be tricky because so often when we test things as developers, we're testing on high speed ethernet on fairly good computers, and that is not an accurate depiction of all our users. Having some accepted benchmarks that work for a wider variety of environments would be very helpful!
Re: Acceptance Testing, your view is correct. Those 'specs' can be provided in a machine format (image, HTMLDom, etc) to be compared to the output from the test run. :). Reading some of the other comments I agree 'Acceptance Testing' covers a wide swath. Maybe break out Security, Compliance, etc. I have no idea what they would be called though. :S
Re: Perf. Testing, Unit tests can pass, integrations pass, but the running application does not respond before a timeout; but maybe does in 120 seconds. A trick I learned was that Chrome can throttle the network speed of a request. This means headless version can as well :). The other side of Perf. Testing is x successful responses in Y seconds under Z load. The measure of these can be benchmarked using APM tools (Application Process Monitors). New Relic, Sentry, and AppDynamics are three of the more well known vendors.
"Application Testing": as wide as the horizon and as deep as a gravity well of a black hole.
I was under the impression that "smoke test" was borrowed from Electrical Engineering. i.e.: "This is the first time we're powering on the device/circuit. Be ready to turn it back of if we see or smell smoke."
Interestingly, this is the first I've heard the term used as a reference to plumbing. TIL, I guess.
So I looked at this and from what I found online (I will admit, I did not verify the sources), the smoke test concept from electrical engineering came from plumbing, so it's a term that's traversed several industries :)
Great post. In "acceptance testing" we can add domain like Security, some tools to include into CI process can search problems like *injections, CSRF, ...
I had no idea that the umbrella term "acceptance testing" covered so much from spec validation to accessibility and security. Do you mind if I update the post and credit you for providing additional domain knowledge here?
As you want :) If you edit, this dev.to article
Find security issues before the code goes into production
Omer Levi Hevroni
I never heard the plumbing metaphor about smoke tests before, but it makes sense. When I think of smoke tests, I think of a different metaphor: "Where there's smoke, there's fire".
If the system can't pass the smoke tests, which are usually simple tests focusing on basic functionality, then there is something seriously wrong. Get ready to call the fire department.
I had always believed that a smoke test was one where you saw it plugging in as "let out the magic smoke", or short-circuited and caught fire.
I'm not arguing that you're wrong. The plumbing use of the term seems to predate the EE use by almost a century.
Awesome article @thejessleigh
Here are my 2 cents :)
Regression testing and smoke testing both can be done with Automation or with Manual or both (it really depends on whether the team has implemented Automated tests or whether they are doing Manually testing only)
In the products that i have worked for,usually both gets done (Automation and Manual)
Acceptance testing
Acceptance testing gets done after end-to-end development.
Here we can consider E2ED as end-to-end development of a feature or end-to-end development for a release (combination of all features)
As you have stated on your article, it really depends on the team and what kind of workflows they have created.
Finally! I now know what these buzzwords mean. Thank you for posting this!
Nice and comprehensive article!
Just want to confirm, is the "Acceptance Testing" similar with "End to End Testing"?
And It might be 'nice' if You explain a little about "Penetration Testing" as security matter :)
Thanks
Great, you have summed up the different testing types so well. However, my view on Acceptance testing is that it is not always manual testing or manual test cases. Instead, it focuses on scenarios which a tester writes to confirm the product/feature functionality keeping the customer context in focus. And since most of us if not everyone who operates in Agile, we have to automate acceptance tests as well. If we don't do so, then it'll be quite tough to produce in-time releases. By the way, I've also got a bit of hands-on with different types of testing and wrote a little piece by myself. Lastly, it always feels a bit improved on learnings after reading such a good article as yours was.
Spot on, but missing some that are increasing important these days:
Penetration Testing (Pen text)
General Security Testing
User Privilege (role) based Testing
Data Protection Testing
Of course one could argue that these could be encapsulated with the other tests, but cyber security should be build in at design stage and then tested to ensure there are no vulnerabilities. Recent York City fiasco is a good example, where developer found a vulnerability after implementation and was repaid by been reported to Yorkshire Police.
I haven't had time to read all comments so sorry if repeating what others have already raised.
Thanks for the post and it covers some important types of testing one should know.
Hi, nice article. Anyway Load tests are part of Performance testing, together with Stress tests and Scalability tests. Means Load/Stress/Scalability tests are tests for specific part of software performance.
Yup, as I indicated in my article, load testing is indeed a specialized form of performance testing.
Great article!
Great post!
Great overview!
Thanks. Now, I am no longer an ignorant dingus; rather, I feel like a loud mouth Bob :). I really learnt a lot from this article. Thanks.
Awesome post! I always get somewhat confused by the different types, and there are a few I haven't heard before.
Thank you for this great writeup. Now I know about Smoke Tests, too! :)
Nice article.