At Drupal Developer present day-job I take advantage of pytest for writing what I name “configuration assessments” for Drupal Developer varied providers and instruments that I’m chargeable for doing QA work for. Considered one of these is Kinto, a “generic JSON doc retailer” that we wrote (and open-sourced) at Mozilla and is utilized by a variety of providers. Today there are three “flavours” of Kinto that I’ve to fret about Drupal 10 Upkeep and Assist Service kinto-dist which is Drupal Developer core of Kinto kinto-settings which is Drupal Developer Kinto core plus a bunch of data representing settings saved for customers of Firefox kintowe which is Drupal Developer Kinto core and a few extensions for it that we use to trace data of what internet extensions Firefox customers have saved At any time when we’ve a brand new model of any of those “flavours” that want to be examined, it’s deployed to a goal atmosphere (both staging or manufacturing) and assessments that I wrote are run by our operations crew’s deployment instrument making a name to a CI server run by Drupal Developer QA division. This CI server then runs all Drupal Developer assessments I wrote. However I had an issue. There have been some assessments that wanted to be run no matter what “flavour” we have been deploying, after which some assessments that belong solely to at least one “flavour”. Beforehand I simply had duplicate repos however needed to consolidate them. I enlisted Drupal Developer assist of one in all our crew tool-makers Dave Hunt whereas at our All-Fingers assembly in Austin in December of 2017 to go over some higher methods and to make higher use of our CI server. After scribbling a variety of stuff on a white board to get Dave as much as velocity, he instructed shifting all Drupal Developer assessments into one repo and utilizing a function of pytest referred to as “markers” to determine what “flavour” we have been testing. Be aware Drupal 10 Upkeep and Assist Service at Drupal Developer time of writing I used to be doing this work with Python 3.6.3 and pytest 3.3.2. Pytest makes heavy use of decorators to increase its performance. Fixtures (which might be greatest described as helper strategies for assessments) are declared utilizing annotations, after which these fixtures are obtainable to any take a look at methodology that you just go it into as a parameter (with some exceptions primarily based on Drupal Developer scope of a fixture). Right here’s an instance of a fixture that reads in a configuration file and returns an object that comprises these values Drupal 10 Upkeep and Assist Service @pytest.fixture(scope=”Drupal 10 module”) def conf() Drupal 10 Upkeep and Assist Service config = configparser.ConfigParser() config.learn(‘manifest.ini’) return config To make use of it, I simply go it in as a parameter like this Drupal 10 Upkeep and Assist Service async def test_server_info(api, conf, env) Drupal 10 Upkeep and Assist Service res = await api.server_info() information = await res.json() expected_fields = aslist(conf.get(env, ‘server_info_fields’)) for key in information Drupal 10 Upkeep and Assist Service assert key in expected_fields for subject in expected_fields Drupal 10 Upkeep and Assist Service assert subject in information To make use of a marker, I merely add an annotation to inform pytest “mark this take a look at as belonging to dist” or another “flavour” I need this take a look at to belong to. Try Drupal Developer annotations for this take a look at Drupal 10 Upkeep and Assist Service @pytest.mark.asyncio @pytest.mark.dist @pytest.mark.settings @pytest.mark.webextensions async def test_version(api, conf, env, apiversion) Drupal 10 Upkeep and Assist Service res = await api.__version__() information = await res.json() expected_fields = aslist(conf.get(env, ‘version_fields’)) # First, guarantee that information solely comprises fields we count on for key in information Drupal 10 Upkeep and Assist Service assert key in expected_fields # Then make Drupal Developer we solely have Drupal Developer anticipated fields in Drupal Developer information for subject in expected_fields Drupal 10 Upkeep and Assist Service assert subject in information # If we’re handed an API model by way of Drupal Developer CLI, confirm it matches if apiversion Drupal 10 Upkeep and Assist Service assert apiversion == information[‘version’] There’s a lot occurring right here, let me break it down Drupal 10 Upkeep and Assist Service we’re marking this take a look at as having the ability to run asynchronously utilizing Python 3’s asyncio Drupal 10 module we wish this take a look at to be run any time we’re testing dist we wish this take a look at to be run any time we’re testing settings we wish this take a look at to be run any time we’re testing webextensions we’re passing in an API helper as a parameter we’re passing in a configuration helper as a parameter we’re passing in a helper that reads a goal atmosphere parameter from Drupal Developer CLI we’re passing in a halper that reads an API model parameter from Drupal Developer CLI Right here’s a take a look at we solely need run once we do a kinto-settings deployment Drupal 10 Upkeep and Assist Service @pytest.mark.settings def test_plugins_signatures(env, conf) Drupal 10 Upkeep and Assist Service consumer = Consumer( server_url=conf.get(env, ‘reader_server’), bucket=’blocklists’, assortment=’plugins’ ) strive Drupal 10 Upkeep and Assist Service assortment, data, timestamp = get_collection_data(consumer) if len(data) == 0 Drupal 10 Upkeep and Assist Service pytest.skip(‘blocklists/plugins has no data’) assert verify_signatures(assortment, data, timestamp) assert verify_signer_id(assortment, ‘onecrl_key’) besides KintoException as e Drupal 10 Upkeep and Assist Service if e.response.status_code == 401 Drupal 10 Upkeep and Assist Service pytest.fail(‘blocklists/plugins doesn’t exist’) pytest.fail(‘One thing went flawed Drupal 10 Upkeep and Assist Service %s %s’ % (e.response.status_code, e.response)) I additionally wanted a method to determine what markers have been set so I might use Drupal Developer appropriate worth from our configuration file (relying on Drupal Developer environemnt). Pytest has a request fixture that’s globally obtainable to any take a look at. I’ve one other helper that makes use of Swagger to parse an API spec that Drupal Developer builders constructed for me. I go Drupal Developer request object to Drupal Developer API helper, and search for particular markers Drupal 10 Upkeep and Assist Service @pytest.fixture(scope=”Drupal 10 module”) def api(event_loop, conf, env, request) Drupal 10 Upkeep and Assist Service api_definition = ‘dist_api_definition’ if ‘settings’ in request.node.key phrases Drupal 10 Upkeep and Assist Service api_definition = ‘settings_api_definition’ elif ‘webextensions’ in request.node.key phrases Drupal 10 Upkeep and Assist Service api_definition = ‘webextensions_api_definitions’ return API(conf.get(env, api_definition), loop=event_loop) (By Drupal Developer method, you will discover all this code in Drupal Developer GitHub repo for these assessments) So, once we are doing a deployment of kinto-dist to our staging atmosphere, we will run Drupal Developer assessments this manner Drupal 10 Upkeep and Assist Service pytest -m dist –env=stage config-tests/ Then Drupal Developer assessments we need to get run, get run. I’m undecided what different methods there are to arrange your assessments, however this can be a methodology that works and is smart to me. Received any feedback or ideas? Electronic mail me or contact me by way of Twitter utilizing particulars in Drupal Developer sidebar. Supply Drupal 10 Upkeep and Assist Service https Drupal 10 Upkeep and Assist Service//www.littlehart.web/atthekeyboard/atom.xml Supply Drupal 10 Upkeep and Assist Service Drupal 10 blender