input_csv ~/RadarCode/geocode-acceptance-tests/input/addresses.csv \ # csv input, use headings as query parameter keys \ # these can be any accessible http server, sometimes we run against our prod index \ We run api-diff with a command that looks something like this: api-diff \ When updating the data in our index, we are usually expecting a steady increase in quality and coverage, and are mostly on the look out for surprising regressions, often caused by changes in data formats. When making a ranking change, there is usually a tradeoff between improved queries and some unintended losses, so we can use api-diff to provide a more quantitative answer as to whether or not the improvements outweigh the losses enough to proceed with the change. In both cases, we are looking to see what changes occurred in the search results. Generate a baseline set of responses for later comparisonĪll you need to get started is a list of queries, as a CSV or URL paths, and two running copies of a server with some changes between them.Īt Radar, our two main use cases for api-diff are 1) when we are making code changes to the parsing and ranking algorithms in Pelias, or 2) when we are updating the data in the search index.Allow a human scorer to quickly score diff’s as positive, neutral, or negative with keyboard shortcuts.Compare a set of saved query response pairs against responses from a new server. ![]() Output diffs to the console, as json, or as an interactive static html page.Compare the responses of a large number of queries between two servers. ![]() What it isĮnter api-diff, an incredibly boringly named suite of tools for evaluating quality or ranking changes between JSON API endpoints. Pelias currently has a great set of regression tests, but we wanted a tool that would let us understand how changes to ranking would affect real-world queries where we didn’t have a verified golden result. It’s hard to have an exhaustive set of tests for every way a query might be written. Geocoders are challenging pieces of software because in the real world, both addresses and queries are messy.Ī sampling of different ways users could search for one address For example, "20 Jay St" becomes "40.7041, -73.9867", along with the metadata that it’s an exact street address in Brooklyn, in New York City, in New York State, in the US. It can and has been used against all sorts of JSON APIs inside of Radar.Ī geocoder is a piece of software that translates freeform strings to coordinates on the earth with structured metadata associated with that place. This post is primarily focused on how we use it against the public Pelias API, but the tool itself is generic. We’re proud to release it on npm and open source it on GitHub today.Ī screenshot of api-diff in console mode, comparing identical API requests sent to two different api servers Motivationĭevelopment of api-diff was prompted by our work on the Pelias geocoder, but it works with any JSON HTTP API. We couldn’t find a suitable one, so we wrote our own, very imaginatively called api-diff. Radar engineers needed a tool for side-by-side evaluations of changes to the services of our recently released suite of geocoding APIs. TL DR: Radar built an open source tool for performing side-by-side evaluations of JSON APIs.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |