Elm Package API & Github API combined (using GraphQL Schema Stitching)

Show and Tell
1

I have a very simple GraphQL server running that grabs packages from the Elm Package API (i.e. the endpoint that gives you search results at https://package.elm-lang.org).

You can try some queries in this interactive GraphQL playground here.

Here’s an example query using this Elm Package GraphQL API. Notice that it jumps from data in the Elm Package API (package name) to data in the Github API (stargazer count):

Window_and_Playground_-https___vxw01_sse_codesandbox_io__and_Creating_a_personal_access_token_for_the_command_line-_GitHub_Help

It connects up these two APIs using GraphQL Schema stitching, which is effectively like a SQL join, but between two APIs rather than two DB tables.

This is just a fun example that I use in my elm-graphql workshop. I hope you enjoy playing around with it! If you haven’t seen GraphQL in action, you can open the query playground and you’ll get autocompletion and error highlighting to help guide you toward building up your first query. This is a nice introduction if you want to read a little more about the basics: https://graphql.org/learn/

Github API Rate Limiting

I’m running this with the token for a Github account I created. The rate limiting policy doesn’t play nicely with some of these queries, so if it gets blocked you can just use your own Github token in a fork of my sandbox. Or you can just check out the code and tweak it for fun if you’d like, too!

Querying From Elm

And if you’re so inclined, you can point the elm-graphql code generator CLI to https://vxw01.sse.codesandbox.io/graphql and then start making some type-safe queries against the API from elm!

Let me know what you think if you end up trying it out!

12 Likes
2

One thing that’s not particularly documented is that the package website does not include “stargazer” information on purpose. The theory is that the rankings should reward quality of API and docs, whereas “stargazers” is very very easily gamed by getting your README on Hacker News. So in my opinion, it is mostly a measure of how good you are at online advertising and social media, and things can have tons of stars with lots of technical weaknesses.

And once you start thinking that “stargazers” is a measure of success, you start to get a culture that is more about the presentation than the actual technical details. This isn’t to say that presentation does not matter. Only that rewarding social media skills over quality has effects on quality overall.

I saw a note from @mfeineis along these lines in the #elm-community channel on slack recently. I haven’t looked into the paper, but I reproduced his comment here:

https://arxiv.org/pdf/1811.07643.pdf A paper on how Github stars are actually used in the wild, very interesting. It’s kind of shocking how much influence active promotion has on gaining stars but I guess we already knew that beforehand. Looking at the conclusion it’s sad that GH stars seem to be all about marketing…

Recommendation #1: Stars are a key metric about the evolution of GitHub projects; therefore, project managers should track and compare the number of stars of their projects with competitor ones.

Recommendation #2: Open source projects require an investment on marketing and advertisement, mainly in social networks and programming forums, like Hacker News.

Recommendation #3: When selecting projects by number of stars, practitioners and researchers should check whether the stars are not concentrated in a short time period or whether they are mostly a consequence of active promotion in social media sites

Their tool has some interesting graphs http://gittrends.io/

I think it’d be a shame if this is how people focused their energy within the Elm community. And that “the natural process” rewarded people who are specialized in “good at twitter” and such.

This kind of “influencer culture” is really common these days. It’s obvious on Instagram and YouTube, but it’s also pretty big in the JS world as well. Drama gets rewarded. Competing for sub-count gets rewarded. Daily uploads gets rewarded. But are these correlated with quality?

Anyway, that’s why I try to rank based on metrics that are more aligned with community participation. Those things are harder to measure, and it’s definitely not perfect either, but I think it’s worth fighting for incentive systems that reward mentorship, teaching, sharing knowledge in meetups, etc. (rather than rewarding inflammatory blogging/tweeting skills)

14 Likes
3

I completely agree, metrics are inherently imperfect because they answer a shallow question (is this highly starred), and become stand-ins for deeper questions (is this a quality API) (Kahneman talks about this idea of substitution bias). And I really like your thinking about putting the focus on choosing the best API for the task, rather than the one with the best marketing.

I also love how the elm community tends to rally behind projects and make them better, rather than sprouting out tons of competing libraries and approaches. Some variety is useful, but the NPM ecosystem takes it to an extreme that I think many find overwhelming. Overall, I find it to be much easier and more enjoyable to look for good Elm packages compared to looking for good NPM packages.

There are a few tricky things when navigating elm packages that I’ve found. For example, if someone forks and publishes the package, it will show up in the search results (sometimes even above the original, while the fork may become inactive and fall behind). I’ve had that happen before, though fortunately the fork dropped out of the search results after 0.19. And it would be nice if there was a better package discovery story (though 3rd party curated lists could probably fill that gap as well as anything).

The goal of this GraphQL endpoint

This GraphQL API isn’t production-ready because it doesn’t play nicely with Github’s rate limiting policy. It’s more of a fun GraphQL endpoint to play around with, rather than a tool for building a new elm package search. Also, I think that people sometimes don’t realize how easy it is to wrap existing REST APIs (whether they’re 3rd party or internal) with a simple GraphQL interface. I’m biased, but I find it so nice to interface with APIs through GraphQL! :slight_smile: And with schema stitching, you can have a single source of truth for multiple services, which makes that story even more compelling. That makes it much more declarative because you don’t have to think about intermediary states where data is partially loaded as it makes requests for a different service. That was my main goal: to show some fun things you can do with GraphQL, and to show how easy it is!

4 Likes
4

On the risk of being off topic.

Does the package website actually “reward quality of API and docs” at the moment? If yes, can we write down the rules how it’s done, there might also be a blog post there, especially with all the other content/context mentioned here.

5

Did not realize this at all, I thought until now it was just for GraphQL APIs. :+1:

6

Sorry to hijack the topic (maybe this can be split into another one), but can you tell more about / point to the code that does the ranking? I asked this question before but didn’t get any answer yet. This information (if it is not too complicated to share) would be super useful to get inspiration for the Coq package index (which is in need of big improvements, compared to the Elm package index).

7

@Zimm_i48 It looks it’s currently the authors which have a ranking assigned to them, rather than the packages. I’m guessing it’s done manually. If you look at the list without entering a search term, you see all packages listed by author. ohanhi looks to be the last person to have an assigned ranking, after him it’s all in alphabetical order. And if you for example search for “geo” you will see all of ianmackenzie’s packages listed first, including one labeled with PRERELEASE (which if packages had rankings should probably be listed last).

8

@Zimm_i48 I’ve added a short explanation to that thread on GitHub! Here it is, for the lazy people like me:

As of right now, the packages are sorted alphabetically but weighted by number of elm-conf appearances by the author.

Sorting happens here:

List.sortOn (negate . _weight . snd) (Map.toList packages)

(permalink)

I think alphabetical is implied because the Map will store packages alphabetically.

Weights are defined at the bottom of the file (permalink).

3 Likes
9

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.