# Deeper Search Index for Elm Packages

**URL:** <https://discourse.elm-lang.org/t/deeper-search-index-for-elm-packages/7967>\
**Category:** Show and Tell\
**Created:** [December 5, 2021, 3:48pm UTC](https://discourse.elm-lang.org/t/deeper-search-index-for-elm-packages/7967 "2021-12-05T15:48:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tkuriyama](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/tkuriyama/32/3930_2.png) [@tkuriyama](https://discourse.elm-lang.org/u/tkuriyama)\
**Post date:** [December 5, 2021, 3:48pm UTC](https://discourse.elm-lang.org/t/deeper-search-index-for-elm-packages/7967/1 "2021-12-05T15:48:58Z")

</div>

As an experiment and learning exercise, I spent some time this week building a deeper search index for Elm packages.

Motivation: I often find that the official Elm package index search returns no results, I suspect because it only indexes package authors, names and summaries. Here, I’ve indexed all READMEs, module documentation, as well as exposed function and type names.

[Try it out here!](https://tarokuriyama.com/elmsearch/)

If anyone is interested, I also wrote a [little blog post](https://tkuriyama.github.io/general/2021/12/03/elm-package-index.html) about the process. I had no experience with search before this project, and would be particularly glad for feedback on whether the vector space model is suitable for this kind of keyword search, or whether there are better alternatives to consider.

---

<div class="post-metadata">

**Author:** ![francescortiz](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/francescortiz/32/1721_2.png) [@francescortiz](https://discourse.elm-lang.org/u/francescortiz)\
**Post date:** [December 7, 2021, 7:52am UTC](https://discourse.elm-lang.org/t/deeper-search-index-for-elm-packages/7967/2 "2021-12-07T07:52:48Z")

</div>

Nice project, I faced the same situation a few times. I think that it can be improved because I have tried looking for i18n and it didn’t find elm-i18next. Also, you don’t need to make a crawler, you can download the source database whenever you want:

[https://package.elm-lang.org/search.json](https://package.elm-lang.org/search.json)

Thanks!

---

<div class="post-metadata">

**Author:** ![tkuriyama](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.elm-lang.org/tkuriyama/32/3930_2.png) [@tkuriyama](https://discourse.elm-lang.org/u/tkuriyama)\
**Post date:** [December 7, 2021, 12:15pm UTC](https://discourse.elm-lang.org/t/deeper-search-index-for-elm-packages/7967/3 "2021-12-07T12:15:45Z")

</div>

Hi @francescortiz thanks for your response!

As far as I can tell, the JSON you linked to only contains the package listings metadata (e.g. author, name, short description). Do you know if the READMEs and module documentation files are also available to retrieve in similar format?

`elm-i18next` is an interesting case, because the distinct string `i18n` never appears in the package documentation (so you’d have to search for `i18next`, which is not ideal). I think the lack of substring matching is a weakness of the generic vector space model I’m using for building the index, so I’ll take this as a reason to learn some other search indexing models 🙂

---

<div class="post-metadata">

**Author:** ![malaire](https://avatars.discourse-cdn.com/v4/letter/m/b782af/32.png) [@malaire](https://discourse.elm-lang.org/u/malaire)\
**Post date:** [December 8, 2021, 4:12am UTC](https://discourse.elm-lang.org/t/deeper-search-index-for-elm-packages/7967/4 "2021-12-08T04:12:06Z")

</div>

(I didn’t check the full source code of your scraper so I don’t know if you already use some of what I mention here.)

Firstly you can get list of all packages and their versions from these URLs. The since-URL is great for incremental updates as you can use the number of items you already have there and only get newer ones instead of all.

```
https://package.elm-lang.org/all-packages
https://package.elm-lang.org/all-packages/since/0

```

Package documentation can be downloaded as JSON from

```
https://package.elm-lang.org/packages/AUTHOR/PACKAGE/VERSION/docs.json

```

for example

```
https://package.elm-lang.org/packages/elm/http/2.0.0/docs.json

```

But that doesn’t seem to include README. To get README without scraping would probably require downloading it from GitHub repository. You can see the GitHub repository of package in

```
https://package.elm-lang.org/packages/AUTHOR/PACKAGE/VERSION/endpoint.json

```

for example

```
https://package.elm-lang.org/packages/elm/http/2.0.0/endpoint.json
```

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex035/uploads/elm_lang/original/1X/50a05e53677a2c3b47776d7abd0f113eb50193a1.png) [@system](https://discourse.elm-lang.org/u/system)\
**Post date:** [December 18, 2021, 4:12am UTC](https://discourse.elm-lang.org/t/deeper-search-index-for-elm-packages/7967/5 "2021-12-18T04:12:15Z")

</div>

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.