Eglot+Tree-Sitter in Emacs 29

I’ve been an Emacs user for about 15 years, and for the most part I use Emacs for org-mode and python development. I’ve happily used Jorgen Schäfer’s elpy as the core of my python development workflow for the last 5 years or so, and I’ve been happy with it. Unfortunately the current maintainer, Gaby Launay, hasn’t had time to work on elpy for over a year now. In one sense this doesn’t matter: elpy is pretty stable; it’s open source so it can’t just disappear on me; and I feel comfortable making minor changes myself.

But there are some exciting new features in core Emacs that made me wonder if I could replicate the functionality I’m used to, without relying on elpy. Specifically, I wanted to try using a language server, Eglot, for linting, code navigation, and refactoring. I also wanted to try out tree-sitter for syntax highlighting. In this post, I’ll talk about my set up and my early impressions.

Preliminaries

Going into this, I knew absolutely nothing about language servers. But I knew which elpy features I enjoyed using, and I had a vague sense of what capabilities other IDEs have (based on people questioning why I use Emacs when VS Code exists). So I hoped to achieve functionality including, sorted approximately descendingly on priority:

• Linting/Type Checking
• Code Reformatting (black)
• Running test cases
• Code navigation (jump to definition, see usages)
• Refactoring (renaming variables and functions)
• Viewing documentation
• Auto-completion
• Code templates (e.g. creating a new function)

Other features commonly available in an IDE include debugging and profiling, but tbh I’m a print-statement kind of debugger so that wasn’t high on my list.

There seem to be several python language servers in use today: python-lsp which replaces the now-unsupported python-language-server; pyright (which seems to only do type-checking?); and ruff-lsp (a language server for the ruff linter). It’s not clear to me how much the functionality of these language servers overlap. I wasn’t sure if I was supposed to pick one, or if I was supposed to use one language server for linting, another for type checking, etc. It turns out I’m supposed to pick just one, so I picked python-lsp (arbitrarily).

For Emacs, there seem to be two packages for working with language servers: eglot and lsp-mode. Eglot was merged into core Emacs in v29. lsp-mode apparently has more functionality but is less performant. I decided to start with Eglot and switch to lsp-mode if Eglot wasn’t doing it for me.

Setting up Eglot

First I compiled Emacs 29. Next I installed python-lsp:

  $ pip install "python-lsp-server[all]"

When I used elpy, I would install dev-specific packages (like black) in the same virtualenv I would use for the project. But I’ve been rethinking that. If I were using, say PyCharm, as my IDE, I wouldn’t have separate instances of PyCharm for different projects. I’d have one installation of PyCharm and perhaps project-specific configurations. I thought, why not have one installation of dev-specific packages, globally, and perhaps project-specific configurations? So I installed the language servers globally, which has been working well so far. One concern I still have here is when I run test cases on libraries with 3rd party dependencies (such as pandas). Without a global install of pandas, I wouldn’t be able to run the test cases. But python-lsp doesn’t have anything for running test cases (see Early Impressions below), so I haven’t run into this problem yet.

The documentation for python-lsp didn’t provide any guidance on how to use with eglot (or lsp-mode, or any IDE for that matter). Meanwhile, the eglot documentation didn’t provide any guidance on how to use python-lsp. It seems like the classic case of: python-lsp expects the IDE to teach the user how to use it; the IDE expects the language server to teach the user how to use it; as a result, I have no idea how to use it!

The next step was figuring out how to call python-lsp from eglot. The first obstacle I ran into was that Emacs couldn’t seem to find pylsp. This is easy to check with M-! pylsp --help, which gave a “command not found” error.

I use pyenv to manage my python versions, and when I installed the language server “globally”, it actually installed into ~/.pyenv/shims/ (which you can figure out with which pylsp). That folder is part of my PATH (as confirmed with M-x getenv PATH), but it still wasn’t working. After googling for a bit, I started to think the problem is that pyenv does weird things when initializing, which is why I have this line: eval "$(pyenv init -)" in my .zshrc. I found a suggestion to use an emacs package, exec-path-from-shell, which worked well. I added this to my init.el:

;; Fix path
(use-package exec-path-from-shell
  :ensure t
  :config
  (when (memq window-system '(mac ns x))
    (exec-path-from-shell-initialize)))

Then, running M-! pylsp --help printed the help for pylsp, as expected.

The next step was to configure eglot to start automatically when opening a python file:

(use-package eglot
  :ensure t
  :defer t
  :hook (python-mode . eglot-ensure))

So next I created a new project using poetry:

  $ poetry new wubba-wubba
  $ cd wubba-wubba && tree

 .
├──  pyproject.toml
├──  README.md
├──  tests
│  └──  __init__.py
└──  wubba_wubba
   └──  __init__.py

Then I opened wubba_wubba/__init__.py in Emacs and started typing:

There were some error indicators on the left fringe (green and orange in the picture). Mousing over one of the lines displayed the error. I fixed all the errors, resulting in this file:

"""Test eglot/pylsp functionality."""


def main() -> int:
    """Run main program."""
    return 1


if __name__ == "__main__":
    main()

Next I put my cursor on the main() declaration in the last line. To my pleasant surprise, the font of the function call changed: it became bold, I suppose to indicate what was in focus. The function definition also became bold, to guide the eye towards the definition. The docstring for the function was displayed in the minibuffer at the bottom of the screen.

Hitting M-. took me to the function definition, and M-, took me back. (In elpy, M-* is the reverse of M-., but seemingly everywhere else in Emacs, it’s M-,)

At this point, I started reviewing the Eglot documentation (within Emacs) to see what other functionality was available. I noted that the docstring presentation was provided by the ElDoc package. Errors are from Flymake (I read somewhere that Eglot doesn’t support Flycheck). Navigation is from Xref, which also supported xref-find-references via M-?. I’m digging that Eglot uses the core Emacs packages for these things. It makes me feel like a person developing in multiple languages (which I don’t really) would be able to pick up a new language easily. Imenu worked as expected, as did code completion. The documentation mentioned that if the markdown-mode package was installed, the docstring would be formatted nicely.

That’s pretty nice!

Refinements

Next I installed a few additional python-lsp add-ons:

  $ pip install pylsp-mypy pylsp-rope python-lsp-ruff

I assumed I would need to restart emacs for any of this to go into effect, so I did. I tried passing a float to a function with a type hint indicating it accepts integers, and I did indeed get an error. The pylsp-rope plugin adds additional capabilities supported by rope, that are apparently not supported in the regular python-lsp. The one that is most useful to me is organizing imports. There is also pyls-isort, but I couldn’t figure out how to use that.

Next I tried to get black working.

  $ pip install python-lsp-black

I restarted emacs and formatted the code (M-x eglot-format) but didn’t really notice any obviously-black functionality. Then I went down a rabbit hole of pylsp configuration, first verifying that I could enable flake8 in place of pyflakes and others, then replacing flake8 with python-lsp-ruff. I wound up with the following config:

(use-package eglot
  :ensure t
  :defer t
  :bind (:map eglot-mode-map
              ("C-c C-d" . eldoc)
              ("C-c C-e" . eglot-rename)
              ("C-c C-o" . python-sort-imports)
              ("C-c C-f" . eglot-format-buffer))
  :hook ((python-mode . eglot-ensure)
         (python-mode . flyspell-prog-mode)
         (python-mode . superword-mode)
         (python-mode . hs-minor-mode)
         (python-mode . (lambda () (set-fill-column 88))))
  :config
  (setq-default eglot-workspace-configuration
                '((:pylsp . (:configurationSources ["flake8"]
                             :plugins (
                                       :pycodestyle (:enabled :json-false)
                                       :mccabe (:enabled :json-false)
                                       :pyflakes (:enabled :json-false)
                                       :flake8 (:enabled :json-false
                                                :maxLineLength 88)
                                       :ruff (:enabled t
                                              :lineLength 88)
                                       :pydocstyle (:enabled t
                                                    :convention "numpy")
                                       :yapf (:enabled :json-false)
                                       :autopep8 (:enabled :json-false)
                                       :black (:enabled t
                                               :line_length 88
                                               :cache_config t)))))))

I’m still not 100% sure black is actually being run, but I guess I’ll cross that bridge when I come to it.

Setting up tree-sitter

Next I switched over to tree-sitter mode, since that was a big motivator for installing Emacs 29!

First you have to install tree-sitter support for each individual language you want to use (I think this will eventually happen automatically). Clone this repo, then run

  $ ./build.sh python

and move the resulting dynamic lib (in the dist folder) to a tree-sitter subdirectory in your emacs config folder:

  $ mkdir ~/.emacs.d/tree-sitter/
  $ mv dist/libtree-sitter-python.dylib ~/.emacs.d/tree-sitter/

Next you have to configure Emacs to use the tree-sitter mode when opening a python file:

;; Open python files in tree-sitter mode.
(add-to-list 'major-mode-remap-alist '(python-mode . python-ts-mode))

Finally, in the eglot hooks replace python-mode with python-ts-mode. And now this all seems to work! For completeness, here is the full set of code I added to my Emacs config:

;; Fix path
(use-package exec-path-from-shell
  :ensure t
  :config
  (when (memq window-system '(mac ns x))
    (exec-path-from-shell-initialize)))

;; Open python files in tree-sitter mode.
(add-to-list 'major-mode-remap-alist '(python-mode . python-ts-mode))

(use-package eglot
  :ensure t
  :defer t
  :bind (:map eglot-mode-map
              ("C-c C-d" . eldoc)
              ("C-c C-e" . eglot-rename)
              ("C-c C-o" . python-sort-imports)
              ("C-c C-f" . eglot-format-buffer))
  :hook ((python-ts-mode . eglot-ensure)
         (python-ts-mode . flyspell-prog-mode)
         (python-ts-mode . superword-mode)
         (python-ts-mode . hs-minor-mode)
         (python-ts-mode . (lambda () (set-fill-column 88))))
  :config
  (setq-default eglot-workspace-configuration
                '((:pylsp . (:configurationSources ["flake8"]
                             :plugins (
                                       :pycodestyle (:enabled :json-false)
                                       :mccabe (:enabled :json-false)
                                       :pyflakes (:enabled :json-false)
                                       :flake8 (:enabled :json-false
                                                :maxLineLength 88)
                                       :ruff (:enabled t
                                              :lineLength 88)
                                       :pydocstyle (:enabled t
                                                    :convention "numpy")
                                       :yapf (:enabled :json-false)
                                       :autopep8 (:enabled :json-false)
                                       :black (:enabled t
                                               :line_length 88
                                               :cache_config t)))))))

Early impressions

Eglot+tree-sitter gives me most of the elpy features I’ve grown to love. It’s a testament to elpy that, to be honest, I’m underwhelmed by Eglot+tree-sitter. I don’t think I can do anything extra that I wasn’t already able to do in elpy. But considering elpy doesn’t seem to be receiving any support these days, maybe just getting back to where I was is all I can ask for.

I think it’s also worth noting that Python development in Emacs has been especially well-supported (thanks to elpy) for years. Apparently this is not the case for other languages, such as Java! The good thing about Eglot+tree-sitter is that they are, in a sense, language-agnostic. You’re going to get basically the same IDE support for a variety of languages. Emacs users are no longer as reliant on language-specific maintainers. We still need language-specific servers, but those servers can be used by all IDEs, not just Emacs. And we still need IDE-specific functionality (Eglot), but that functionality can be used across all languages, not just python. So nobody needs to be an expert at Python+Emacs simultaneously. The python experts will work on the language server, and the Emacs experts will work on Eglot. This seems like a much more sustainable division of labor.

The big thing I’m still missing is the ability to easily run automated tests (e.g. pytest). Elpy made it so easy to run a single test or all tests in a file, or all tests across all files. It made test-driven development the most fluid way to develop code. I don’t want to run pytest. I want to position point on a test case, hit a keystroke, and run just that test. Elpy just worked. But for the last few years I’ve worked in large engineering orgs with infrastructure that discouraged local development in favor of running code on custom servers, or in Docker containers. So a more customized approach to running tests is warranted. And I’d love to use transient, the user interface of magit, to quickly select command line arguments (such as --last-failed) for pytest. So maybe this is just the motivation I need to create my own testing library.

I still have some open questions I may resolve in the near future:

• Is pylsp the best python language server in 2023? Pyright and ruff-lsp are both worth considering.
• What does lsp-mode offer to justify deviating from core Emacs?
• What new functionality can I use, that wasn’t available in elpy?
• Should I keep using a global language server install, or should I install the language server in the same virtual environment I use for a project?