In zsh, `set -o vi` interferes with `history-search-end`

Since I've moved to zsh, the history searching using the up arrow hasn't been working. Today I've decided to sit down and investigate.

It turns out that set -o vi needs to be executed before history-search-end as shown below:

# This need to be above the history-search-end plugin
set -o vi

# History search using prefix
autoload -U history-search-end
zle -N history-beginning-search-backward-end history-search-end
zle -N history-beginning-search-forward-end history-search-end
bindkey "^[[A" history-beginning-search-backward-end
bindkey "^[[B" history-beginning-search-forward-end

I asked gemini-2.5-pro, and it says that set -o vi sets the key bindings to Vi, but history-search-end lives in the Emacs world, so these 2 can interfere with each other.

I didn't dig deeper since swapping the order of execution works.