π One of the Weirdest Bugs
In n8n, some workflows suddenly stopped working. β οΈ
They were still listed, but when I tried to access them β 404 Not Found.
Yet, I could find them in the Postgres DB, so clearly the 404 was wrong. βοΈ
Digging Deeper π΅οΈ
- Only the workflows with a Gmail trigger were broken.
- Tried creating a new Gmail trigger β my credentials were missing. β οΈ
- Checked the n8n credentials list β the Gmail credential was there, but clicking it β 404.
Something was fishy π
Back to the Postgres DB:
- The credential existed.
- Searching for
id = 'abc'
β β no result. - Searching for
id LIKE '%abc%'
β β result found.
Thought: maybe a trailing whitespace?
Nope β the ID was clean.
I even compared the values in binary to make sure they are identical.
Then I asked @grok for ideas, and the 4th suggestion stood out:
π οΈ Rebuild indexes?
I have recently changed the postgres image to pgvector. But everything else worked. Could the indexes suddenly become corrupt?
The Fix β
Ran:
REINDEX TABLE credentials_entity;
Boom π₯
- The ID could be found.
- The credential worked.
- Gmail triggers functioned.
- Workflows came back to life. π
π§ Lessons Learned
- After upgrading from a Postgres image to pgvector, REINDEX might be necessary.
- If data exists but queries return 404, think corrupted indexes.
- Broken core DB functionality = weird system behavior + misleading errors.
- Always follow the rabbit hole when debugging. π³οΈπ
Question for the experts:
Why was the index corrupted? Why was it only corrupted for a single value in the entire database? How can that even happen after changing the postgres image, if the volume is the same?
PS: I actually wrote down my thought process while it was happening: https://community.n8n.io/t/suddenly-cant-open-some-workflows-they-return-404/155650