Version 1.18 crashes for me due to mailer.HOST config

[nunoperalta]

Description

Hello,

In a loop, 1.18 keeps crashing forever:

Server listening on :: port 22.
Server listening on 0.0.0.0 port 22.
2022/12/30 05:11:07 cmd/web.go:106:runWeb() [I] Starting Gitea on PID: 15
2022/12/30 05:11:07 ...s/setting/setting.go:613:deprecatedSetting() [E] Deprecated fallback `[server]` `LFS_CONTENT_PATH` present. Use `[lfs]` `PATH` instead. This fallback will be removed in v1.19.0
2022/12/30 05:11:07 cmd/web.go:160:runWeb() [I] Global init
2022/12/30 05:11:07 ...s/setting/setting.go:613:deprecatedSetting() [E] Deprecated fallback `[server]` `LFS_CONTENT_PATH` present. Use `[lfs]` `PATH` instead. This fallback will be removed in v1.19.0
2022/12/30 05:11:07 routers/init.go:116:GlobalInitInstalled() [I] Git Version: 2.36.3, Wire Protocol Version 2 Enabled (home: /data/gitea/home)
2022/12/30 05:11:07 routers/init.go:117:GlobalInitInstalled() [I] AppPath: /usr/local/bin/gitea
2022/12/30 05:11:07 routers/init.go:118:GlobalInitInstalled() [I] AppWorkPath: /app/gitea
2022/12/30 05:11:07 routers/init.go:119:GlobalInitInstalled() [I] Custom path: /data/gitea
2022/12/30 05:11:07 routers/init.go:120:GlobalInitInstalled() [I] Log path: /data/gitea/log
2022/12/30 05:11:07 routers/init.go:121:GlobalInitInstalled() [I] Configuration file: /data/gitea/conf/app.ini
2022/12/30 05:11:07 routers/init.go:122:GlobalInitInstalled() [I] Run Mode: Prod
Received signal 15; terminating.

Once I downgrade to 1.17, all is working again.

I'm not sure what to do, or how to investigate further.

MySQL is running fine:

2022-12-30T05:10:29.580620Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.31'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.

Also, I'm not sure how to fix the LFS_CONTENT_PATH issue.
What I currently have is:

LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/git/lfs
LFS_JWT_SECRET   = ...

Do I just move the LFS_CONTENT_PATH like this?

[lfs]
PATH = /data/git/lfs

and leave the other LFS_* as they are?

Thank you very much.

Gitea Version

1.18

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

Bundled in Docker

Operating System

Bundled in Docker

How are you running Gitea?

services:
  server:
    container_name: gitea_server
    image: gitea/gitea:latest
    environment:
      - USER_UID=...
      - USER_GID=...
      - DB_TYPE=mysql
      - DB_HOST=db:3306
      - RUN_MODE=prod
      - ROOT_URL="https://..../"
    env_file:
      - ./db.env
    restart: always
    networks:
      - gitea
    volumes:
      - /.../data:/data
    ports:
      - "127.0.0.1:...:3000"
    depends_on:
      - db
  db:
    container_name: gitea_db
    image: mysql:8.0
    restart: always
    env_file:
      - ./db.env
    networks:
      - gitea
    volumes:
      - /.../mysql:/var/lib/mysql
    cap_add:
      # mbind: Operation not permitted
      - SYS_NICE

networks:
  gitea:
    external: false

Database

MySQL

[lunny]

I tested in my macOS with gitea/gitea:latest which is point to 1.18.0 currently. It's work.

docker pull gitea/gitea:latest
docker run -p 3000:3000 gitea/gitea:latest

You can add the new section in your app.ini and old content could be kept and be removed after this work.

[lfs]
PATH = /data/git/lfs

[rafipiccolo]

same bug for me, but i cant revert to version 1.17
because i now have this error :

Server listening on :: port 22.
Server listening on 0.0.0.0 port 22.
Your database (migration version: 231) is for a newer Gitea, you can not use the newer database for this old Gitea release (224).
Gitea will exit to keep your database safe and unchanged. Please use the correct Gitea release, do not change the migration version manually (incorrect manual operation may lose data).
Received signal 15; terminating.

stuck as hell :) I'm considering restoring a backup or waiting for patch.

[techknowlogick]

signal 15 is SIGTERM, which means Gitea isn't exiting something else is telling it to exit. Are you able to provide "trace logs" for what is happening prior to exit?

[rafipiccolo]

how would you activate "trace logs" for gitea with docker ?

the thing is, the containers dies right after start so i believe it is killed from inside the image. (same second)

the only mecanism that could kill the container from outside that i know of is with docker swarm : where a faulty healthcheck would kill the unhealthy container (also it would leave it about 1:30 to reach healtyness before sending a sigterm)

[nunoperalta]

It does take a few seconds to terminate, though.

How do I do a "trace logs"? I'm not finding anything about "docker trace logs". The only thing I find is the normal docker logs which I've already provided on the original post above.

btw, I also had a look at the mailer breaking changes, but I believe there's nothing to be changed on my side:

[mailer]
ENABLED = true
HOST    = ...
FROM    = ...
USER    = ...
PASSWD  = ...

[wolfogre]

@nunoperalta @rafipiccolo Could you please just set the log level to Trace and restart the Gitea to see what it outputs?

Update the conf file like:

[log]
LEVEL = Trace

I tried to figure it out but didn't find anything. I believe more logs would be helpful.

[rafipiccolo]

thanks for your comment.
i noticed a fresh install with version 1.18 works just fine.

from what i see, it is indeed a problem with a previous configuration.
ps: i never changed configs, they are all autogenerated by gitea itself
therefore i would expect gitea to update it automatically to not crash upon updates.

you comment led me to this file : /gitea/conf/app.ini

[log]
ROOT_PATH = /data/gitea/log
MODE      = file
LEVEL     = trace

for me the logs stay the same as nunoperalta

but i see this in /gitea/log/gitea.log

2022/12/30 10:30:55 ...dules/setting/log.go:335:newLogService() [I] Gitea Log Mode: File(File:trace)
2022/12/30 10:30:55 ...dules/setting/log.go:249:generateNamedLogger() [I] Router Log: Console(console:trace)
2022/12/30 10:30:55 ...les/setting/cache.go:76:newCacheService() [I] Cache Service Enabled
2022/12/30 10:30:55 ...les/setting/cache.go:91:newCacheService() [I] Last Commit Cache Service Enabled
2022/12/30 10:30:55 ...s/setting/session.go:73:newSessionService() [I] Session Service Enabled
2022/12/30 10:30:55 ...s/setting/setting.go:613:deprecatedSetting() [E] Deprecated fallback `[mailer]` `HOST` present. Use `[mailer]` `SMTP_ADDR` instead. This fallback will be removed in v1.19.0
2022/12/30 10:30:55 ...es/setting/mailer.go:75:newMailService() [F] Invalid mailer.HOST (in-v3.mailjet.com): address in-v3.mailjet.com: missing port in address

[rafipiccolo]

OK, i happened to edit the file, or configure a mail server via the UX. It was a long time ago, I don't remember.
i needed to replace HOST with SMTP_ADDR, and add SMTP_PORT

which does it :

[mailer]
ENABLED   = true
SMTP_ADDR = in-v3.mailjet.com
SMTP_PORT = 587
FROM      = xxx@xxx.fr
USER      = xxx
PASSWD    = xxxx

now it work again.

please dont break gitea for a mail configuration :)

[nunoperalta]

Thanks.

Strange... I don't see any extra logs since I put LEVEL = trace :(

Server listening on :: port 22.
Server listening on 0.0.0.0 port 22.
2022/12/30 10:39:58 cmd/web.go:106:runWeb() [I] Starting Gitea on PID: 15
2022/12/30 10:39:58 cmd/web.go:160:runWeb() [I] Global init
2022/12/30 10:39:58 routers/init.go:116:GlobalInitInstalled() [I] Git Version: 2.36.3, Wire Protocol Version 2 Enabled (home: /data/gitea/home)
2022/12/30 10:39:58 routers/init.go:117:GlobalInitInstalled() [I] AppPath: /usr/local/bin/gitea
2022/12/30 10:39:58 routers/init.go:118:GlobalInitInstalled() [I] AppWorkPath: /app/gitea
2022/12/30 10:39:58 routers/init.go:119:GlobalInitInstalled() [I] Custom path: /data/gitea
2022/12/30 10:39:58 routers/init.go:120:GlobalInitInstalled() [I] Log path: /data/gitea/log
2022/12/30 10:39:58 routers/init.go:121:GlobalInitInstalled() [I] Configuration file: /data/gitea/conf/app.ini
2022/12/30 10:39:58 routers/init.go:122:GlobalInitInstalled() [I] Run Mode: Prod
Received signal 15; terminating.

[nunoperalta]

Ah, @rafipiccolo - seems it was what fixed it for me too!

I did look at the Breaking Changes notice, and somehow I did NOT understand what was said there 😅🤦🏼‍♂️

The combined mailer.HOST option has been deprecated in favor of the new mailer.SMTP_ADDR and mailer.SMTP_PORT options.

I guess it's my fault.
Would be nice if the logs said that, the same way they complained about LFS :)

[rafipiccolo]

my bad too, i'll read it next time,
But i didn't expect a breaking change on a minor release.

[nunoperalta]

Yeah exactly. Especially when it says "deprecated" and not "removed".

[wolfogre]

Strange... I don't see any extra logs since I put LEVEL = trace :(

Server listening on :: port 22.
Server listening on 0.0.0.0 port 22.
2022/12/30 10:39:58 cmd/web.go:106:runWeb() [I] Starting Gitea on PID: 15
2022/12/30 10:39:58 cmd/web.go:160:runWeb() [I] Global init
2022/12/30 10:39:58 routers/init.go:116:GlobalInitInstalled() [I] Git Version: 2.36.3, Wire Protocol Version 2 Enabled (home: /data/gitea/home)
2022/12/30 10:39:58 routers/init.go:117:GlobalInitInstalled() [I] AppPath: /usr/local/bin/gitea
2022/12/30 10:39:58 routers/init.go:118:GlobalInitInstalled() [I] AppWorkPath: /app/gitea
2022/12/30 10:39:58 routers/init.go:119:GlobalInitInstalled() [I] Custom path: /data/gitea
2022/12/30 10:39:58 routers/init.go:120:GlobalInitInstalled() [I] Log path: /data/gitea/log
2022/12/30 10:39:58 routers/init.go:121:GlobalInitInstalled() [I] Configuration file: /data/gitea/conf/app.ini
2022/12/30 10:39:58 routers/init.go:122:GlobalInitInstalled() [I] Run Mode: Prod
Received signal 15; terminating.

@nunoperalta

Let me guess, your Gitea config is:

[log]
MODE      = file

So most logs will output to the file, but you check logs from console with docker logs, so you can only see very few logs that have been printed to console at the very beginning.

If I'm right, you can find the error logs in the file. It's not your fault, I think Gitea can do better.