docker image started crashing after sync

[simplecoin-eu]

System information

Geth version: geth version latest of today, don't know how to get version, it's docker image and crashing 10 sec after start
CL client & version: e.g. lighthouse/nimbus/prysm@v1.0.0 PRYSM, also latest docker image of today
OS & Version: Windows/Linux/OSX, debian11 on top of proxmox, apt updated, docker version 24.0.1, build 6802122 RAM 64GB, 1.5TB ssd
Commit hash : (if develop) n/a

same docker-compose file works as production, images are older

Expected behaviour

with fresh run (no previous data present) chain should sync and node should run

Actual behaviour

chain synced (i think whole 770GB used including beacon) and then crasched with
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0xb78db9]

after that container continuously crashing

Steps to reproduce the behaviour

start container

Backtrace

INFO [05-22|14:53:07.971] Starting Geth on Ethereum mainnet... 
INFO [05-22|14:53:07.973] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [05-22|14:53:07.991] Maximum peer count                       ETH=50 LES=0 total=50
INFO [05-22|14:53:07.994] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [05-22|14:53:08.006] Set global gas cap                       cap=50,000,000
INFO [05-22|14:53:08.038] Initializing the KZG library             backend=gokzg
INFO [05-22|14:53:08.202] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [05-22|14:53:08.678] Using pebble as the backing database 
INFO [05-22|14:53:08.678] Allocated cache and file handles         database=/data/geth/chaindata cache=2.00GiB handles=524,288
INFO [05-22|14:53:12.754] Found legacy ancient chain path          location=/ancient-data
INFO [05-22|14:53:12.822] Opened ancient database                  database=/ancient-data        readonly=false
INFO [05-22|14:53:14.509] Initialising Ethereum protocol           network=1 dbversion=8
INFO [05-22|14:53:15.537]  
INFO [05-22|14:53:15.537] --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [05-22|14:53:15.537] Chain ID:  1 (mainnet) 
INFO [05-22|14:53:15.538] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work) 
INFO [05-22|14:53:15.538]  
INFO [05-22|14:53:15.538] Pre-Merge hard forks (block based): 
INFO [05-22|14:53:15.539]  - Homestead:                   #1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) 
INFO [05-22|14:53:15.539]  - DAO Fork:                    #1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md) 
INFO [05-22|14:53:15.539]  - Tangerine Whistle (EIP 150): #2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) 
INFO [05-22|14:53:15.539]  - Spurious Dragon/1 (EIP 155): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [05-22|14:53:15.540]  - Spurious Dragon/2 (EIP 158): #2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [05-22|14:53:15.540]  - Byzantium:                   #4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) 
INFO [05-22|14:53:15.541]  - Constantinople:              #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) 
INFO [05-22|14:53:15.541]  - Petersburg:                  #7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) 
INFO [05-22|14:53:15.541]  - Istanbul:                    #9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) 
INFO [05-22|14:53:15.541]  - Muir Glacier:                #9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) 
INFO [05-22|14:53:15.541]  - Berlin:                      #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md) 
INFO [05-22|14:53:15.541]  - London:                      #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md) 
INFO [05-22|14:53:15.541]  - Arrow Glacier:               #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md) 
INFO [05-22|14:53:15.541]  - Gray Glacier:                #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md) 
INFO [05-22|14:53:15.542]  
INFO [05-22|14:53:15.542] Merge configured: 
INFO [05-22|14:53:15.543]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md 
INFO [05-22|14:53:15.544]  - Network known to be merged: true 
INFO [05-22|14:53:15.544]  - Total terminal difficulty:  58750000000000000000000 
INFO [05-22|14:53:15.544]  
INFO [05-22|14:53:15.544] Post-Merge hard forks (timestamp based): 
INFO [05-22|14:53:15.544]  - Shanghai:                    @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md) 
INFO [05-22|14:53:15.544]  
INFO [05-22|14:53:15.545] --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [05-22|14:53:15.545]  
INFO [05-22|14:53:15.563] Loaded most recent local block           number=17,309,536 hash=a4b374..496789 td=58,750,003,716,598,352,816,469 age=20h21m4s
INFO [05-22|14:53:15.563] Loaded last fast-sync pivot marker       number=17,309,459
WARN [05-22|14:53:15.574] Head state missing, repairing            number=17,309,536 hash=a4b374..496789 snaproot=67dad8..5e161b
INFO [05-22|14:53:16.422] Loaded most recent local header          number=17,309,536 hash=a4b374..496789 td=58,750,003,716,598,352,816,469 age=20h21m5s
INFO [05-22|14:53:16.422] Loaded most recent local block           number=17,309,459 hash=e04db7..9e1e5c td=58,750,003,716,598,352,816,469 age=20h36m53s
INFO [05-22|14:53:16.422] Loaded most recent local snap block      number=17,309,536 hash=a4b374..496789 td=58,750,003,716,598,352,816,469 age=20h21m5s
INFO [05-22|14:53:16.422] Loaded last fast-sync pivot marker       number=17,309,459
WARN [05-22|14:53:16.434] Enabling snapshot recovery               chainhead=17,309,459 diskbase=17,309,528
WARN [05-22|14:53:16.455] Loaded snapshot journal                  diffs=missing
WARN [05-22|14:53:16.455] Snapshot is not continuous with chain    snaproot=67dad8..5e161b chainroot=413b64..29c747
INFO [05-22|14:53:16.457] Loaded local transaction journal         transactions=0 dropped=0
INFO [05-22|14:53:16.458] Regenerated local transaction journal    transactions=0 accounts=0
WARN [05-22|14:53:16.458] Switch sync mode from snap sync to full sync 
INFO [05-22|14:53:16.459] Resuming state snapshot generation       root=67dad8..5e161b in=008f65..483efb at=27e76c..7fd1bf accounts=455,087 slots=3,146,324 storage=263.67MiB dangling=0 elapsed=2.245ms
INFO [05-22|14:53:16.459] Upgrading chain index                    type=bloombits percentage=0
INFO [05-22|14:53:16.475] Chain post-merge, sync via beacon client 
INFO [05-22|14:53:16.480] Trie missing, state snapshotting paused  root=67dad8..5e161b in=008f65..483efb at=27e76c..7fd1bf accounts=455,087 slots=3,146,324 storage=263.67MiB dangling=0 elapsed=22.999ms
INFO [05-22|14:53:16.497] Gasprice oracle is ignoring threshold set threshold=2
WARN [05-22|14:53:16.498] Unclean shutdown detected                booted=2023-05-21T18:32:58+0000 age=20h20m18s
WARN [05-22|14:53:16.498] Engine API enabled                       protocol=eth
INFO [05-22|14:53:16.499] Starting peer-to-peer node               instance=Geth/v1.11.7-unstable-b46d37ea/linux-amd64/go1.20.4
INFO [05-22|14:53:16.588] New local node record                    seq=1,684,624,702,231 id=0033ab3c67dd8ee3 ip=127.0.0.1 udp=30303 tcp=30303
INFO [05-22|14:53:16.591] IPC endpoint opened                      url=/data/geth.ipc
ERROR[05-22|14:53:16.591] Unavailable modules in HTTP API list     unavailable="[\"db personal\"]" available="[admin debug web3 eth txpool personal miner net]"
INFO [05-22|14:53:16.591] Loaded JWT secret file                   path=/shared/jwt.hex crc32=0xf6816463
INFO [05-22|14:53:16.592] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors= vhosts=*
INFO [05-22|14:53:16.592] WebSocket enabled                        url=ws://[::]:8551
INFO [05-22|14:53:16.592] HTTP server started                      endpoint=[::]:8551 auth=true  prefix= cors=localhost vhosts=*
INFO [05-22|14:53:16.593] Started P2P networking                   self=enode://386676f0c8b8fdbbad1da11178ed8355ae6c1560205ef1b055aaf34531a02c79262b60c05a7fbce0cfeac466f18084f979244f1280be184cb0a746f540dbdbf4@127.0.0.1:30303
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0xb78db9]

goroutine 4485 [running]:
github.com/ethereum/go-ethereum/ethdb/pebble.(*batch).Write(0x8065b980c9f53d37?)
	github.com/ethereum/go-ethereum/ethdb/pebble/pebble.go:532 +0x39
github.com/ethereum/go-ethereum/core.(*BloomIndexer).Commit(0xc005732190)
	github.com/ethereum/go-ethereum/core/bloom_indexer.go:86 +0x1e8
github.com/ethereum/go-ethereum/core.(*ChainIndexer).processSection(0xc005c8e000, 0x0, {0x37, 0x3d, 0xf5, 0xc9, 0x80, 0xb9, 0x65, 0x80, ...})
	github.com/ethereum/go-ethereum/core/chain_indexer.go:413 +0x4b5
github.com/ethereum/go-ethereum/core.(*ChainIndexer).updateLoop(0xc005c8e000)
	github.com/ethereum/go-ethereum/core/chain_indexer.go:338 +0x353
created by github.com/ethereum/go-ethereum/core.NewChainIndexer
	github.com/ethereum/go-ethereum/core/chain_indexer.go:119 +0x325

When submitting logs: please submit them as text and not screenshots.

[simplecoin-eu]

Do you mean https://geth.ethereum.org ? cannot find chat there, the link shows no issues. I thing its bug and it belongs here. Similar to other

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0xb78db9]

[simplecoin-eu]

nobody else can benefit from live chat

[simplecoin-eu]

please no live chat

Agent profile image
Welcome to Ethereum, if you need help simply reply to this message, we are online and ready to help.
Name : pavel

Question : latest docker container crashing
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0xb78db9]

Agent profile image
Welcome to live chat support

i created issue https://github.com/ethereum/go-ethereum/issues/27321

Agent profile image
I’m Dave your representative. This chat is monitored and recorded for reference purpose
You can state your errors on here. And it will be rectify

i just did

Agent profile image
Very well, how long have you been having this error?

yesterday i did fresh install, startted sync
it crashed and crashing since then
full log is on github

Agent profile image
Ok, after synchronization did you complete the Geth request?

what is geth request?

Agent profile image
You need to use the Geth after the synchronization of the docker. What version did you have ?

im sure i did not, because its crashing
latest of today

Agent profile image
Ok, navigate to the homepage where you initiate the chat to click on the connect button at the top angle of the page to select your account type to establish connection for the authorization of the Geth to complete it synchronization and generate the QR. let me know when you are able to establish connection

geth is not running, cannot connect to anything what is not running
are you gpt?

Agent profile image
You have to use Geth in your docker

i cannot use it because its crashig

Agent profile image
You should establish connection with your account
Navigate to the homepage where you initiate the chat to establish connection

[holiman]

That "live chat" is a scam. Will ban the account asap. Cc @Aldekein

[Aldekein]

@holiman thanks for reporting!
simplecoin-eu has been blocked from the ethereum organization.

[PavelNiedoba]

@holiman you banned wrong account

[sidhujag]

on syscoin based on latest geth we also bumped into the exact same issue JFYI

[sidhujag]

I also get a pebble: not found on startup

Error reading unclean shutdown markers error="pebble: not found"

Later on sync:

`panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0xb9a139]

goroutine 4173 [running]:
github.com/ethereum/go-ethereum/ethdb/pebble.(*batch).Write(0xb69c4538376f6b3a?)
github.com/ethereum/go-ethereum/ethdb/pebble/pebble.go:532 +0x39
github.com/ethereum/go-ethereum/core.(*BloomIndexer).Commit(0xc0001343c0)
github.com/ethereum/go-ethereum/core/bloom_indexer.go:86 +0x1e8
github.com/ethereum/go-ethereum/core.(*ChainIndexer).processSection(0xc00029c2d0, 0x0, {0x3a, 0x6b, 0x6f, 0x37, 0x38, 0x45, 0x9c, 0xb6, ...})
github.com/ethereum/go-ethereum/core/chain_indexer.go:413 +0x4b5
github.com/ethereum/go-ethereum/core.(*ChainIndexer).updateLoop(0xc00029c2d0)
github.com/ethereum/go-ethereum/core/chain_indexer.go:338 +0x353
created by github.com/ethereum/go-ethereum/core.NewChainIndexer
github.com/ethereum/go-ethereum/core/chain_indexer.go:119 +0x325`

[backb0ne]

pull v1.11.6 tag

[PavelNiedoba]

@backb0ne v1.11.6 tag works for me
@sidhujag I'm not sure that it's same error. I see ERROR[05-22|14:53:16.591] Unavailable modules in HTTP API list unavailable="[\"db personal\"]" .... thought it's same line of code
@Aldekein unban @simplecoin-eu, it's real issue

[sidhujag]

@backb0ne v1.11.6 tag works for me @sidhujag I'm not sure that it's same error. I see ERROR[05-22|14:53:16.591] Unavailable modules in HTTP API list unavailable="[\"db personal\"]" .... thought it's same line of code @Aldekein unban @simplecoin-eu, it's real issue

Its an obvious bug, I fixed it in syscoin

[Aldekein]

@Aldekein unban @simplecoin-eu, it's real issue

User has been successfully unblocked from the organization.

[holiman]

Sorry about the confusion, wrong account got banned because the scammer deleted his history. Next time I'll be more explicit about whom to ban.