[berlin] guix gc throws: "error: build daemon out of memory"

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
important
M
M
Maxim Cournoyer wrote on 21 Aug 2023 19:19
[cuirass] guix gc throws: "error: build daemon out of memory"
(name . bug-guix)(address . bug-guix@gnu.org)
87y1i3byz6.fsf@gmail.com
Hi,

Since a few days on Berlin, the following appears in /var/log/mcron.log:

Toggle snippet (3 lines)
2023-08-22 04:00:29 28215 /gnu/store/4lmpqf8dnk86avq5xa2l6bznxnmrz27x-guix-1.4.0-7.44bbfc2/bin/guix gc: guix gc: error: build daemon out of memory

It's not just a fluke, according to the kernel's dmesg:

Toggle snippet (3 lines)
[7190243.475312] __vm_enough_memory: pid: 28300, comm: guix-daemon, not enough memory for the allocation

Once it appears, it can be triggered simply by running 'guix gc'. The
problem can be worked around by restarting the Guix daemon, with:

Toggle snippet (3 lines)
herd restart guix-daemon

Any ideas what could cause such a condition?

The machine hasn't been rebooted in 83 days, perhaps we should start
with that.

--
Thanks,
Maxim
M
M
Maxim Cournoyer wrote on 26 Aug 2023 18:36
control message for bug #65446
(address . control@debbugs.gnu.org)
873505mflq.fsf@gmail.com
retitle 65446 [berlin] guix gc throws: "error: build daemon out of memory"
quit
L
L
Ludovic Courtès wrote on 2 Sep 2023 09:37
(address . control@debbugs.gnu.org)
87r0ngedkm.fsf@gnu.org
severity 65446 important
quit
L
L
Ludovic Courtès wrote on 2 Sep 2023 13:33
Re: bug#65446: [berlin] guix gc throws: "error: build daemon out of memory"
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87jzt8e2mv.fsf@gnu.org
Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (17 lines)
> Since a few days on Berlin, the following appears in /var/log/mcron.log:
>
> 2023-08-22 04:00:29 28215 /gnu/store/4lmpqf8dnk86avq5xa2l6bznxnmrz27x-guix-1.4.0-7.44bbfc2/bin/guix gc: guix gc: error: build daemon out of memory
>
>
> It's not just a fluke, according to the kernel's dmesg:
>
> [7190243.475312] __vm_enough_memory: pid: 28300, comm: guix-daemon, not enough memory for the allocation
>
>
> Once it appears, it can be triggered simply by running 'guix gc'. The
> problem can be worked around by restarting the Guix daemon, with:
>
> herd restart guix-daemon
>
> Any ideas what could cause such a condition?

No idea!

I ran “guix gc -C0” minutes ago, which reproduced the problem;
/var/log/guix-daemon.log reads this:

Toggle snippet (4 lines)
2023-09-02 21:40:31 accepted connection from pid 12853, user ludo
2023-09-02 21:40:50 unexpected build daemon error: std::bad_alloc

and the kernel writes what you wrote above:

Toggle snippet (5 lines)
Sep 2 21:40:50 localhost vmunix: [8204269.198144] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
Sep 2 21:40:50 localhost vmunix: [8204269.207396] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
Sep 2 21:40:50 localhost vmunix: [8204269.216639] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation

My reading of the Linux code is ‘__vm_enough_memory’ is used by Linux
Security Modules (LSM), so perhaps something changed in our kernel
config, be it compile-time or run-time? (Cc: Leo just in case.)

This is with Linux-libre 6.3.4-gnu.

Ludo’.
?