builtin:git-download capability detection not working for the bordeaux build farm

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Christopher Baines
  • Simon Tournier
Owner
unassigned
Submitted by
Christopher Baines
Severity
important
C
C
Christopher Baines wrote on 17 Nov 2023 13:39
(address . bug-guix@gnu.org)
87pm08f41q.fsf@cbaines.net
The bordeaux build farm depends on computing the derivations on one
machine, then potentially building them on a different machine.

Some of the build machines don't have a new enough guix-daemon that
understands builtin:git-download, so derivations that use this are
sometimes failing (e.g. [1])


One potential approach to address this is somehow have the data service
indicate that it wants compatible derivations when computing them,
rather than to have guix do feature detection against the guix-daemon
that is being used at the point the derivations are being computed.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmVX3sFfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xd58g/+IBe+dYxUo+IkUVhSmXhtTBsb5gkYDnpa
rbFGAmUOMe99Yqtv2iz0RgV/aO2a1C/BxW3RVJXdEVUGFmM9MxqpOph2TLQ2s7Om
mPGbgzjEtJyoVWItAnD5x9K9WCC7ln1U+SpscZurv33geq8fmB4iTTWhpmBjt+U8
A8YBqoSKYgjgiWKi4T/J1e/Bcn7SbZ0RnSSW7OGLBwoqEAzu4qa/NCUGidMsHZ/0
xn3sUXd/iOqd9+HfYgl7c7/wrDVBqF6Fsp+GEzR3OKoXLha6ZFRxSTOrUpSsHZ/W
z9Q3/3zuGV5a8faNnZWzDaYMP6kNAYYrBBSqx58zS8xI74JpT5i8lG28jocfPmUG
k9JWXjAcM1etFEPI0Z70Mf3UUQi+0ng+oVbZc4DSryJPfZbqYb5wQlv3gytk5tYm
xYNtfZesLqRcuPWUlk3XvEIengQbWlABBk15RVDCMA1SFytIdiehMVrf1CopcwOC
g1+hS9QEK++mEZOoTxs8fPDtzT1F2G+Kot6TQjhFro8pwBHA+ObmQpTNDH9/Ss9I
VgCAcsqRYOF1jyAVDwvmDv4pCeLIuaBZA0FVqCUzC1vcsfJKkAY81Oc7momq2+5d
iX0mLBdopi5SaHkbovEz8T1ph1jUA76qhbuO+2mzdfYwBsluzDOpmxzg3dEtWT7w
9mSQ5lZ8dow=
=DeUn
-----END PGP SIGNATURE-----

S
S
Simon Tournier wrote on 20 Nov 2023 01:59
86zfz8219i.fsf@gmail.com
Hi Chris,

On Fri, 17 Nov 2023 at 21:39, Christopher Baines <mail@cbaines.net> wrote:

Toggle quote (7 lines)
> The bordeaux build farm depends on computing the derivations on one
> machine, then potentially building them on a different machine.
>
> Some of the build machines don't have a new enough guix-daemon that
> understands builtin:git-download, so derivations that use this are
> sometimes failing (e.g. [1])

Do you mean that the drv file is generated using a new daemon then this
drv file is “offloaded” to another machine using an older daemon and
thus fails to build it?


Toggle quote (5 lines)
> One potential approach to address this is somehow have the data service
> indicate that it wants compatible derivations when computing them,
> rather than to have guix do feature detection against the guix-daemon
> that is being used at the point the derivations are being computed.

Other solutions are: downgrade the daemon of one of the machine or
update the daemon of other machines, no?

Why would the latter not be possible?

Cheers
simon
C
C
Christopher Baines wrote on 20 Nov 2023 02:29
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 67250@debbugs.gnu.org)
87cyw4g1hd.fsf@cbaines.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (15 lines)
> Hi Chris,
>
> On Fri, 17 Nov 2023 at 21:39, Christopher Baines <mail@cbaines.net> wrote:
>
>> The bordeaux build farm depends on computing the derivations on one
>> machine, then potentially building them on a different machine.
>>
>> Some of the build machines don't have a new enough guix-daemon that
>> understands builtin:git-download, so derivations that use this are
>> sometimes failing (e.g. [1])
>
> Do you mean that the drv file is generated using a new daemon then this
> drv file is “offloaded” to another machine using an older daemon and
> thus fails to build it?

Roughly, yep.

Toggle quote (10 lines)
>> One potential approach to address this is somehow have the data service
>> indicate that it wants compatible derivations when computing them,
>> rather than to have guix do feature detection against the guix-daemon
>> that is being used at the point the derivations are being computed.
>
> Other solutions are: downgrade the daemon of one of the machine or
> update the daemon of other machines, no?
>
> Why would the latter not be possible?

It's possible, just difficult. Because with the current guix-daemon, the
build coordinator can only build things that can be GC'd, some machines
deliberately run older daemons to allow building newer things without
hitting this issue.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmVbNX5fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XfSyg//Yhf+0cVF3Qw626oFC1NZPqR6jZc2pYvV
Wei5tO5SKUn47BeO6T3wz9kCQSuZoV3IM+ShpLKzp5jmLz2y/r7HH78T6EPR5hK6
PqSgUCV06mua4sAg/EiHADgo1lsM8koPiUo3QvkvqlFxgruG+QV4icoFYVq1EEMH
PUD+10hgnYIMhr+Y8CiWwF+EnboOAbHAa9TAMghkkkhLXcjG8smG7JLt0JarhIzj
mTMNq77s2BJ6NIgPRXWyagyxbbOroorDtsilYKB1ODQg8MRt6O4FcNJZlVIdctRS
ZedqHtM7OzKJGq39xU4RmSXs7RW1sBVBS/WBKVfuFSoWJWg9KGXrePdt7zm/feJ3
p/GdFRon2y0OTXusMrGB2Wf4H2vGn/CqWfnyAexJcvA7jHgY8AYLwfJLoT17thJD
b74n9AhCc5Zllg9Oggsbg4o1lJJOLcqbv21ExWYBNtUcW4DZukgJUkTk3hN596Ws
CtRsxMoO3EznvFi+7Y/K3PbGFtqmr/zTfluKhuP6Qr4xKv92Y9xZ5AakTHRb9klq
gF7KNr/cKwBSbRBYEKvC/hBHEfgSQYklFU16GKMts7yxPlAC44CQfkCKck1tsNVo
vTfKDU8qCWX+Xoco1Md3xxgJxRlo5wTVtYvXK7uexAXD49lMqKVXxdhoHakHZG4Y
BDdn5vO6Kb0=
=injp
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 22 Nov 2023 02:19
(name . Christopher Baines)(address . mail@cbaines.net)(address . 67250@debbugs.gnu.org)
87bkbmm6o1.fsf@gnu.org
Hi,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (14 lines)
> The bordeaux build farm depends on computing the derivations on one
> machine, then potentially building them on a different machine.
>
> Some of the build machines don't have a new enough guix-daemon that
> understands builtin:git-download, so derivations that use this are
> sometimes failing (e.g. [1])
>
> 1: https://bordeaux.guix.gnu.org/build/10cc5622-6b1f-4f28-ad9a-41cf796d7a15
>
> One potential approach to address this is somehow have the data service
> indicate that it wants compatible derivations when computing them,
> rather than to have guix do feature detection against the guix-daemon
> that is being used at the point the derivations are being computed.

Would it work for you if we added a keyword argument to
‘open-connection’ in (guix store) that would let you select which
builtins to use?

As in:

(open-connection
#:assume-available-builtin-builders '("download"))

Ludo’.
L
L
Ludovic Courtès wrote on 22 Nov 2023 02:19
control message for bug #67250
(address . control@debbugs.gnu.org)
87a5r6m6np.fsf@gnu.org
severity 67250 important
quit
S
S
Simon Tournier wrote on 28 Nov 2023 06:10
Re: bug#67250: builtin:git-download capability detection not working for the bordeaux build farm
(address . 67250@debbugs.gnu.org)
87msuy9dem.fsf@gmail.com
Hi,

On Wed, 22 Nov 2023 at 11:19, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (5 lines)
> As in:
>
> (open-connection
> #:assume-available-builtin-builders '("download"))

Instead, why not check in ’git-fetch’? Currently, the test is done
against the local daemon, right?

Toggle snippet (17 lines)
(define* (git-fetch ref hash-algo hash
#:optional name
#:key (system (%current-system))
guile git)
"Return a fixed-output derivation that fetches REF, a <git-reference>
object. The output is expected to have recursive hash HASH of type
HASH-ALGO (a symbol). Use NAME as the file name, or a generic name if #f."
(mlet %store-monad ((builtins (built-in-builders*)))
(if (member "git-download" builtins)
(git-fetch/built-in ref hash-algo hash name
#:system system)
(git-fetch/in-band ref hash-algo hash name
#:system system
#:guile guile
#:git git))))

For example, why not a test as:

(if (and SOMETHING
(member "git-download" builtins))

And this SOMETHING could be a global variable, as
%assume-builtin-builder, set to true by default and turn to false with
an environment variable, as GUIX_ASSUME_BUILTIN.


Cheers,
simon
C
C
Christopher Baines wrote on 28 Nov 2023 08:54
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 67250@debbugs.gnu.org)
87y1ehlsup.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (27 lines)
> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> The bordeaux build farm depends on computing the derivations on one
>> machine, then potentially building them on a different machine.
>>
>> Some of the build machines don't have a new enough guix-daemon that
>> understands builtin:git-download, so derivations that use this are
>> sometimes failing (e.g. [1])
>>
>> 1: https://bordeaux.guix.gnu.org/build/10cc5622-6b1f-4f28-ad9a-41cf796d7a15
>>
>> One potential approach to address this is somehow have the data service
>> indicate that it wants compatible derivations when computing them,
>> rather than to have guix do feature detection against the guix-daemon
>> that is being used at the point the derivations are being computed.
>
> Would it work for you if we added a keyword argument to
> ‘open-connection’ in (guix store) that would let you select which
> builtins to use?
>
> As in:
>
> (open-connection
> #:assume-available-builtin-builders '("download"))

Yep, that would at least allow freezing the available builtins going
forward and separating it from the running daemon.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmVmG65fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XfwYQ/+IUTIhIaLgxttyy41q7Us57Ellh1bOIeB
yiceqRqT12w1QvT0G8oazuvc8ZBdr3BLENEvcAqL01dDX0JevcSePuOMSQO8tEtk
0Rxy+iJ6ozdzjcsv9wXGHbcucwdfmhlT6yUVvkIhPIFxL29Dp8GqLlrNfWRMbl3n
1u6MQKgzTDq6yq0gK/2Y3cfoBJUvHuWtX7z/lF4+if7EeEQRolIaqmGt2xRr1sYu
5r8mheyXBdr0Ri2VyxWInQ7PM4WhGsYSRnp86fL3yUEsRcrtG/mAUQRgHv8hPlTy
KiQQICISDK7XYIFYjG+Dci2XqBp9qqGGiam9rLFqucepO9+bwSuq5WdxxmEeUG0m
Y3GGshyjlNWb0B9mvlcz7eRipHvSKHi2s/mbgy3ljypyaJuAO3d+w65B91n/gEXw
IN9GWXt4UF+mtiYYVztlFY+/+sB2NJ4KGh8qzuJKEJanlS2V5Qs6XVtP1P6LZ6ej
UPaL3fAgMBsDTE8L2xOgUKnS8qfC6ogQHmjellUmnjz4siss3veltcb13lEv/1dI
nwcrQsCXwv1MnYBXzorN7gu4VP5AUKwYpjG/zKMhLWXRzkK4HG601FKYAutraOYv
xRviG3ZoK8otcvS+qS4OzVu8vtRzh4TihJNEV0qyrsJqjOJe+SDIs+Y2q1Xp6kjn
1fZVGYJJVIc=
=MQR4
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote 7 days ago
(address . 67250@debbugs.gnu.org)
87pltjb6fk.fsf@cbaines.net
Christopher Baines <mail@cbaines.net> writes:

Toggle quote (32 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi,
>>
>> Christopher Baines <mail@cbaines.net> skribis:
>>
>>> The bordeaux build farm depends on computing the derivations on one
>>> machine, then potentially building them on a different machine.
>>>
>>> Some of the build machines don't have a new enough guix-daemon that
>>> understands builtin:git-download, so derivations that use this are
>>> sometimes failing (e.g. [1])
>>>
>>> 1: https://bordeaux.guix.gnu.org/build/10cc5622-6b1f-4f28-ad9a-41cf796d7a15
>>>
>>> One potential approach to address this is somehow have the data service
>>> indicate that it wants compatible derivations when computing them,
>>> rather than to have guix do feature detection against the guix-daemon
>>> that is being used at the point the derivations are being computed.
>>
>> Would it work for you if we added a keyword argument to
>> ‘open-connection’ in (guix store) that would let you select which
>> builtins to use?
>>
>> As in:
>>
>> (open-connection
>> #:assume-available-builtin-builders '("download"))
>
> Yep, that would at least allow freezing the available builtins going
> forward and separating it from the running daemon.

I've now sent some patches which add this option to #71038.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmZIq69fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdyAg/+IrxdTR5jNvjbENQbYbde48p0pQnRspQx
sTcPBZatt0gfzKI0wyxcDotnxB6K1qv6KjLHVHW1Lusky3WlF385YmSu54XnOQkY
N7Ki5KsN8FBuUX2O8+g/YQt4T7eUdVRQKNRknur1FjyTaF/bKcy9XbbHcYjm+Ais
gtbdaNR0syiur2TFKcXYV7Ns7+zAlVoQ+TCrVoQi/HhBWvqsK8KZ+jxcORFDosiW
a+kpPASR96CupgPfF1mv3KtHkeFxehRRtY5D0/tCSvxV10S1V4he/ntwPBXOP5Iv
ZMcdDhsL9XH2y6oFD6uZrk/FwHx112M3MaopfwVLLgvaIdyjlr1yz/+bSwPPSVNf
B5k5kObRHr1XtiNKJTREvkF5uj8De/cFvS4bjkSeNvGzUwuvHjJ9h9+6ZwK6fapJ
kIJ+JC1aNbtAFUbpCaoYC98wHALmYRjG9isNxjJoXCwRDTdNm5NLrLe2c0dXLm/R
JMEPXsxPReyKSoJZq5/8QrtUMGsOQOd81CjFpLhoh+Hx6jGg9S8oNslk6nY9hNKs
HIKX0iOGV01n2mkr2nrsncwk65lq6bjjPKbx/adiZDNSnEANdhpD0eucTbNj2Mmq
hWdita0M4yvKXEZbgr9ospSLaFtclZRsxQeDRpcj9y2NMmQXlPUYzy+RPmvL8fki
xlEfJXW6U14=
=u9nJ
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote 7 days ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 67250@debbugs.gnu.org)
87jzjrb60a.fsf@cbaines.net
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (37 lines)
> On Wed, 22 Nov 2023 at 11:19, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> As in:
>>
>> (open-connection
>> #:assume-available-builtin-builders '("download"))
>
> Instead, why not check in ’git-fetch’? Currently, the test is done
> against the local daemon, right?
>
> --8<---------------cut here---------------start------------->8---
> (define* (git-fetch ref hash-algo hash
> #:optional name
> #:key (system (%current-system))
> guile git)
> "Return a fixed-output derivation that fetches REF, a <git-reference>
> object. The output is expected to have recursive hash HASH of type
> HASH-ALGO (a symbol). Use NAME as the file name, or a generic name if #f."
> (mlet %store-monad ((builtins (built-in-builders*)))
> (if (member "git-download" builtins)
> (git-fetch/built-in ref hash-algo hash name
> #:system system)
> (git-fetch/in-band ref hash-algo hash name
> #:system system
> #:guile guile
> #:git git))))
> --8<---------------cut here---------------end--------------->8---
>
> For example, why not a test as:
>
> (if (and SOMETHING
> (member "git-download" builtins))
>
> And this SOMETHING could be a global variable, as
> %assume-builtin-builder, set to true by default and turn to false with
> an environment variable, as GUIX_ASSUME_BUILTIN.

I can think of a couple of reasons not to do this via an environment
variable.

Having the logic in git-fetch via an environment variable would mean
that if other builtins are added, the code generating derivations using
them would also have to check the environment variable. Specifying the
builtin's that are available on a store connection would work
automatically in the case of new builtins being added..

There's also trade-offs in control when using environment variables, or
deciding when to take the value from an environment variable and switch
to passing the value around through arguments or other means. Consider
the case where you have a single process which has multiple store
connections open and it wants to specify different available builtin's
on particular store connections. This would become quite difficult when
using an environment variable, but work easily when specifying the
builtins available on the store connection.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmZIrdVfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdqjA//b4o7EtVAvNwn74bDVPz+ol0tR+YxAvnd
yDVROXMS8Bd+KXobt+bjb3Bfp2bKi0XIUq9SaHYTJDJryVqEjBbgVyKSf1tGcWTh
yLCakfYs2Aqb+G30Q0y4/kGDCTlNEB0vQs7NXRgsyZNrpiowNsux7k1aXpgd0Mbg
Hz/oiY8tIDtpUXHQIZ98oxli9lv1FXd1GMLQUEb8dSBE/VP1tCjGxX0+JIgKMaCo
J2zZBy7pInE0z//7yjq95YTu00tfBlaMgg+COQYAJ+ilcLBsLcdIHGQiRm85Rtx3
AQn32qGcR/vFcl6I0bJotVSo5X5hbu4+pDKPDlztvMa5t4MfdlY9/NF8vZ0VdKdG
SBRZoc+7JZfGB5oDbUyL+RoL73D8R/byX4Qc3v/oq5IRCpLJiUu915ZS09mhDR1x
kCOJ5vV2JHv0NFAx8goQYplocN+CrqJ0stp5lFEbqRNKEPjKpFP3UnaxUCE6ncUc
uUYG8B9IF3BQHNgSTdi9PjKaD3sCyRURiiM1b0tvzM8DQJycjHWPkGve34lk1Z/9
CYq7q2oyrLQKKJ0cMGFN18ZQbEu2Mb+E7vyBxxYK2ZgIGBRBNsJCj4QnBJDhTrY6
f3bdHqvL/Ew5+PI91T9E5NRDn/8g/9eYiKGJIY42St+HhOc19JeC5qYFy6MoLie9
ZUy+SFnqAM8=
=owCS
-----END PGP SIGNATURE-----

?