Increase ETA update interval

[christiangnrd]

See discussion

Looking at https://buildkite.com/julialang/opencl-dot-jl/builds/1086/table?jid=019f6c40-f664-4ccd-a4e4-600ef1bf2e33&tab=output taking forever (probably #124), I noticed that the logs are currently 1.3MB purely because of status updates. This allows the update interval to be customized.

Other potential approaches are auto detect CI and set it to 5 sec. Could also add an environment variable since this is geared towards CI

[giordano]

Looking at buildkite.com/julialang/opencl-dot-jl/builds/1086/table?jid=019f6c40-f664-4ccd-a4e4-600ef1bf2e33&tab=output taking forever (probably #124)

For what is worth, I find --verbose very helpful as it provides more info, like when the job was started, the available memory, and the init time (as you know well!). All of this in my experience is useful to figure out what's going on. Setting JULIA_TEST_MAXRSS_MB=1800 gives me a decent performance. Right today I started experimenting with a dynamic mechanism to set JULIA_TEST_MAXRSS_MB

[christiangnrd]

For what is worth, I find --verbose very helpful as it provides more info

Indeed

[giordano]

More on point about the PR, I think the "problem" is that buildkite seems to have a fancy terminal which does the live update thing. GitHub Actions doesn't do that at all, so the log is quite simple and readable. Maybe that's what you want to change, rather than the update interval?

[christiangnrd]

But also I don't think this needs to be customizable. Looking at the code, this updates every second or when it needs to. I assume the every second update is just for up to date timing info, which has minute-level resolution. Updating every second is probably way overkill and could be changed to something like 5 or 10 seconds without any noticeable difference?

[christiangnrd]

Okay I'm convinced just increasing the update interval is the best approach. I made it 20 seconds since this value really only matters when no tests are starting/ending and to for the ETA (rounded to the closest minute) so every 20 seconds is plenty often IMO

[giordano]

I assume the every second update is just for up to date timing info, which has minute-level resolution.

It's also used to show the list of running jobs, no? That may change more frequently.

I'm still of the idea that you don't want Buildkite to show the live data at all, that completely messes up the log anyway.

[christiangnrd]

It's also used to show the list of running jobs, no? That may change more frequently.

The loop itself is run every 0.1 seconds, and if there are any changes to display it bypasses the 1s (on main) interval, so running tests should be handled by that path. At least they seem to be for me locally

[giordano]

I think this change is OK per se (granularity of the ETA is of ~minutes, so having an update frequency coarser than a second is reasonable), but I'm still not convinced it's the right fix for your problem (BuildKite logs being messy)

[christiangnrd]

This'll at least cut log file sizes for tests that hang by almost 20x. Maybe a way to manually disable the status update would be useful but I'll wait for someone to ask for it before adding it