14:31:57 #startmeeting Pulp Triage 2021-05-25 14:31:57 #info daviddavis has joined triage 14:31:57 !start 14:31:57 Meeting started Tue May 25 14:31:57 2021 UTC. The chair is daviddavis. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:31:57 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:31:57 The meeting name has been set to 'pulp_triage_2021-05-25' 14:31:57 daviddavis: daviddavis has joined triage 14:32:09 open floor https://hackmd.io/SVCMjpwXTfOMqF2OeyyLRw 14:32:12 #info ipanova has joined triage 14:32:12 !here 14:32:12 ipanova: ipanova has joined triage 14:32:17 #info ttereshc has joined triage 14:32:17 !here 14:32:17 ttereshc: ttereshc has joined triage 14:32:18 #info ppicka_ has joined triage 14:32:18 !here 14:32:18 ppicka_: ppicka_ has joined triage 14:32:24 #info mcorr has joined triage 14:32:24 !here 14:32:24 mcorr: mcorr has joined triage 14:32:25 #info ggainey[m] has joined triage 14:32:25 !here 14:32:26 ggainey[m]: ggainey[m] has joined triage 14:32:27 Any incompatible releases for upcoming 3.13 left? (except pulp_python which can be released only after 3.13 is out) 14:32:30 #info dalley has joined triage 14:32:30 !here 14:32:30 dalley: dalley has joined triage 14:32:36 #info gerrod has joined triage 14:32:36 !here 14:32:36 gerrod: gerrod has joined triage 14:32:41 sorry, I got distracted discussing a PR with Bruno 14:32:46 #info fao89 has joined triage 14:32:46 !here 14:32:46 fao89: fao89 has joined triage 14:33:20 #info dkliban has joined triage 14:33:20 !here 14:33:20 dkliban: dkliban has joined triage 14:33:39 I'm not aware of any, pulp_rpm is out for sure 14:34:21 #info bmbouter has joined triage 14:34:21 !here 14:34:21 bmbouter: bmbouter has joined triage 14:34:21 pulp_ansible and pulp_maven 14:34:58 i believe pulp_ansible and pulp_maven still need to be released for compatibility with 3.13 14:35:43 dkliban: do you think your release automation will be ready? 14:36:06 i am trying to fully finish it this afternoon 14:36:16 I didn't release pulp-ansible, so you could have a plugin to test 14:36:19 dkliban++ 14:36:19 ggainey[m]: dkliban's karma is now 594 14:36:33 dkliban++ 14:36:33 fao89: dkliban's karma is now 595 14:37:17 those releases are not blocking pulpcore one, right? just oci images can't be pushed right away 14:37:24 that's right 14:37:28 yep 14:37:32 great 14:38:00 next topic? 14:38:04 yes 14:38:06 but if the release automation script is successful with pulp-ansible and pulp-maven, may be worth to use on pulpcore 14:38:50 Topic: Does it make sense to start doing regular Z releases (for the latest Y stream) and not wait with bugfixes for the next Y release. 14:39:13 I think it may if we had it fully automated 14:39:18 I added this one, mostly based on user's struggles and expectations 14:39:23 I have an example 14:39:25 as it is now though I don't think we can afford it 14:39:28 I think this is kind of chicken & egg with release automation 14:39:49 yeah, if release-auto is solid, I could see us just auto-backporting anything that doesn't need a migration every week or two 14:39:54 this was backported to 3.11 but never released in 3.12 and is waiting on 3.13 https://github.com/pulp/pulpcore/commit/52bb2ab94be24652ca12ad27daa56328832fc8a7 14:40:06 that's not good 14:40:10 ouch 14:40:14 a user upgraded from 3.11 with a fix to the latest - 3.12.z 14:40:22 and hits the issue 14:40:26 it isn't but also ... things in open source aren't fixed because someone out there hasn't fixed it yet 14:40:28 it is very unexpected 14:40:44 i agree that it's unexpected 14:40:51 me too 14:40:53 ttereshc: i agree 14:40:59 upgrade tests will help with that 14:41:11 the user is our great wibbit1 who was going live withpulp that week and had to pacth their system 14:41:37 yup and users had to downgrade django also on the list 14:41:40 I would think, at a minimum, anything backported to older releases (3.7, 3.11) need to automatically go into latest-released-version, to avoid this exact issue 14:41:57 I just want to raise the awareness and increase the priority to produce Z releases for the latest Y 14:42:03 yeah ... but we need to have better automation in place. 14:42:08 +1 14:42:13 so let's come back to this topic again next week 14:42:24 agreed 14:42:25 heh, sounds like a fine plan :) 14:42:30 thank you 14:42:44 I think we should continue to invest in release automation over almost all things until it's literally click-to-release 14:42:50 +100 14:43:17 Topic: libera.chat and Project registering 14:43:29 fun 14:43:32 I think mcorr submitted an application to register pulp on libera 14:43:35 I sent an email to express our interest in registering Pulp 14:43:37 snap 14:43:40 hehe 14:43:58 daviddavis: you have a doc answering the project-questions yeah? 14:44:13 What do people think about matrix as primary with a Libera bridge? 14:44:30 I would really like to see us off of freenode as soon as there's a matrix-bridge to libera - we have a growing matrix community 14:44:34 I'm really hoping we use matrix as primary 14:44:48 matrix++ 14:44:48 fao89: matrix's karma is now 1 14:44:53 mcorr: that's exactly what I'm experimenting with, for me personally 14:44:55 and I'm also not sure we need a secondary (just saying) 14:45:01 There's this beta called Matrix Spaces that is an evolution on from Community and it looks very nice. 14:45:05 bmbouter: do we know how bots work on matrix? 14:45:10 we do 14:45:22 ggainey: I can look into that 14:45:28 oh how... yeah there are do 14:45:29 @mcorr 14:45:32 oops, bah 14:45:33 docs 14:45:57 mcorr: I'm thinking about pulpbot and karma and meetings like, say, this one :) 14:45:59 yeah 14:46:09 Sure, ggainey let me take that as an item 14:46:20 we will need to move our meeting-bot over at the least. mcorr ty! 14:46:28 +1 14:46:38 Karma is good for the soul also 14:46:39 we could move to matrix for all the rooms except #pulp-meeting until the bot is in place 14:47:04 I will set up a space and Pulp rooms and advertise. What do you think? 14:47:18 We currently only have bridges 14:47:32 I mean, works4me, but that's partially because I've spent the last hour getting set up on matrix :) 14:47:52 survivor bias 14:47:54 :) 14:48:01 i am +1 on moving everything to matrix except meeting 14:49:17 ok, next topic? 14:49:28 sure 14:49:33 +1 14:50:09 Topic [combined]: Should we move our user mailing lists to Github Discussions or Discourse? 14:50:13 +1 for github discussions 14:50:31 huh, interesting 14:50:37 what's the reason for moving? 14:50:40 daviddavis: can we have Discussions on an organization level rather than repo level? 14:50:49 so we have 2 mailing list pulp-list and pulp-dev how we decouple those? 14:51:04 mcorr: I think they are all ported to organization level 14:51:14 from a "lowering impedance" POV, having everything next to the code (discussions, GH issues) makes sense 14:51:18 no, they are on the repo level I believe 14:51:24 yes on repo level 14:51:32 I think pulpcore would be the primary one in that case 14:51:48 I find the repo level issue the biggest detractor from them. 14:51:51 oh bleah - yeah, def pick one instead one-per-plugin 14:51:58 yea agreed 14:52:02 could anyone state the goals or the problems we are solving with the move? 14:52:05 yes 14:52:10 ttereshc: the logic would be to make it easier for users to ask and answer questions, not sign up to a ML to ask one question 14:52:21 yup what mcorr said 14:52:33 with Foreman, for example, we have data to show how engagement increased on RFCs after the move to Discourse 14:52:33 also, things like searching mailing lists aren't possible (or are difficult) 14:52:59 (well, our mailing lists anyway. Mailman is...kinda suboptimal, alas :( ) 14:53:14 ok, thx 14:53:21 it's also not browser native which creates more friction 14:53:42 Discourse is plugin based, which would allow us to centralise more of our activities around it 14:54:05 the negative would be either we pay for hosting, or self host, which would require some initial setup 14:54:09 any example of known projects which use github discussions? (to see how it looks at scale) 14:54:31 we avoid self-hosting at pretty much all costs 14:54:32 ttereshc: good question 14:54:33 yea, I have an example at https://github.com/daviddavis/pulpcore/discussions but also https://github.com/vercel/next.js/discussions 14:54:34 It's so new I've yet to see it in action but +1 to ttereshc 's point 14:54:44 we also avoid pay-for services at basically any cost 14:54:47 yeah, self-hosting is a non-starter, we have too much to do already 14:55:04 Can we change the categories daviddavis ? 14:55:17 yea, they're customizable 14:55:31 ty daviddavis , that helps 14:55:32 it is possible to have discussions at org level: https://github.com/github 14:55:41 oh nice 14:55:48 awesome! 14:55:48 niccceee 14:55:54 it gathers discussions from all repos: https://github.com/orgs/github/discussions 14:56:00 ah nice 14:56:07 also I think there is a big benefit to just using tooling from one provider as much as possible (github in this case) 14:56:10 RE categories - that would also answer the "differentiate between -list and -dev", yeah? 14:56:21 +1 to this from me if we can host on an org level 14:56:32 ggainey[m]: I think so 14:56:41 we could have a Development category 14:56:56 and then an Announcement category and Help category? 14:56:57 if we look at the real reason we have two mailing lists it's because people didn't want to constantly be emailed for one or the other... with people not receiving email I'm not sure it really matters anymore 14:57:08 +1 14:57:09 +1 to these categories 14:57:11 maybe we can make this That Summer When Pulp Got All Its Comms In One Place :) 14:57:22 lol 14:57:33 yes please! 14:57:53 my probably one of arguments would be to move to discussions once we move to github issues. it will be very un logical to have this spread 14:58:10 I mean, there's a bunch of work to do to get there - but even Grant The Curmudgeon thinks we should do this (or at least something a lot like this) 14:58:12 what is blocking us moving to github issues? 14:58:21 a well thought out, prioritized plan 14:58:22 the amount of work 14:58:24 yup 14:58:27 yeah, zacly 14:58:30 lol 14:58:31 which is why I don't want to connect these things 14:58:36 and priorities 14:58:39 we need to retire the mailing lists ... and soon 14:58:49 +1 not to connect those 14:59:01 +1 to not connect 14:59:22 is there a roadmap or timeline or is it just a hope atm to move to GH issues? 14:59:25 bmbouter: i would like to avoid frequent explanations of why i need to start a discussion on github but i still cannot file a github issue.. 14:59:35 mcorr: it's Aspirational still 14:59:54 github discussions seemed to improved from what I saw few months ago, it looks quite convenient, so +1 to move there 15:00:15 ipanova: I'm happy to handle those conversations :) 15:00:41 we already have feedback from users that our workflows are very much spread/decentralized. 15:00:56 yea, but issue tracking and mailing lists are already spread IMO 15:01:03 on pulpkhan we announce to the world we going full github 15:01:10 agreed, I don't think this makes it any worse 15:01:18 oh I was thinking switch to discussions like in a few days... 15:01:19 i am excited about how GH discussions look regardless the argument i have previously stated 15:01:24 I think ppl will put up with decentralized if/when we have a plan for how/when we're fixing that 15:01:36 heheh 15:01:56 what if we set up pulpcore discussions as a trial for a month or so and then we decommision the mailing list if everyone is happy? 15:02:07 can we not do repo level though ever 15:02:16 +1 to both of the above 15:02:19 wait can't we do repo level? (I'm confused) 15:02:25 full github: actions (x) discussions (loading), issues (loading) 15:02:31 We can but I really don't think we should 15:02:41 I think we have to do repo level although you can access discussion across repos at the org level 15:02:54 Ah 15:03:02 the example fao89 shared (https://github.com/orgs/github/discussions) shows discussions across repos 15:03:10 but we can just standardize and do only pulpcore 15:03:24 from what I saw, org level == gather all repo level 15:03:39 oic yup, that's fine pulpcore can default, other plugins for other plugin-specific topics 15:03:41 can someone raise an issue from the org view? 15:03:41 yeah - let's not make users guess which plugin they think they're having an issue with 15:03:52 (esp since I don't think you can move a GH discussion from one repo to another? is that correct?) 15:03:59 can we recap this as a proposal with an AI? 15:03:59 mcorr: a discussion? no 15:04:13 hmmm 15:04:23 that's a major detractor imv 15:04:31 yeah the one downside w/ GH issues vs plan.io, is you can't move issues 15:04:48 we can't continue with plan.io tho 15:05:10 yeah, it's a downside, but I think the downsides of "where we are" outweighs it 15:05:17 and part of the issues is that NextThing will always have one thing maybe not as great, which creates a huge advantage for our current status quo 15:05:36 yeah exactly, so instead of thinking about one specific thing we loose think about it as a balance of what we get versus loose 15:05:42 fair point, just Discourse does this better 15:05:46 also lose 15:05:51 versus loose 15:05:53 english is hard 15:05:55 heh 15:06:07 bmbouter: in Matrix, you can edit your typos 15:06:11 (which I hate, actually :) ) 15:06:16 ha 15:06:17 oh I know, 3 year matrix user over here 15:06:22 ha! kk 15:06:31 so does anyone object to setting up a pulpcore discourse and trying it out? 15:06:35 err discussion 15:06:41 ololol 15:06:44 ggainey[m]: I love that feature! 15:06:44 hehe 15:07:04 hehe 15:07:11 can we admit to the community that we want off plan.io 15:07:21 I think so 15:07:26 absolutely, it was a topic at the last pulpcon 15:07:26 Perfect 15:07:44 ok, I'll take an AI and email the lists 15:07:47 with a proposal 15:07:51 I think I'd like to see a list of actions/steps, even if it's just a list of bullet points 15:07:59 there we go 15:08:04 daviddavis++ 15:08:04 ggainey[m]: daviddavis's karma is now 517 15:08:06 daviddavis: +! and ty 15:08:12 thanks daviddavis 15:08:21 Topic: What to replace freenode with? 15:08:39 matrix 15:08:41 Matrix 15:08:43 so I didn't realize we had agenda undone from last time 15:08:45 Matrix 15:08:59 so this is a duplicate agenda item accidentally put by me 15:09:03 +1 15:09:04 I would like to see us go matrix-primary. I think it's going to take time to get users off of IRC, that aren't already on Matrix. I really want us off of freenode asap 15:09:31 +1 15:09:34 triage time? 15:09:43 !next 15:09:45 +1 15:09:45 daviddavis: 13 issues left to triage: 8810, 8803, 8801, 8798, 8797, 8796, 8793, 8791, 8789, 8782, 8781, 8779, 8744 15:09:45 #topic https://pulp.plan.io/issues/8810 15:09:46 RM 8810 - fao89 - NEW - Operator fails to deploy pulp-api 15:09:47 https://pulp.plan.io/issues/8810 15:09:58 !skip 15:09:59 #topic https://pulp.plan.io/issues/8803 15:09:59 daviddavis: 12 issues left to triage: 8803, 8801, 8798, 8797, 8796, 8793, 8791, 8789, 8782, 8781, 8779, 8744 15:10:00 RM 8803 - bmbouter - NEW - As a user, I have docs on how to integrate with a Keycloak server 15:10:01 https://pulp.plan.io/issues/8803 15:10:06 we need to change triage query 15:10:10 +1 15:10:16 #idea Proposed for #8803: convert to user story 15:10:16 !propose other convert to user story 15:10:16 daviddavis: Proposed for #8803: convert to user story 15:10:25 +1 15:10:31 #agreed convert to user story 15:10:31 !accept 15:10:31 daviddavis: Current proposal accepted: convert to user story 15:10:32 #topic https://pulp.plan.io/issues/8801 15:10:32 daviddavis: 11 issues left to triage: 8801, 8798, 8797, 8796, 8793, 8791, 8789, 8782, 8781, 8779, 8744 15:10:33 RM 8801 - bmbouter - NEW - Move existing Webserver Auth docs to new Authenticaiton top-level section 15:10:34 https://pulp.plan.io/issues/8801 15:10:42 !propose task 15:10:42 daviddavis: Error: "propose" is not a valid command. 15:10:44 same 15:10:50 #idea Proposed for #8801: convert to task 15:10:50 !propose other convert to task 15:10:50 daviddavis: Proposed for #8801: convert to task 15:10:55 #agreed convert to task 15:10:55 !accept 15:10:55 daviddavis: Current proposal accepted: convert to task 15:10:56 #topic https://pulp.plan.io/issues/8798 15:10:56 daviddavis: 10 issues left to triage: 8798, 8797, 8796, 8793, 8791, 8789, 8782, 8781, 8779, 8744 15:10:57 RM 8798 - daviddavis - NEW - Content summary shows incorrect numbers when previous version is deleted 15:10:58 https://pulp.plan.io/issues/8798 15:10:59 +1 15:11:10 accept 15:11:17 +1 15:11:19 +1 15:11:27 #idea Proposed for #8798: Leave the issue as-is, accepting its current state. 15:11:27 !propose accept 15:11:27 daviddavis: Proposed for #8798: Leave the issue as-is, accepting its current state. 15:11:35 #agreed Leave the issue as-is, accepting its current state. 15:11:35 !accept 15:11:35 daviddavis: Current proposal accepted: Leave the issue as-is, accepting its current state. 15:11:36 daviddavis: 9 issues left to triage: 8797, 8796, 8793, 8791, 8789, 8782, 8781, 8779, 8744 15:11:36 #topic https://pulp.plan.io/issues/8797 15:11:37 RM 8797 - wibbit - NEW - Pulp location of CSS etc for API browser (and/or dependancies) 15:11:38 https://pulp.plan.io/issues/8797 15:12:16 that sceenshot tho :) 15:12:26 welcome to 1996 15:13:34 accept? 15:13:44 +1 15:13:56 #idea Proposed for #8797: Leave the issue as-is, accepting its current state. 15:13:56 !propose accept 15:13:56 daviddavis: Proposed for #8797: Leave the issue as-is, accepting its current state. 15:14:04 #agreed Leave the issue as-is, accepting its current state. 15:14:04 !accept 15:14:04 daviddavis: Current proposal accepted: Leave the issue as-is, accepting its current state. 15:14:05 #topic https://pulp.plan.io/issues/8796 15:14:05 daviddavis: 8 issues left to triage: 8796, 8793, 8791, 8789, 8782, 8781, 8779, 8744 15:14:06 RM 8796 - xzhang1 - NEW - `pulpcore-manager` failed for AttributeError: 'Settings' object has no attribute 'CONTENT_ORIGIN' 15:14:07 https://pulp.plan.io/issues/8796 15:14:24 user's found the solution and asked to close - close notabug 15:14:27 +1 15:14:37 +1 15:14:38 #idea Proposed for #8796: close notabug 15:14:38 !propose other close notabug 15:14:38 daviddavis: Proposed for #8796: close notabug 15:14:43 +1 15:14:48 err works for me rather 15:14:54 #agreed close notabug 15:14:54 !accept 15:14:54 daviddavis: Current proposal accepted: close notabug 15:14:55 #topic https://pulp.plan.io/issues/8793 15:14:55 daviddavis: 7 issues left to triage: 8793, 8791, 8789, 8782, 8781, 8779, 8744 15:14:56 RM 8793 - gerrod - POST - Retained_versions doesn't properly keep all content present in repository 15:14:57 https://pulp.plan.io/issues/8793 15:15:00 actually, does not really matter 15:15:06 oh ok 15:15:12 in post 15:15:18 accept and add to the sprint 15:15:19 accept, add to sprint 15:15:20 +1 15:15:21 #idea Proposed for #8793: accept and add to sprint 15:15:21 !propose other accept and add to sprint 15:15:21 daviddavis: Proposed for #8793: accept and add to sprint 15:15:22 #agreed accept and add to sprint 15:15:22 !accept 15:15:22 daviddavis: Current proposal accepted: accept and add to sprint 15:15:23 daviddavis: 6 issues left to triage: 8791, 8789, 8782, 8781, 8779, 8744 15:15:23 #topic https://pulp.plan.io/issues/8791 15:15:24 RM 8791 - wibbit - NEW - Usability of "fields" when querying rpm content types 15:15:25 https://pulp.plan.io/issues/8791 15:15:51 oh yeah, we had some discussion w/wibbit on this one 15:15:53 I remember discussing this one 15:16:31 I think fields require full list, including the required ones 15:16:34 yeah - you can limit fields, but if you don't include mandatory ones (which aren't documented anywhere/clearly), you're going to have a Bad Time 15:17:06 it's very painful, +1 to fix this ux issue 15:17:22 I'd say accept, I don't know that I'd add to the sprint - but yeah, we def want to do "something" here 15:17:50 ttereshc: i think i recall something similar lubos fixed once already.. 15:18:03 but yeah +1 to accept 15:18:22 #idea Proposed for #8791: Leave the issue as-is, accepting its current state. 15:18:22 !propose accept 15:18:22 daviddavis: Proposed for #8791: Leave the issue as-is, accepting its current state. 15:18:31 #agreed Leave the issue as-is, accepting its current state. 15:18:31 !accept 15:18:31 daviddavis: Current proposal accepted: Leave the issue as-is, accepting its current state. 15:18:32 #topic https://pulp.plan.io/issues/8789 15:18:32 daviddavis: 5 issues left to triage: 8789, 8782, 8781, 8779, 8744 15:18:33 RM 8789 - wibbit - NEW - Take down readthedocs sites 15:18:34 https://pulp.plan.io/issues/8789 15:18:40 task 15:18:44 dkliban: can you work on this ^? 15:18:54 and add - this is confusing :( 15:19:05 #idea Proposed for #8789: convert to dkliban task 15:19:05 !propose other convert to dkliban task 15:19:05 daviddavis: Proposed for #8789: convert to dkliban task 15:19:07 yes please! 15:19:10 yeah i can 15:19:13 dkliban++ 15:19:13 daviddavis: dkliban's karma is now 596 15:19:16 #agreed convert to dkliban task 15:19:16 !accept 15:19:16 daviddavis: Current proposal accepted: convert to dkliban task 15:19:16 #topic https://pulp.plan.io/issues/8782 15:19:17 daviddavis: 4 issues left to triage: 8782, 8781, 8779, 8744 15:19:17 also I think python never moved maybe? 15:19:18 dkliban++ 15:19:18 RM 8782 - alikins - NEW - Intermittent psycopg2.errors.AdminShutdown errors in galaxy_ng dev env and pulp-all-in-one 15:19:19 https://pulp.plan.io/issues/8782 15:19:20 ggainey[m]: dkliban's karma is now 597 15:19:40 it would be also good to agree how we do it, since they show up in google search 15:19:42 yeah https://docs.pulpproject.org/pulp_python/ 15:19:55 gerrod: fyi ^ 15:20:03 can we have a redirect or some announcement and not just put it down 15:20:09 +1 15:20:20 we aren't in control of redirects 15:20:54 we could post an announcement 15:21:05 ttereshc: we should also take down docs for the migration plugin 15:21:22 these sites are already on docs.pulpproject.org? 15:21:23 ipanova, it's there, in the list 15:21:41 ttereshc: ok 15:21:48 I think each one needs to be checked... I just happened to notice pulp_python wasn't but I didn't check every one (just the ones for my prezi) 15:21:57 +1 15:22:05 pulp-installer and pulp-operator are on readthedocs 15:22:26 maybe we need to move them for consistency 15:22:32 sounds like we need an epic, to clean up all the pieces parts 15:22:36 yeah 15:22:40 I see. well hopefully dkliban will do all this and send out an announcement. 15:22:47 2021: The Summer Pulp Got Consistent :) 15:22:52 hehe 15:22:54 (both use mkdocs which deploys differently than plugins) 15:23:06 ahhhh 15:23:38 so, 8782 - accept? 15:23:55 +1 to accept 15:24:02 convert to a task 15:24:09 to an epic task 15:24:12 8782? 15:24:19 oops, sorry 15:24:21 no :) 15:24:27 #idea Proposed for #8782: Leave the issue as-is, accepting its current state. 15:24:27 !propose accept 15:24:28 daviddavis: Proposed for #8782: Leave the issue as-is, accepting its current state. 15:24:29 yeah so re 8782 ... I have also seen this 15:24:37 I propose we leave untriaged 15:24:42 ttereshc: I'll tell dkliban to convert it to an epic ;) 15:24:44 we have no reproducer and it needs resolution 15:24:49 #idea Proposed for #8782: Skip this issue for this triage session. 15:24:49 !propose skip 15:24:49 daviddavis: Proposed for #8782: Skip this issue for this triage session. 15:24:54 ah kk 15:24:58 +1 15:25:05 bmbouter: is alikins aware? 15:25:13 aware of ? 15:25:20 that we need a reproducer from him 15:25:35 so I think I have to best reproducer actually 15:25:40 oh ok 15:25:41 he said he couldn't reproduce it, same with @himnel 15:25:46 +1 15:25:47 but it only happens about 25% of the time 15:25:54 I can post what I have on it 15:25:57 re 8782, yeah. I haven't had a chance to reproduce it well. But bmbouter as seen it. 15:26:06 +1 15:26:12 let's skip now and I'll post what I have on it 15:26:17 #agreed Skip this issue for this triage session. 15:26:17 !accept 15:26:17 daviddavis: Current proposal accepted: Skip this issue for this triage session. 15:26:19 #topic https://pulp.plan.io/issues/8781 15:26:19 daviddavis: 3 issues left to triage: 8781, 8779, 8744 15:26:21 RM 8781 - newswangerd - NEW - /pulp/api/v3/ throws a 500 15:26:22 https://pulp.plan.io/issues/8781 15:26:32 huh, that's rude 15:26:56 it's not happening for me though 15:26:59 me neither 15:27:19 let me ask about what plugins and pulpcore version they're using 15:27:28 the traceback leads to the galaxy_ng 15:27:35 yea 15:27:35 agreed 15:27:43 I'll comment 15:27:48 #idea Proposed for #8781: daviddavis to comment 15:27:48 !propose other daviddavis to comment 15:27:48 daviddavis: Proposed for #8781: daviddavis to comment 15:27:59 and this endpoint should return schema I think, and not 404 15:28:08 yea, that's what it does for me 15:28:12 but I don't have galaxy ng 15:28:17 yeah, I just tested and that's what I see 15:28:30 +1 to proposal 15:28:35 but yeah, I'm on pulp2-pulp3-centos 15:28:58 +1 15:29:13 #agreed daviddavis to comment 15:29:13 !accept 15:29:13 daviddavis: Current proposal accepted: daviddavis to comment 15:29:14 #topic https://pulp.plan.io/issues/8779 15:29:14 daviddavis: 2 issues left to triage: 8779, 8744 15:29:16 RM 8779 - adam.winberg@smhi.se - NEW - Task started on removed worker 15:29:17 https://pulp.plan.io/issues/8779 15:30:01 oh wow - I think this is the problem we've seen several times now, when something catastrophic happens on pulp3 15:30:16 yup 15:30:23 trying to recover from "tasks assigned to a worker that doesn't exist any more" 15:30:28 have we seen it on 3.12+? 15:30:37 hrm 15:30:39 so we made a series of "improvements" I'd have to look at versions but I think as far back as 3.9 15:31:21 dalley: I think we should consider reverting anything on 3.11+ ... 15:31:24 or do nothing 15:31:32 and recommend users use the new tasking system with 3.14 15:32:37 user is on 3.11, there were fixes in 3.12, so I'm not sure if they affect this behavior but they might 15:32:44 this is going to bite us , esp on systems under heavy load (since that's where "I ran out of RAM/disk" is most likely to happen) 15:32:51 that's potentially the correct thing for users, but "downstream" needs a working tasking system one way or another 15:32:58 yeah 15:33:12 I don't think we can make the split data over postgresql and redis reliable 15:33:14 it's not like the tasking system was working properly prior to these fixes even if they're incomplete 15:33:16 or transactionally safe 15:33:43 I think the only way to solve this is to have downstream (or anyone) switch to the new system with 3.14 15:34:16 but we can tackle this in #pulp-dev, for triage what do we want to do? 15:34:28 skip? 15:34:33 I'm ok w/ skip 15:34:37 try to collect all of these issues and figure which are dups 15:34:37 or accept 15:35:05 dalley: +1 to that 15:35:10 personally, we know this is an issue, we should probably accept 15:35:23 #idea Proposed for #8779: Leave the issue as-is, accepting its current state. 15:35:23 !propose accept 15:35:23 daviddavis: Proposed for #8779: Leave the issue as-is, accepting its current state. 15:35:38 that's ok w/ me, my concern is that after accepting we don't act 15:35:39 yeah, +1 15:35:52 also I don't know how to use commas 15:35:53 +1 15:35:53 yeah, no, we absolutely need to get this figured out 15:36:00 dalley does tho :) 15:36:08 lol 15:36:08 heheh 15:36:08 :) 15:36:28 #idea Proposed for #8779: Skip this issue for this triage session. 15:36:28 !propose skip 15:36:28 daviddavis: Proposed for #8779: Skip this issue for this triage session. 15:36:36 whichever issue is left standing afterwards should probably be high sev / high prio 15:36:37 I'm ok w/ either, let's convo about it in #pulp-dev 15:36:41 agreed 15:36:43 #agreed Skip this issue for this triage session. 15:36:43 !accept 15:36:43 daviddavis: Current proposal accepted: Skip this issue for this triage session. 15:36:44 daviddavis: 1 issues left to triage: 8744 15:36:44 #topic https://pulp.plan.io/issues/8744 15:36:45 RM 8744 - equipe_serveurs@exane.com - NEW - Pulp3 database migration broken with newly release click 8 15:36:46 https://pulp.plan.io/issues/8744 15:36:54 is this one fixed? 15:36:57 this fixed now yeah? 15:37:13 * daviddavis hears an echo 15:37:14 ^fixed^is fixed 15:37:36 it'll be fixed on 3.11 and upcoming 3.13 15:37:45 depends on the pulpcore version 15:37:53 ah 15:37:56 3.12 may break for example 15:37:57 pulpcore==3.11.2, pulpcore==3.13.0+ 15:38:14 yup and recalling our convo from earlier, we can't afford to do better right now 15:38:23 yep 15:38:35 in fact I'm still trying to get 3.11.2 out the door :/ 15:38:43 fun times 15:38:46 ja 15:39:03 accept and add, then close when 3.13 goes out? 15:39:22 I can also comment on it now indicating which pulpcore will resolve it... 15:39:30 or that they can manually downgrade click and upgrade RQ 15:39:42 #idea Proposed for #8744: bmbouter to comment on issue 15:39:42 !propose other bmbouter to comment on issue 15:39:42 daviddavis: Proposed for #8744: bmbouter to comment on issue 15:39:46 +1 15:39:52 #agreed bmbouter to comment on issue 15:39:52 !accept 15:39:52 daviddavis: Current proposal accepted: bmbouter to comment on issue 15:39:53 daviddavis: No issues to triage. 15:39:55 #endmeeting 15:39:55 !end