14:30:49 #startmeeting Pulp Triage 2021-06-08 14:30:49 #info dkliban has joined triage 14:30:49 !start 14:30:49 Meeting started Tue Jun 8 14:30:49 2021 UTC. The chair is dkliban. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:30:49 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:30:49 The meeting name has been set to 'pulp_triage_2021-06-08' 14:30:49 dkliban: dkliban has joined triage 14:32:04 #info ppicka has joined triage 14:32:04 !here 14:32:04 ppicka: ppicka has joined triage 14:32:34 I am going to wait a little bit longer for people to join before we begin discussion on our only topic for today 14:32:36 #info daviddavis has joined triage 14:32:36 !here 14:32:36 daviddavis: daviddavis has joined triage 14:32:39 #info bmbouter has joined triage 14:32:39 !here 14:32:39 bmbouter: bmbouter has joined triage 14:32:52 Topic: pulpcore 3.14 release date 14:33:01 tomorrow 14:33:01 #info ttereshc has joined triage 14:33:01 !here 14:33:01 ttereshc: ttereshc has joined triage 14:33:26 #info ipanova has joined triage 14:33:26 !here 14:33:26 ipanova: ipanova has joined triage 14:33:27 daviddavis: we should look at the list of issues still in progress 14:33:38 dkliban: I know I was joking 14:33:48 daviddavis: i wanted to take you very seriously 14:33:53 haha 14:33:58 :D 14:34:08 here's the milestone https://pulp.plan.io/versions/218 14:34:24 I think the big thing though was the tasking system 14:34:41 yeah 14:34:42 is it merged now? 14:34:50 no 14:34:56 #info ggainey[m] has joined triage 14:34:56 !here 14:34:56 ggainey[m]: ggainey[m] has joined triage 14:35:20 #info gerrod has joined triage 14:35:20 !here 14:35:20 gerrod: gerrod has joined triage 14:35:30 what are the next steps for the tasking system?> 14:35:36 today at the pulpcore meeting we discussed that it needs more discussion 14:35:45 I think 14:35:51 bmbouter, x9c4 ^ 14:35:59 I'm confused about what we're talking about 14:36:11 are we talking about 3.14, or issues or ? 14:36:21 #info mikedep333 has joined triage 14:36:21 !here 14:36:21 mikedep333: mikedep333 has joined triage 14:36:26 3.14 and tasking system changes readiness 14:36:47 ah ok 14:37:09 bmbouter: we are discussing when to release 3.14 ... and it seems like we are waiting on tasking system chnages plus some smaller issues 14:37:13 my understanding is that the issue wibbit experienced was root caused in not the tasking system 14:37:30 and for 3.14 we are waiting on the tasking system change plus what is already on the sprint 14:37:38 s/sprint/milestone/ 14:38:08 x9c4: the tasking system epic needs the 3.14 milestone on it 14:39:03 our 3.13 release was may 25th 14:39:16 and we need 3.15 to release at end of july early aug 14:39:37 so what about june 29 as target 3.14 release date (also I'll be the releaser of it) 14:39:44 +1 14:39:46 we have released 3.13 on may 27, can we target 3.14 end of month also? 14:40:05 +1 to June 29 14:40:25 June 29th works for me 14:40:26 works4me, +1 14:40:32 +1 14:40:44 I would hold a go/no-go checkin on the 15th and then again on the 22nd 14:40:56 sunds gud 14:41:04 sounds good 14:41:11 +1 toall that 14:41:12 +1 14:41:29 x9c4: can you tag your tasking system redmine issues with 3.14 please? 14:41:41 whenever you see ^ is fine 14:41:58 Does anyone else have a topic they wish to discuss at open floor before we move to bug triage? 14:43:02 nope 14:43:10 in that case let's do it ... 14:43:13 !next 14:43:15 #topic https://pulp.plan.io/issues/8873 14:43:15 dkliban: 12 issues left to triage: 8873, 8870, 8868, 8867, 8866, 8865, 8864, 8859, 8852, 8839, 8782, 8779 14:43:16 RM 8873 - mwaqas@bloombergindustry.com - NEW - pulp 3 stops when encounters 403 error 14:43:17 https://pulp.plan.io/issues/8873 14:44:20 bmbouter: ^ do you have any thoghts? 14:44:55 there has been talk of having an option to continue 14:45:46 katello wanted continuation on 404 14:45:51 i say we should accept but not add to the sprint right now 14:46:01 it would really need to be a story 14:46:04 so this sounds like an RFE - also, needs some serious discussion, what does it mean to sync only-some-of a repository? 14:46:13 exactly 14:46:15 agreed convert to RFE 14:46:16 so yeah - story, accept, not on sprint, would be my take 14:46:23 yeah, ad it would be nice to have an option, different users might want different behavior 14:46:35 agreed 14:46:39 #idea Proposed for #8873: convert to story 14:46:39 !propose other convert to story 14:46:39 dkliban: Proposed for #8873: convert to story 14:46:41 +1 to allofthen 14:46:47 coolio 14:46:47 that* 14:47:05 #agreed convert to story 14:47:05 !accept 14:47:05 dkliban: Current proposal accepted: convert to story 14:47:06 #topic https://pulp.plan.io/issues/8870 14:47:06 dkliban: 11 issues left to triage: 8870, 8868, 8867, 8866, 8865, 8864, 8859, 8852, 8839, 8782, 8779 14:47:07 RM 8870 - dalley - NEW - Packages which weren't published at the repository root are available at the repository root 14:47:08 https://pulp.plan.io/issues/8870 14:47:40 is this an rpm issue? 14:47:41 move to rpm? 14:48:00 yeah i think so 14:48:09 +1 14:48:10 yeah 14:48:11 +1 14:48:14 +1 14:48:17 I don't believe it's necessarily an RPM issue 14:48:22 it might be, though 14:48:42 I don't object to moving it until we know 14:48:46 a reproduer with pulp_file would show 14:48:53 but as its rpm right now I think it's rpm 14:49:03 #idea Proposed for #8870: move to RPM project 14:49:03 !propose other move to RPM project 14:49:03 dkliban: Proposed for #8870: move to RPM project 14:49:50 #agreed move to RPM project 14:49:50 !accept 14:49:50 dkliban: Current proposal accepted: move to RPM project 14:49:51 #topic https://pulp.plan.io/issues/8868 14:49:51 dkliban: 10 issues left to triage: 8868, 8867, 8866, 8865, 8864, 8859, 8852, 8839, 8782, 8779 14:49:52 RM 8868 - daviddavis - NEW - Bindings don't properly namespace API 14:49:53 https://pulp.plan.io/issues/8868 14:50:12 we discsused this with daviddavis and it needs to be added to the sprint 14:50:17 +1 14:50:23 also we'll have to notify Katello about this 14:50:26 since the bindings will change 14:50:31 #idea Proposed for #8868: accept and add to sprint 14:50:31 !propose other accept and add to sprint 14:50:31 dkliban: Proposed for #8868: accept and add to sprint 14:50:41 +1 14:51:04 #agreed accept and add to sprint 14:51:04 !accept 14:51:05 dkliban: Current proposal accepted: accept and add to sprint 14:51:05 #topic https://pulp.plan.io/issues/8867 14:51:06 dkliban: 9 issues left to triage: 8867, 8866, 8865, 8864, 8859, 8852, 8839, 8782, 8779 14:51:07 RM 8867 - jsherril@redhat.com - NEW - downloads from cdn.redhat fail with 'Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:877)]' 14:51:08 https://pulp.plan.io/issues/8867 14:51:55 so this is tough because issues on the cdn are best resolved on the cdn 14:51:56 do they need to use rate limiting? 14:52:15 i suggest to email exd folks and raise the awareness and see what they reply 14:52:26 this looks to me like a cfg-error on a cdn node 14:52:27 maybe we need a discussion with cdn folks 14:52:29 yeah 14:52:30 +1 ipanova 14:52:33 +1 14:52:39 should we skip for now? 14:52:46 at least to set expectation on the load and so on 14:52:55 yeah skip 14:52:57 i can start the thread for now and refer to this issue, let's skip for now 14:53:04 ipanova++ 14:53:04 dkliban: ipanova's karma is now 227 14:53:07 ipanova++ 14:53:07 ttereshc: ipanova's karma is now 228 14:53:08 ipanova++ 14:53:09 ggainey[m]: ipanova's karma is now 229 14:53:11 !skip 14:53:12 dkliban: 8 issues left to triage: 8866, 8865, 8864, 8859, 8852, 8839, 8782, 8779 14:53:12 #topic https://pulp.plan.io/issues/8866 14:53:13 RM 8866 - daviddavis - NEW - api.json is out of date 14:53:14 https://pulp.plan.io/issues/8866 14:53:35 +1 to remove 14:53:40 i believe the files should be removed 14:53:47 ok, I can update to a task then 14:53:48 +1 o remove 14:53:51 convert to a task 14:53:51 yeah, does anything rely on this file existing? 14:53:51 let's convert it to a task and add to sprint 14:53:55 +1 14:53:56 coolio 14:53:58 and add to gitignore? 14:54:00 some plugins already don't have it, i think 14:54:02 sure 14:54:03 ggainey[m], it used to but not anymore 14:54:09 gotcha 14:54:18 #idea Proposed for #8866: convert to task and add to sprint 14:54:18 !propose other convert to task and add to sprint 14:54:18 dkliban: Proposed for #8866: convert to task and add to sprint 14:54:33 i would probably not add to the sprint >.> 14:54:58 +1 to remove 14:55:03 it's a good task to do between the meetings 14:55:08 not add to sprint, either take as assigned or backlog (to me) 14:55:09 you can assign it tome 14:55:20 damnit, I just assigned it to me 14:55:23 lol 14:55:24 :D 14:55:24 haha 14:55:27 I'll unassign 14:55:28 lol 14:55:47 #idea Proposed for #8866: convert to task and let ttereshc and daviddavis battle it out 14:55:47 !propose other convert to task and let ttereshc and daviddavis battle it out 14:55:47 dkliban: Proposed for #8866: convert to task and let ttereshc and daviddavis battle it out 14:55:53 yes! 14:55:54 :) 14:55:56 (: 14:56:03 lol yes! 14:56:03 #agreed convert to task and let ttereshc and daviddavis battle it out 14:56:03 !accept 14:56:03 dkliban: Current proposal accepted: convert to task and let ttereshc and daviddavis battle it out 14:56:04 #topic https://pulp.plan.io/issues/8865 14:56:04 dkliban: 7 issues left to triage: 8865, 8864, 8859, 8852, 8839, 8782, 8779 14:56:05 RM 8865 - jsherril@redhat.com - NEW - Range requests for on demand content return the full file (Kickstarts fail for on_demand repos) 14:56:06 https://pulp.plan.io/issues/8865 14:56:19 oh yeah dalley and I were talking about this on fri 14:56:35 i see there was no success in reproducing it 14:56:42 ipanova, I reproduced on 3.11 14:56:51 I think we should write a test for it against 3.14 at least 14:56:55 ah nice 14:57:04 I had trouble on 3.14, but I'm not sure if I missed something, so I was going to try again today 14:57:15 I used a slightly different process 14:57:18 dalley: can you do it with a test? 14:57:22 #idea Proposed for #8865: accept and add to sprint 14:57:22 !propose other accept and add to sprint 14:57:22 dkliban: Proposed for #8865: accept and add to sprint 14:57:29 +1 14:57:32 +1 14:57:40 since dalley is already spending time on it 14:57:42 +1 14:57:49 it also has a katello flag 14:57:51 #agreed accept and add to sprint 14:57:51 !accept 14:57:51 dkliban: Current proposal accepted: accept and add to sprint 14:57:51 #topic https://pulp.plan.io/issues/8864 14:57:52 dkliban: 6 issues left to triage: 8864, 8859, 8852, 8839, 8782, 8779 14:57:52 dalley: can yo ualso take as ASSIGNED if working on it 14:57:53 RM 8864 - ehelms@redhat.com - NEW - Workers go OOM while trying to sync RHEL 7 14:57:54 https://pulp.plan.io/issues/8864 14:58:03 ah, rhel7 14:58:11 wait, 8Gb? that's...a small system 14:58:16 no, 3.5 14:58:23 sure, I can work on it. but lol I got volunteered quickly :) 14:58:28 3.5 GB available 14:58:42 yeah that isn't gonna work 14:58:46 3.5Gb 'available', system in the issue has 8Gb total 14:58:53 yeah 14:58:56 dalley: it's ok if not to work on it we just need to know 14:59:08 bmbouter, no, it's fine 15:00:09 the repo metadata decompresses to like 4gb 15:00:16 yup 15:00:24 I would move it to rpm and comment that it's not enough RAM 15:00:26 and there's some overhead as well 15:00:32 +1 15:00:54 +1 15:00:55 we have bigger issues with copy and depsolving of rhel7, 21GB is not enough 15:01:10 +1 15:01:18 #idea Proposed for #8864: move to rpm 15:01:18 !propose other move to rpm 15:01:18 ttereshc: Proposed for #8864: move to rpm 15:01:39 also - this is 3.11, I know we (meaning dalley) did some memory work 'recently', was that in 3.11? 15:01:46 +1 to move, tho 15:01:54 #idea Proposed for #8864: more to RPM project 15:01:54 !propose other more to RPM project 15:01:54 dkliban: Proposed for #8864: more to RPM project 15:01:57 I think we backported everything improtant 15:02:05 ah right 15:02:17 ggainey[m], yeah IIRC it should be in 3.11 15:02:20 #agreed more to RPM project 15:02:20 !accept 15:02:20 dkliban: Current proposal accepted: more to RPM project 15:02:21 #topic https://pulp.plan.io/issues/8859 15:02:21 dkliban: 5 issues left to triage: 8859, 8852, 8839, 8782, 8779 15:02:22 RM 8859 - wibbit - NEW - Minor documentation error 15:02:23 https://pulp.plan.io/issues/8859 15:02:38 accept, mark as docs 15:02:43 add documentation tag 15:02:54 #idea Proposed for #8859: accept and add documentation tag 15:02:54 !propose other accept and add documentation tag 15:02:54 dkliban: Proposed for #8859: accept and add documentation tag 15:02:57 +1 15:03:00 +1 15:03:11 #agreed accept and add documentation tag 15:03:11 !accept 15:03:11 dkliban: Current proposal accepted: accept and add documentation tag 15:03:12 dkliban: 4 issues left to triage: 8852, 8839, 8782, 8779 15:03:12 #topic https://pulp.plan.io/issues/8852 15:03:13 RM 8852 - jsherril@redhat.com - NEW - Delay publication file deletion to orphan cleanup to aid in online backups 15:03:14 https://pulp.plan.io/issues/8852 15:03:42 I posted my concern as comment 1 15:03:51 RFE 15:04:17 #idea Proposed for #8852: convert to story 15:04:17 !propose other convert to story 15:04:17 dkliban: Proposed for #8852: convert to story 15:04:28 +1 15:04:28 +1 15:04:30 and then we can keep the conversation going there 15:04:44 yeah sounds reasonable - looks like plenty of discussion to be had there :) 15:04:57 #agreed convert to story 15:04:57 !accept 15:04:57 dkliban: Current proposal accepted: convert to story 15:04:58 #topic https://pulp.plan.io/issues/8839 15:04:58 dkliban: 3 issues left to triage: 8839, 8782, 8779 15:05:00 RM 8839 - vk - NEW - file content upload performance needs improvement -- currently about 5x slower than rsync 15:05:01 https://pulp.plan.io/issues/8839 15:05:26 I think we can just accept. see https://pulp.plan.io/issues/8839#note-8 15:05:32 agreed 15:05:38 ah, kk 15:05:57 #idea Proposed for #8839: Leave the issue as-is, accepting its current state. 15:05:57 !propose accept 15:05:57 dkliban: Proposed for #8839: Leave the issue as-is, accepting its current state. 15:06:20 +1 15:06:25 #agreed Leave the issue as-is, accepting its current state. 15:06:25 !accept 15:06:25 dkliban: Current proposal accepted: Leave the issue as-is, accepting its current state. 15:06:26 #topic https://pulp.plan.io/issues/8782 15:06:26 dkliban: 2 issues left to triage: 8782, 8779 15:06:27 RM 8782 - alikins - NEW - Intermittent psycopg2.errors.AdminShutdown errors in galaxy_ng dev env and pulp-all-in-one 15:06:28 https://pulp.plan.io/issues/8782 15:07:57 accept? 15:08:07 yeah 15:08:24 #idea Proposed for #8782: Leave the issue as-is, accepting its current state. 15:08:24 !propose accept 15:08:24 dkliban: Proposed for #8782: Leave the issue as-is, accepting its current state. 15:08:28 we were skipping it, because it's not clear what to do 15:08:31 yup 15:08:45 I worry about it because if we accept it gets lost 15:08:52 definitly 15:09:13 let's skip until we have capacity to investigate 15:09:24 cause it will require someone digging into this 15:09:34 to fully understand the problem 15:09:36 bmbouter: there is a reproducer, no? 15:09:42 sort of, not 100% 15:10:31 alright, let's skip :) 15:10:38 #idea Proposed for #8782: Skip this issue for this triage session. 15:10:38 !propose skip 15:10:38 dkliban: Proposed for #8782: Skip this issue for this triage session. 15:11:02 #agreed Skip this issue for this triage session. 15:11:02 !accept 15:11:02 dkliban: Current proposal accepted: Skip this issue for this triage session. 15:11:03 #topic https://pulp.plan.io/issues/8779 15:11:03 dkliban: 1 issues left to triage: 8779 15:11:04 RM 8779 - adam.winberg@smhi.se - NEW - Task started on removed worker 15:11:05 https://pulp.plan.io/issues/8779 15:11:18 our favorite! 15:11:26 yes indeed, was thinking just that 15:12:35 yeah we discussed it today at the pulpcore meeting 15:12:39 same rationale as prev I guess - skip and revisit, till we have time to dig into current-tasking and figure out what's going on 15:12:39 accept? 15:12:56 i am inclined to accept 15:12:56 did we decide at the pulpcore meeting if we are just going to revert all the changes till this problem goes away? 15:13:02 it needs to be resolved within a month 15:14:15 I honestly don't think reverting changes is going to help this particular problem. We had "task assigned to dead worker" issues prior to 3.7/8/9/10 as well 15:14:28 but less users to report them 15:16:06 if we need a solution "soon" - accept-and-add 15:16:12 (and we prob do, this is a really painful failure state) 15:16:15 #idea Proposed for #8779: accept and add to sprint 15:16:15 !propose other accept and add to sprint 15:16:15 dkliban: Proposed for #8779: accept and add to sprint 15:16:27 +1 15:16:45 +1 15:16:48 I do not think we settled on the solution 15:16:49 #agreed accept and add to sprint 15:16:49 !accept 15:16:49 dkliban: Current proposal accepted: accept and add to sprint 15:16:50 dkliban: No issues to triage. 15:16:51 + high prio high sev 15:16:53 just that it needs reslution 15:17:33 +1 to accept and add to sprint 15:17:43 alright folks ... that's all we have for today 15:17:55 if it needs resolution within a month we can put the milestone on it 15:18:31 +1 15:18:38 ipanova, I'll check the deadlines, thanks 15:18:38 dkliban: ty for leading the triage, you're back in the game! 15:18:44 hehehe 15:18:45 you are welcome! 15:18:47 #endmeeting 15:18:47 !end